Skip to content

Commit 9fe2466

Browse files
committed
0.9.1 WIP
Former-commit-id: 792afa1
1 parent d9a3e8e commit 9fe2466

17 files changed

Lines changed: 511 additions & 85 deletions

README.md

Lines changed: 33 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,35 @@
11
# pgdiff - PostgreSQL schema diff
22

3-
pgdiff compares the schema between two PostgreSQL 9 databases and generates alter statements to be *manually* run against the second database. The provided pgdiff.sh script helps automate the process. At the moment, not everything in the schema is compared, but the things considered important are: roles, sequences, tables, columns (and their default values), primary keys, unique constraints, foreign keys, roles, ownership information, and grants.
3+
pgdiff compares the schema between two PostgreSQL 9 databases and generates alter statements to be *manually* run against the second database to make them match. The provided pgdiff.sh script helps automate the process.
44

5-
An important feature is that pgdiff never modifies a database directly. You alone are responsible for verifying the generated SQL *before* running it against your database, so you can have confidence this is safe to try and see what SQL gets generated.
5+
pgdiff is transparent in what it does, so it never modifies a database directly. You alone are responsible for verifying the generated SQL *before* running it against your database, so you can have confidence that pgdiff is safe to try and see what SQL gets generated.
66

7-
It is written to be easy to add and improve the accuracy of the diff. If you find something that seems wrong and you want me to look at it, please send me two schema-only database dumps that I can test with (Use the --schema-only option with pg\_dump)
7+
pgdiff is written to be easy to improve the accuracy of the diff. If you find something that seems wrong and you want me to look at it, please send me two schema-only database dumps that I can test with (Use the --schema-only option with pg\_dump)
88

99
### download
10-
[osx64](https://github.com/joncrlsn/pgrun/raw/master/bin-osx64/pgdiff "OSX 64-bit version")
11-
[osx32](https://github.com/joncrlsn/pgrun/raw/master/bin-osx32/pgdiff "OSX version")
12-
[linux64](https://github.com/joncrlsn/pgrun/raw/master/bin-linux64/pgdiff "Linux 64-bit version")
13-
[linux32](https://github.com/joncrlsn/pgrun/raw/master/bin-linux32/pgdiff "Linux version")
14-
[win64](https://github.com/joncrlsn/pgrun/raw/master/bin-win64/pgdiff.exe "Windows 64-bit version")
15-
[win32](https://github.com/joncrlsn/pgrun/raw/master/bin-win32/pgdiff.exe "Windows version")
16-
10+
[osx](https://github.com/joncrlsn/pgrun/raw/master/bin-osx/pgdiff "OSX version")
11+
[linux](https://github.com/joncrlsn/pgrun/raw/master/bin-linux/pgdiff "Linux version")
12+
[windows](https://github.com/joncrlsn/pgrun/raw/master/bin-win/pgdiff.exe "Windows version")
1713

1814
### usage
15+
pgdiff [options] <schemaType>
16+
17+
(where options are defined below and <schemaType> can be: ROLE, SEQUENCE, TABLE, COLUMN, INDEX, FUNCTION, VIEW, FOREIGN\_KEY, OWNER, GRANT\_RELATIONSHIP, GRANT\_ATTRIBUTE, TRIGGER)
1918

20-
pgdiff [options] <schemaType>
19+
I've found that there is an ideal order for running the different schema types. For example, you'll always want to add new tables before you add new columns. This is the order that has worked for me, however "your mileage may vary".
2120

21+
1. FUNCTION
22+
1. ROLE
23+
1. SEQUENCE
24+
1. TABLE
25+
1. VIEW
26+
1. OWNER
27+
1. COLUMN
28+
1. INDEX
29+
1. FOREIGN\_KEY
30+
1. GRANT\_RELATIONSHIP
31+
1. GRANT\_ATTRIBUTE
32+
1. TRIGGER
2233

2334
### options
2435

@@ -36,6 +47,16 @@ options | explanation
3647
-p, --port2 | second db port number. default is 5432
3748
-D, --dbname1 | first db name
3849
-d, --dbname2 | second db name
50+
-O, --option1 | first db options. example: sslmode=disable
51+
-o, --option2 | second db options. example: sslmode=disable
52+
3953

54+
### version history
55+
1. 0.9.0 - Implemented ROLE, SEQUENCE, TABLE, COLUMN, INDEX, FOREIGN\_KEY, OWNER, GRANT\_RELATIONSHIP, GRANT\_ATTRIBUTE
56+
1. 0.9.1 - Added VIEW, FUNCTION, and TRIGGER (Thank you, Shawn Carroll AKA SparkeyG)
4057

41-
&lt;schemaType&gt; can be: ROLE, SEQUENCE, TABLE, COLUMN, INDEX, FOREIGN_KEY, OWNER, GRANT_RELATIONSHIP, GRANT_ATTRIBUTE
58+
### todo
59+
1. fix SQL for adding an array column
60+
1. allow editing of individual SQL lines after failure (this would be done in the script pgdiff.sh)
61+
1. store failed SQL statements in an error file for later fixing and rerunning?
62+
1. add windows script (or even better: re-write bash script in Go)

bin-linux64/pgdiff.REMOVED.git-id

Lines changed: 0 additions & 1 deletion
This file was deleted.

bin-osx64/pgdiff.REMOVED.git-id

Lines changed: 0 additions & 1 deletion
This file was deleted.

bin-win64/pgdiff.exe.REMOVED.git-id

Lines changed: 0 additions & 1 deletion
This file was deleted.

build.sh

Lines changed: 29 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,45 +1,42 @@
1-
#!/bin/bash
1+
#!/bin/bash -x
22

3-
appname=pgdiff
3+
SCRIPT_DIR="$(dirname `ls -l $0 | awk '{ print $NF }'`)"
44

5-
if [[ -d bin-linux32 ]]; then
6-
GOOS=linux GOARCH=386 go build -o bin-linux32/${appname}
7-
echo "Built linux32."
8-
else
9-
echo "Skipping linux32. No bin-linux32 directory."
10-
fi
5+
[[ -z $APPNAME ]] && APPNAME=pgdiff
116

12-
if [[ -d bin-linux64 ]]; then
13-
GOOS=linux GOARCH=amd64 go build -o bin-linux64/${appname}
14-
echo "Built linux64."
7+
if [[ -d bin-linux ]]; then
8+
tempdir="$(mktemp -d -t $APPNAME)"
9+
workdir="$tempdir/$APPNAME"
10+
echo $workdir
11+
mkdir -p $workdir
12+
# Build the executable
13+
GOOS=linux GOARCH=386 go build -o "$workdir/$APPNAME"
14+
# Download pgrun to the temp directory
15+
wget -O "$workdir/pgrun" "https://github.com/joncrlsn/pgrun/raw/master/bin-linux/pgrun"
16+
# Copy the bash runtime script to the temp directory
17+
cp pgdiff.sh "$workdir/"
18+
cd "$tempdir"
19+
zip -r "${APPNAME}.zip" $APPNAME
20+
mv "${APPNAME}.zip" "$SCRIPT_DIR/bin-linux/"
21+
cd -
22+
echo "Built linux."
1523
else
16-
echo "Skipping linux64. No bin-linux64 directory."
24+
echo "Skipping linux. No bin-linux directory."
1725
fi
1826

19-
if [[ -d bin-osx32 ]]; then
20-
GOOS=darwin GOARCH=386 go build -o bin-osx32/${appname}
21-
echo "Built osx32."
22-
else
23-
echo "Skipping osx32. No bin-osx32 directory."
24-
fi
27+
#### DON'T LEAVE THIS
28+
exit 1
2529

26-
if [[ -d bin-osx64 ]]; then
27-
GOOS=darwin GOARCH=amd64 go build -o bin-osx64/${appname}
28-
echo "Built osx64."
30+
if [[ -d bin-osx ]]; then
31+
GOOS=darwin GOARCH=386 go build -o bin-osx/${APPNAME}
32+
echo "Built osx32."
2933
else
30-
echo "Skipping osx64. No bin-osx64 directory."
34+
echo "Skipping osx. No bin-osx directory."
3135
fi
3236

33-
if [[ -d bin-win32 ]]; then
34-
GOOS=windows GOARCH=386 go build -o bin-win32/${appname}.exe
37+
if [[ -d bin-win ]]; then
38+
GOOS=windows GOARCH=386 go build -o bin-win/${APPNAME}.exe
3539
echo "Built win32."
3640
else
37-
echo "Skipping win32. No bin-win32 directory."
38-
fi
39-
40-
if [[ -d bin-win64 ]]; then
41-
GOOS=windows GOARCH=amd64 go build -o bin-win64/${appname}.exe
42-
echo "Built win64."
43-
else
44-
echo "Skipping win64. No bin-win64 directory."
41+
echo "Skipping win. No bin-win directory."
4542
fi

column.go

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -89,10 +89,13 @@ func (c *ColumnSchema) Add() {
8989
if c.get("data_type") == "character varying" {
9090
maxLength, valid := getMaxLength(c.get("character_maximum_length"))
9191
if !valid {
92-
fmt.Println("-- WARNING: varchar column has no maximum length. Set to 1024")
92+
fmt.Println("-- WARNING: varchar column has no maximum length. Setting to 1024, which could result in data loss")
9393
}
9494
fmt.Printf("ALTER TABLE %s ADD COLUMN %s %s(%s)", c.get("table_name"), c.get("column_name"), c.get("data_type"), maxLength)
9595
} else {
96+
if c.get("data_type") == "ARRAY" {
97+
fmt.Println("-- Note that adding of array data types are not yet generated properly.")
98+
}
9699
fmt.Printf("ALTER TABLE %s ADD COLUMN %s %s", c.get("table_name"), c.get("column_name"), c.get("data_type"))
97100
}
98101

@@ -124,15 +127,15 @@ func (c *ColumnSchema) Change(obj interface{}) {
124127
max1, max1Valid := getMaxLength(c.get("character_maximum_length"))
125128
max2, max2Valid := getMaxLength(c2.get("character_maximum_length"))
126129
if (max1Valid || !max2Valid) && (max1 != c2.get("character_maximum_length")) {
127-
if !max1Valid {
128-
fmt.Println("-- WARNING: varchar column has no maximum length. Setting to 1024")
129-
}
130+
//if !max1Valid {
131+
// fmt.Println("-- WARNING: varchar column has no maximum length. Setting to 1024, which may result in data loss.")
132+
//}
130133
max1Int, err1 := strconv.Atoi(max1)
131134
check("converting string to int", err1)
132135
max2Int, err2 := strconv.Atoi(max2)
133136
check("converting string to int", err2)
134137
if max1Int < max2Int {
135-
fmt.Println("-- WARNING: The next statement will shorten a character varying column.")
138+
fmt.Println("-- WARNING: The next statement will shorten a character varying column, which may result in data loss.")
136139
}
137140
fmt.Printf("ALTER TABLE %s ALTER COLUMN %s TYPE character varying(%s);\n", c.get("table_name"), c.get("column_name"), max1)
138141
}
@@ -190,8 +193,7 @@ SELECT table_name
190193
FROM information_schema.columns
191194
WHERE table_schema = 'public'
192195
AND is_updatable = 'YES'
193-
-- We do not depend on this sorting correctly
194-
ORDER BY table_name, column_name COLLATE "C" ASC;`
196+
ORDER BY table_name, column_name;`
195197

196198
rowChan1, _ := pgutil.QueryStrings(conn1, sql)
197199
rowChan2, _ := pgutil.QueryStrings(conn2, sql)

function.go

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
//
2+
// Copyright (c) 2016 Jon Carlson. All rights reserved.
3+
// Use of this source code is governed by an MIT-style
4+
// license that can be found in the LICENSE file.
5+
//
6+
7+
package main
8+
9+
import "fmt"
10+
import "sort"
11+
import "database/sql"
12+
import "github.com/joncrlsn/pgutil"
13+
import "github.com/joncrlsn/misc"
14+
15+
// ==================================
16+
// FunctionRows definition
17+
// ==================================
18+
19+
// FunctionRows is a sortable slice of string maps
20+
type FunctionRows []map[string]string
21+
22+
func (slice FunctionRows) Len() int {
23+
return len(slice)
24+
}
25+
26+
func (slice FunctionRows) Less(i, j int) bool {
27+
return slice[i]["function_name"] < slice[j]["function_name"]
28+
}
29+
30+
func (slice FunctionRows) Swap(i, j int) {
31+
slice[i], slice[j] = slice[j], slice[i]
32+
}
33+
34+
// FunctionSchema holds a channel streaming function information from one of the databases as well as
35+
// a reference to the current row of data we're viewing.
36+
//
37+
// FunctionSchema implements the Schema interface defined in pgdiff.go
38+
type FunctionSchema struct {
39+
rows FunctionRows
40+
rowNum int
41+
done bool
42+
}
43+
44+
// get returns the value from the current row for the given key
45+
func (c *FunctionSchema) get(key string) string {
46+
if c.rowNum >= len(c.rows) {
47+
return ""
48+
}
49+
return c.rows[c.rowNum][key]
50+
}
51+
52+
// NextRow increments the rowNum and tells you whether or not there are more
53+
func (c *FunctionSchema) NextRow() bool {
54+
if c.rowNum >= len(c.rows)-1 {
55+
c.done = true
56+
}
57+
c.rowNum = c.rowNum + 1
58+
return !c.done
59+
}
60+
61+
// Compare tells you, in one pass, whether or not the first row matches, is less than, or greater than the second row
62+
func (c *FunctionSchema) Compare(obj interface{}) int {
63+
c2, ok := obj.(*FunctionSchema)
64+
if !ok {
65+
fmt.Println("Error!!!, Compare(obj) needs a FunctionSchema instance", c2)
66+
return +999
67+
}
68+
69+
val := misc.CompareStrings(c.get("function_name"), c2.get("function_name"))
70+
//fmt.Printf("-- Compared %v: %s with %s \n", val, c.get("function_name"), c2.get("function_name"))
71+
return val
72+
}
73+
74+
// Add returns SQL to create the function
75+
func (c FunctionSchema) Add() {
76+
fmt.Println("-- STATEMENT-BEGIN")
77+
fmt.Println(c.get("definition"))
78+
fmt.Println("-- STATEMENT-END")
79+
}
80+
81+
// Drop returns SQL to drop the function
82+
func (c FunctionSchema) Drop() {
83+
fmt.Println("-- Note that CASCADE in the statement below will also drop any triggers depending on this function.")
84+
fmt.Println("-- Also, if there are two functions with this name, you will need to add arguments to identify the correct one to drop.")
85+
fmt.Println("-- (See http://www.postgresql.org/docs/9.4/interactive/sql-dropfunction.html) ")
86+
fmt.Printf("DROP FUNCTION %s CASCADE;\n", c.get("function_name"))
87+
}
88+
89+
// Change handles the case where the function names match, but the definition does not
90+
func (c FunctionSchema) Change(obj interface{}) {
91+
c2, ok := obj.(*FunctionSchema)
92+
if !ok {
93+
fmt.Println("Error!!!, Change needs a FunctionSchema instance", c2)
94+
}
95+
if c.get("definition") != c2.get("definition") {
96+
fmt.Println("-- This function is different so we'll recreate it:")
97+
// The definition column has everything needed to rebuild the function
98+
fmt.Println("-- STATEMENT-BEGIN")
99+
fmt.Println(c.get("definition"))
100+
fmt.Println("-- STATEMENT-END")
101+
}
102+
}
103+
104+
// compareFunctions outputs SQL to make the functions match between DBs
105+
func compareFunctions(conn1 *sql.DB, conn2 *sql.DB) {
106+
sql := `
107+
SELECT p.oid::regprocedure AS function_name
108+
, t.typname AS return_type
109+
, pg_get_functiondef(p.oid) AS definition
110+
FROM pg_proc AS p
111+
JOIN pg_type t ON (p.prorettype = t.oid)
112+
JOIN pg_namespace n ON (n.oid = p.pronamespace)
113+
JOIN pg_language l ON (p.prolang = l.oid AND l.lanname IN ('c','plpgsql', 'sql'))
114+
WHERE n.nspname = 'public';
115+
`
116+
117+
rowChan1, _ := pgutil.QueryStrings(conn1, sql)
118+
rowChan2, _ := pgutil.QueryStrings(conn2, sql)
119+
120+
rows1 := make(FunctionRows, 0)
121+
for row := range rowChan1 {
122+
rows1 = append(rows1, row)
123+
}
124+
sort.Sort(rows1)
125+
126+
rows2 := make(FunctionRows, 0)
127+
for row := range rowChan2 {
128+
rows2 = append(rows2, row)
129+
}
130+
sort.Sort(rows2)
131+
132+
// We must explicitly type this as Schema here
133+
var schema1 Schema = &FunctionSchema{rows: rows1, rowNum: -1}
134+
var schema2 Schema = &FunctionSchema{rows: rows2, rowNum: -1}
135+
136+
// Compare the functions
137+
doDiff(schema1, schema2)
138+
}

0 commit comments

Comments
 (0)