Greenplum 增加standby以及master-standby切换

master:db-192-168-101-115

pg_ctl stop -D /data01/gpadmin/gpdata/master/gpseg-1/       //模拟故障

standby:db-192-168-101-116

gpactivatestandby -d /data01/gpadmin/gpdata/master/gpseg-1   //激活standby

以下是激活日志

20170328:15:15:53:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:------------------------------------------------------

20170328:15:15:53:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Standby data directory    = /data01/gpadmin/gpdata/master/gpseg-1

20170328:15:15:53:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Standby port              = 1922

20170328:15:15:53:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Standby running           = yes

20170328:15:15:53:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Force standby activation  = no

20170328:15:15:53:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:------------------------------------------------------

Do you want to continue with standby master activation? Yy|Nn (default=N):

> y

20170328:15:15:59:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-found standby postmaster process

20170328:15:15:59:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Updating transaction files filespace flat files...

20170328:15:15:59:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Updating temporary files filespace flat files...

20170328:15:15:59:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Promoting standby...

20170328:15:15:59:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[DEBUG]:-Waiting for connection...

20170328:15:16:00:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Standby master is promoted

20170328:15:16:00:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Reading current configuration...

20170328:15:16:00:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[DEBUG]:-Connecting to dbname='template1'

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Writing the gp_dbid file - /data01/gpadmin/gpdata/master/gpseg-1/gp_dbid...

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-But found an already existing file.

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Hence removed that existing file.

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Creating a new file...

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Wrote dbid: 1 to the file.

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Now marking it as read only...

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Verifying the file...

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:------------------------------------------------------

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-The activation of the standby master has completed successfully.

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-db-192-168-101-116 is now the new primary master.

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-You will need to update your user access mechanism to reflect

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-the change of master hostname.

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Do not re-start the failed master while the fail-over master is

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-operational, this could result in database corruption!

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-MASTER_DATA_DIRECTORY is now /data01/gpadmin/gpdata/master/gpseg-1 if

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-this has changed as a result of the standby master activation, remember

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-to change this in any startup scripts etc, that may be configured

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-to set this value.

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-MASTER_PORT is now 1922, if this has changed, you

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-may need to make additional configuration changes to allow access

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-to the Greenplum instance.

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Refer to the Administrator Guide for instructions on how to re-activate

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-the master to its previous state once it becomes available.

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-Query planner statistics must be updated on all databases

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-following standby master activation.

20170328:15:16:01:008793 gpactivatestandby:db-192-168-101-116:gpadmin-[INFO]:-When convenient, run ANALYZE against all user databases.

查看配置的standby信息，这里已经看不到了，因为现在standby已经是master，原来的master故障

gpstate -f

20170328:15:19:29:009123 gpstate:db-192-168-101-116:gpadmin-[INFO]:-Starting gpstate with args: -f

20170328:15:19:29:009123 gpstate:db-192-168-101-116:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.12.0 build 1'

20170328:15:19:29:009123 gpstate:db-192-168-101-116:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.12.0 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Feb 27 2017 20:45:12'

20170328:15:19:29:009123 gpstate:db-192-168-101-116:gpadmin-[INFO]:-Obtaining Segment details from master...

20170328:15:19:29:009123 gpstate:db-192-168-101-116:gpadmin-[INFO]:-Standby master instance not configured

20170328:15:19:29:009123 gpstate:db-192-168-101-116:gpadmin-[INFO]:--------------------------------------------------------------

20170328:15:19:29:009123 gpstate:db-192-168-101-116:gpadmin-[INFO]:--pg_stat_replication

20170328:15:19:29:009123 gpstate:db-192-168-101-116:gpadmin-[INFO]:--------------------------------------------------------------

20170328:15:19:29:009123 gpstate:db-192-168-101-116:gpadmin-[INFO]:-No entries found.

20170328:15:19:29:009123 gpstate:db-192-168-101-116:gpadmin-[INFO]:--------------------------------------------------------------

添加standby：

gpinitstandby -s db-192-168-101-115.sky-mobi.com 

注意这里添加的时候，会check master的数据库目录，要把原master节点的的目录删除或者重命名

cd /data01/gpadmin/gpdata/master_hank

mv gpseg-1 gpseg-1_bak

cd /data01/gpadmin/gpdata/master

mv gpseg-1 gpseg-1_bak

The filespace locations on the master must be mapped to

locations on the standby.  These locations must be empty on the

standby master host.  The default provided is the location of

the filespace on the master (except if the master and the

standby are hosted on the same node or host). In most cases the

defaults can be used.

Enter standby filespace location for filespace hank (default: /data01/gpadmin/gpdata/master_hank/gpseg-1):

> 

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Validating environment and parameters for standby initialization...

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Checking for filespace directory /data01/gpadmin/gpdata/master/gpseg-1 on db-192-168-101-115.sky-mobi.com

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Checking for filespace directory /data01/gpadmin/gpdata/master_hank/gpseg-1 on db-192-168-101-115.sky-mobi.com

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:------------------------------------------------------

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Greenplum standby master initialization parameters

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:------------------------------------------------------

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Greenplum master hostname               = db-192-168-101-116.sky-mobi.com

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Greenplum master data directory         = /data01/gpadmin/gpdata/master/gpseg-1

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Greenplum master port                   = 1922

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Greenplum standby master hostname       = db-192-168-101-115.sky-mobi.com

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Greenplum standby master port           = 1922

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Greenplum standby master data directory = /data01/gpadmin/gpdata/master/gpseg-1

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Greenplum update system catalog         = On

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:------------------------------------------------------

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:- Filespace locations

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:------------------------------------------------------

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-pg_system -> /data01/gpadmin/gpdata/master/gpseg-1

20170328:15:33:11:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-hank -> /data01/gpadmin/gpdata/master_hank/gpseg-1

Do you want to continue with standby master initialization? Yy|Nn (default=N):

> y

20170328:15:33:18:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Syncing Greenplum Database extensions to standby

20170328:15:33:18:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-The packages on db-192-168-101-115.sky-mobi.com are consistent.

20170328:15:33:18:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Adding standby master to catalog...

20170328:15:33:18:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Database catalog updated successfully.

20170328:15:33:19:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Updating pg_hba.conf file...

20170328:15:33:25:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-pg_hba.conf files updated successfully.

20170328:15:33:26:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Updating filespace flat files...

20170328:15:33:26:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Filespace flat file updated successfully.

20170328:15:33:26:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Starting standby master

20170328:15:33:26:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Checking if standby master is running on host: db-192-168-101-115.sky-mobi.com  in directory: /data01/gpadmin/gpdata/master/gpseg-1

20170328:15:33:29:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Cleaning up pg_hba.conf backup files...

20170328:15:33:35:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Backup files of pg_hba.conf cleaned up successfully.

20170328:15:33:35:009995 gpinitstandby:db-192-168-101-116:gpadmin-[INFO]:-Successfully created standby master on db-192-168-101-115.sky-mobi.com

命令查看已经添加成功，原来的master现在是standby

gpstate -f

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:-Starting gpstate with args: -f

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.12.0 build 1'

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.12.0 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Feb 27 2017 20:45:12'

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:-Obtaining Segment details from master...

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:-Standby master details

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:-----------------------

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:-   Standby address          = db-192-168-101-115.sky-mobi.com

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:-   Standby data directory   = /data01/gpadmin/gpdata/master/gpseg-1

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:-   Standby port             = 1922

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:-   Standby PID              = 11836

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:-   Standby status           = Standby host passive

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:--------------------------------------------------------------

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:--pg_stat_replication

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:--------------------------------------------------------------

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:--WAL Sender State: streaming

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:--Sync state: sync

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:--Sent Location: 0/14000000

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:--Flush Location: 0/14000000

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:--Replay Location: 0/14000000

20170328:15:34:36:010250 gpstate:db-192-168-101-116:gpadmin-[INFO]:--------------------------------------------------------------

切换回原来的master

现在的master：db-192-168-101-116

pg_ctl stop -D /data01/gpadmin/gpdata/master/gpseg-1/    //停服务

重命名目录：

cd /data01/gpadmin/gpdata/master_hank

mv gpseg-1 gpseg-1_bak

cd /data01/gpadmin/gpdata/master

mv gpseg-1 gpseg-1_bak

现在的standby：db-192-168-101-115

gpactivatestandby -d /data01/gpadmin/gpdata/master/gpseg-1   //激活standby

gpinitstandby  -s db-192-168-101-116.sky-mobi.com   //添加standby

参考：

http://gpdb.docs.pivotal.io/43120/admin_guide/highavail/topics/g-recovering-a-failed-master.html