GPDB的spread方式测试

在GPDB的mirror安装中，有两种方法，默认的group方式是将A机器备份在B机器上，这种方式有一个弊端，在A机器宕机后，B机器上就运行了*2的节点数量的运算和存储，这样势必导致整个集群的运算性能极具下降，甚至不能使用；因此官方也推荐mirror的使用方式为spread，前提是机器的segment host数量为每台机器上的segment+1，举个例子：有4台机器作为seghost，每单机器3个segment,分别abcd机器，abcd每台机器上有seg1 seg2 seg3,对应关系如下

机器名   节点名        mirror机器名    mirror节点名A        seg1        B               seg2A        seg2        C               seg3A        seg3        D               seg1....

以上只是实例，当时实际情况不可能这么规整，具体的对应关系可以使用gpstate -c查看，例如我实际的对应关系如下

20160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-Obtaining Segment details from master...20160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:--------------------------------------------------------------20160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:--Current GPDB mirror list and status20160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:--Type = Spread20160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:--------------------------------------------------------------20160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Status                             Data State     Primary      Datadir                               Port    Mirror       Datadir                               Port20160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   blue02-em1   /home/gpadmin/data/gpdatap1/gpseg0    40000   blue03-em2   /home/gpadmin/data/gpdatam1/gpseg0    5000020160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   blue02-em2   /home/gpadmin/data/gpdatap2/gpseg1    40001   blue04-em3   /home/gpadmin/data/gpdatam2/gpseg1    5000120160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   blue02-em3   /home/gpadmin/data/gpdatap3/gpseg2    40002   blue05-em1   /home/gpadmin/data/gpdatam3/gpseg2    5000220160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   blue03-em1   /home/gpadmin/data/gpdatap1/gpseg3    40000   blue04-em2   /home/gpadmin/data/gpdatam1/gpseg3    5000020160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   blue03-em2   /home/gpadmin/data/gpdatap2/gpseg4    40001   blue05-em3   /home/gpadmin/data/gpdatam2/gpseg4    5000120160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   blue03-em3   /home/gpadmin/data/gpdatap3/gpseg5    40002   blue02-em1   /home/gpadmin/data/gpdatam3/gpseg5    5000220160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   blue04-em1   /home/gpadmin/data/gpdatap1/gpseg6    40000   blue05-em2   /home/gpadmin/data/gpdatam1/gpseg6    5000020160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   blue04-em2   /home/gpadmin/data/gpdatap2/gpseg7    40001   blue02-em3   /home/gpadmin/data/gpdatam2/gpseg7    5000120160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   blue04-em3   /home/gpadmin/data/gpdatap3/gpseg8    40002   blue03-em1   /home/gpadmin/data/gpdatam3/gpseg8    5000220160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   blue05-em1   /home/gpadmin/data/gpdatap1/gpseg9    40000   blue02-em2   /home/gpadmin/data/gpdatam1/gpseg9    5000020160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   blue05-em2   /home/gpadmin/data/gpdatap2/gpseg10   40001   blue03-em3   /home/gpadmin/data/gpdatam2/gpseg10   5000120160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:-   Primary Active, Mirror Available   Synchronized   blue05-em3   /home/gpadmin/data/gpdatap3/gpseg11   40002   blue04-em1   /home/gpadmin/data/gpdatam3/gpseg11   5000220160816:13:59:43:018642 gpstate:blue01-em1:gpadmin-[INFO]:--------------------------------------------------------------

以下我们验证采用spread方式后，在一台机器宕机后，查看mirror的启动及数据分配情况。
1.down机前检查机器情况重点内容

select * from gp_segment_configuration order by 1; dbid | content | role | preferred_role | mode | status | port  |  hostname  |  address   | replication_port | san_mounts ------+---------+------+----------------+------+--------+-------+------------+------------+------------------+------------    1 |      -1 | p    | p              | s    | u      |  5432 | blue01-em1 | blue01-em1 |                  |     2 |       0 | p    | p              | s    | u      | 40000 | blue02-em1 | blue02-em1 |            41000 |     3 |       1 | p    | p              | s    | u      | 40001 | blue02-em1 | blue02-em2 |            41001 |     4 |       2 | p    | p              | s    | u      | 40002 | blue02-em1 | blue02-em3 |            41002 |     5 |       3 | p    | p              | s    | u      | 40000 | blue03-em1 | blue03-em1 |            41000 |     6 |       4 | p    | p              | s    | u      | 40001 | blue03-em1 | blue03-em2 |            41001 |     7 |       5 | p    | p              | s    | u      | 40002 | blue03-em1 | blue03-em3 |            41002 |     8 |       6 | p    | p              | s    | u      | 40000 | blue04-em1 | blue04-em1 |            41000 |     9 |       7 | p    | p              | s    | u      | 40001 | blue04-em1 | blue04-em2 |            41001 |    10 |       8 | p    | p              | s    | u      | 40002 | blue04-em1 | blue04-em3 |            41002 |    11 |       9 | p    | p              | s    | u      | 40000 | blue05-em1 | blue05-em1 |            41000 |    12 |      10 | p    | p              | s    | u      | 40001 | blue05-em1 | blue05-em2 |            41001 |    13 |      11 | p    | p              | s    | u      | 40002 | blue05-em1 | blue05-em3 |            41002 |    14 |       0 | m    | m              | s    | u      | 50000 | blue03-em1 | blue03-em2 |            51000 |    15 |       1 | m    | m              | s    | u      | 50001 | blue04-em1 | blue04-em3 |            51001 |    16 |       2 | m    | m              | s    | u      | 50002 | blue05-em1 | blue05-em1 |            51002 |    17 |       3 | m    | m              | s    | u      | 50000 | blue04-em1 | blue04-em2 |            51000 |    18 |       4 | m    | m              | s    | u      | 50001 | blue05-em1 | blue05-em3 |            51001 |    19 |       5 | m    | m              | s    | u      | 50002 | blue02-em1 | blue02-em1 |            51002 |    20 |       6 | m    | m              | s    | u      | 50000 | blue05-em1 | blue05-em2 |            51000 |    21 |       7 | m    | m              | s    | u      | 50001 | blue02-em1 | blue02-em3 |            51001 |    22 |       8 | m    | m              | s    | u      | 50002 | blue03-em1 | blue03-em1 |            51002 |    23 |       9 | m    | m              | s    | u      | 50000 | blue02-em1 | blue02-em2 |            51000 |    24 |      10 | m    | m              | s    | u      | 50001 | blue03-em1 | blue03-em3 |            51001 |    25 |      11 | m    | m              | s    | u      | 50002 | blue04-em1 | blue04-em1 |            51002 | (25 rows)select gp_segment_id,count(*) from testdata group by 1; gp_segment_id | count ---------------+-------             0 |     1             6 |     2            11 |     1             1 |     2             4 |     2             9 |     1             2 |     1             7 |     1             3 |     1

可以看到虽然我们的表是distributed randomly创建的，但是数据并不是严格按照每个节点一条数据的分布的。

2.kill掉blue05的进程
pkill -9 postgres

3.检查目前的状况

select * from gp_segment_configuration order by 1;dbid | content | role | preferred_role | mode | status | port  |  hostname  |  address   | replication_port | san_mounts ------+---------+------+----------------+------+--------+-------+------------+------------+------------------+------------    1 |      -1 | p    | p              | s    | u      |  5432 | blue01-em1 | blue01-em1 |                  |     2 |       0 | p    | p              | s    | u      | 40000 | blue02-em1 | blue02-em1 |            41000 |     3 |       1 | p    | p              | s    | u      | 40001 | blue02-em1 | blue02-em2 |            41001 |     4 |       2 | p    | p              | c    | u      | 40002 | blue02-em1 | blue02-em3 |            41002 |     5 |       3 | p    | p              | s    | u      | 40000 | blue03-em1 | blue03-em1 |            41000 |     6 |       4 | p    | p              | c    | u      | 40001 | blue03-em1 | blue03-em2 |            41001 |     7 |       5 | p    | p              | s    | u      | 40002 | blue03-em1 | blue03-em3 |            41002 |     8 |       6 | p    | p              | c    | u      | 40000 | blue04-em1 | blue04-em1 |            41000 |     9 |       7 | p    | p              | s    | u      | 40001 | blue04-em1 | blue04-em2 |            41001 |    10 |       8 | p    | p              | s    | u      | 40002 | blue04-em1 | blue04-em3 |            41002 |    11 |       9 | m    | p              | s    | d      | 40000 | blue05-em1 | blue05-em1 |            41000 |    12 |      10 | m    | p              | s    | d      | 40001 | blue05-em1 | blue05-em2 |            41001 |    13 |      11 | m    | p              | s    | d      | 40002 | blue05-em1 | blue05-em3 |            41002 |    14 |       0 | m    | m              | s    | u      | 50000 | blue03-em1 | blue03-em2 |            51000 |    15 |       1 | m    | m              | s    | u      | 50001 | blue04-em1 | blue04-em3 |            51001 |    16 |       2 | m    | m              | s    | d      | 50002 | blue05-em1 | blue05-em1 |            51002 |    17 |       3 | m    | m              | s    | u      | 50000 | blue04-em1 | blue04-em2 |            51000 |    18 |       4 | m    | m              | s    | d      | 50001 | blue05-em1 | blue05-em3 |            51001 |    19 |       5 | m    | m              | s    | u      | 50002 | blue02-em1 | blue02-em1 |            51002 |    20 |       6 | m    | m              | s    | d      | 50000 | blue05-em1 | blue05-em2 |            51000 |    21 |       7 | m    | m              | s    | u      | 50001 | blue02-em1 | blue02-em3 |            51001 |    22 |       8 | m    | m              | s    | u      | 50002 | blue03-em1 | blue03-em1 |            51002 |    23 |       9 | p    | m              | c    | u      | 50000 | blue02-em1 | blue02-em2 |            51000 |    24 |      10 | p    | m              | c    | u      | 50001 | blue03-em1 | blue03-em3 |            51001 |    25 |      11 | p    | m              | c    | u      | 50002 | blue04-em1 | blue04-em1 |            51002 | (25 rows)

以上关键字段含义：
role:该节点现在的角色，取值为p(parimary)或m(mirror)
preferred_role:该节点被定义的角色，取值为p(primary)或m(mirror)
mode：主备同步的状态，取值s(synchronized已同步)、r(resyncing正在同步)、c(change logging不同步)
status:判断节点的值，取值为u(up正常运行)和d(down节点失败)

从以上我们可以看出hostname为blue05-em1的41000、41001、41002的同步端口上的原先为主节点角色已经变更为mirror，并且已经down机（进程信息丢失）

4.后续检查：
我们检查数据是否可以查询查询：

select * from testdate; #可以正常查询数据  insert into testdata select 3,'test'; #可以正常插入select gp_segment_id,count(*) from testdata group by 1; gp_segment_id | count ---------------+-------             6 |     2             1 |     2            11 |     1             9 |     1             4 |     2             0 |     1             7 |     1             3 |     1             2 |     1

可以使用gpstate -s查看mirror情况

20160816:13:36:42:017605 gpstate:blue01-em1:gpadmin-[INFO]:-Starting gpstate with args: -m20160816:13:36:42:017605 gpstate:blue01-em1:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.9.0 build 1'20160816:13:36:42:017605 gpstate:blue01-em1:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.9.0 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Aug  8 2016 05:36:26'20160816:13:36:42:017605 gpstate:blue01-em1:gpadmin-[INFO]:-Obtaining Segment details from master...20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:--------------------------------------------------------------20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:--Current GPDB mirror list and status20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:--Type = Spread20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:--------------------------------------------------------------20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:-   Mirror       Datadir                               Port    Status              Data Status       20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:-   blue03-em2   /home/gpadmin/data/gpdatam1/gpseg0    50000   Passive             Synchronized20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:-   blue04-em3   /home/gpadmin/data/gpdatam2/gpseg1    50001   Passive             Synchronized20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[WARNING]:-blue05-em1   /home/gpadmin/data/gpdatam3/gpseg2    50002   Failed                                <<<<<<<<20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:-   blue04-em2   /home/gpadmin/data/gpdatam1/gpseg3    50000   Passive             Synchronized20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[WARNING]:-blue05-em3   /home/gpadmin/data/gpdatam2/gpseg4    50001   Failed                                <<<<<<<<20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:-   blue02-em1   /home/gpadmin/data/gpdatam3/gpseg5    50002   Passive             Synchronized20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[WARNING]:-blue05-em2   /home/gpadmin/data/gpdatam1/gpseg6    50000   Failed                                <<<<<<<<20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:-   blue02-em3   /home/gpadmin/data/gpdatam2/gpseg7    50001   Passive             Synchronized20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:-   blue03-em1   /home/gpadmin/data/gpdatam3/gpseg8    50002   Passive             Synchronized20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:-   blue02-em2   /home/gpadmin/data/gpdatam1/gpseg9    50000   Acting as Primary   Change Tracking20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:-   blue03-em3   /home/gpadmin/data/gpdatam2/gpseg10   50001   Acting as Primary   Change Tracking20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:-   blue04-em1   /home/gpadmin/data/gpdatam3/gpseg11   50002   Acting as Primary   Change Tracking20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[INFO]:--------------------------------------------------------------20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[WARNING]:-3 segment(s) configured as mirror(s) are acting as primaries20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[WARNING]:-3 segment(s) configured as mirror(s) have failed20160816:13:36:43:017605 gpstate:blue01-em1:gpadmin-[WARNING]:-3 mirror segment(s) acting as primaries are in change tracking

因此，我们可以断定，blue05的机器down机，节点数据已经转移，备用节点已经启用，正常的业务不受影响，而且数据可以正常插入。

5.修复
要恢复到原来的架构，有两种可能情况，第一种情况，当故障主机启动后，这台主机上的节点会自动启动，当然现在已经都是Mirror节点了，角色是不会切换的，
而且数据是不一致状态。这种情况只需要使用gprecoverseg命令同步一下数据后，就可以切换到原来的架构了。
输入gprecoverseg 然后输入Y

Continue with segment recovery procedure Yy|Nn (default=N):> y20160816:13:42:20:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-6 segment(s) to recover20160816:13:42:20:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Ensuring 6 failed segment(s) are stopped20160816:13:42:22:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segmentsupdating flat files20160816:13:42:27:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Updating configuration with new mirrors20160816:13:42:28:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Updating mirrors. 20160816:13:42:29:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Starting mirrors20160816:13:42:29:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait..... 20160816:13:42:31:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Process results...20160816:13:42:31:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Updating configuration to mark mirrors up20160816:13:42:31:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Updating primaries20160816:13:42:31:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Commencing parallel primary conversion of 6 segments, please wait.... 20160816:13:42:32:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Process results...20160816:13:42:32:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Done updating primaries20160816:13:42:32:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-******************************************************************20160816:13:42:32:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Updating segments for resynchronization is completed.20160816:13:42:32:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-For segments updated successfully, resynchronization will continue in the background.20160816:13:42:32:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-20160816:13:42:32:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Use  gpstate -s  to check the resynchronization progress.20160816:13:42:32:017728 gprecoverseg:blue01-em1:gpadmin-[INFO]:-******************************************************************

按照提示，查看节点信息，这里只检查mirror
gpstate -m

20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-Obtaining Segment details from master...20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:--------------------------------------------------------------20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:--Current GPDB mirror list and status20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:--Type = Spread20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:--------------------------------------------------------------20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   Mirror       Datadir                               Port    Status              Data Status    20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   blue03-em2   /home/gpadmin/data/gpdatam1/gpseg0    50000   Passive             Synchronized20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   blue04-em3   /home/gpadmin/data/gpdatam2/gpseg1    50001   Passive             Synchronized20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   blue05-em1   /home/gpadmin/data/gpdatam3/gpseg2    50002   Passive             Synchronized20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   blue04-em2   /home/gpadmin/data/gpdatam1/gpseg3    50000   Passive             Synchronized20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   blue05-em3   /home/gpadmin/data/gpdatam2/gpseg4    50001   Passive             Synchronized20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   blue02-em1   /home/gpadmin/data/gpdatam3/gpseg5    50002   Passive             Synchronized20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   blue05-em2   /home/gpadmin/data/gpdatam1/gpseg6    50000   Passive             Synchronized20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   blue02-em3   /home/gpadmin/data/gpdatam2/gpseg7    50001   Passive             Synchronized20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   blue03-em1   /home/gpadmin/data/gpdatam3/gpseg8    50002   Passive             Synchronized20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   blue02-em2   /home/gpadmin/data/gpdatam1/gpseg9    50000   Acting as Primary   Synchronized20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   blue03-em3   /home/gpadmin/data/gpdatam2/gpseg10   50001   Acting as Primary   Synchronized20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:-   blue04-em1   /home/gpadmin/data/gpdatam3/gpseg11   50002   Acting as Primary   Synchronized20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[INFO]:--------------------------------------------------------------20160816:13:43:27:017884 gpstate:blue01-em1:gpadmin-[WARNING]:-3 segment(s) configured as mirror(s) are acting as primaries

我们看到数据同步已经完成，但是角色还没有转换过来，需要使用gprecoverseg -r
我们看到数据同步已经完成，但是角色还没有转换过来，需要使用gprecoverseg -r进行角色切换

Continue with segment rebalance procedure Yy|Nn (default=N):> y20160816:13:45:24:017935 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Getting unbalanced segments20160816:13:45:24:017935 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Stopping unbalanced primary segments..... 20160816:13:45:26:017935 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Triggering segment reconfiguration20160816:13:45:30:017935 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Starting segment synchronization................. 20160816:13:45:47:017935 gprecoverseg:blue01-em1:gpadmin-[INFO]:-******************************************************************20160816:13:45:47:017935 gprecoverseg:blue01-em1:gpadmin-[INFO]:-The rebalance operation has completed successfully.20160816:13:45:47:017935 gprecoverseg:blue01-em1:gpadmin-[INFO]:-There is a resynchronization running in the background to bring all20160816:13:45:47:017935 gprecoverseg:blue01-em1:gpadmin-[INFO]:-segments in sync.20160816:13:45:47:017935 gprecoverseg:blue01-em1:gpadmin-[INFO]:-20160816:13:45:47:017935 gprecoverseg:blue01-em1:gpadmin-[INFO]:-Use gpstate -e to check the resynchronization progress.20160816:13:45:47:017935 gprecoverseg:blue01-em1:gpadmin-[INFO]:-******************************************************************gpstate -e查看同步进程20160816:13:46:29:018199 gpstate:blue01-em1:gpadmin-[INFO]:-----------------------------------------------------20160816:13:46:29:018199 gpstate:blue01-em1:gpadmin-[INFO]:-Segment Mirroring Status Report20160816:13:46:29:018199 gpstate:blue01-em1:gpadmin-[INFO]:-----------------------------------------------------20160816:13:46:29:018199 gpstate:blue01-em1:gpadmin-[INFO]:-Segment Pairs in Resynchronization20160816:13:46:29:018199 gpstate:blue01-em1:gpadmin-[INFO]:-   Current Primary   Port    Resync mode   Est. resync progress   Total resync objects   Objects to resync   Data synced   Est. total to sync                      Est. resync end time   Change tracking size   Mirror       Port20160816:13:46:29:018199 gpstate:blue01-em1:gpadmin-[INFO]:-   blue05-em1        40000   Incremental   100%                   0                      0                   224 kB        Sync complete; awaiting config change                          128 bytes              blue02-em2   5000020160816:13:46:29:018199 gpstate:blue01-em1:gpadmin-[INFO]:-   blue05-em2        40001   Incremental   100%                   0                      0                   224 kB        Sync complete; awaiting config change                          128 bytes              blue03-em3   5000120160816:13:46:29:018199 gpstate:blue01-em1:gpadmin-[INFO]:-   blue05-em3        40002   Incremental   100%                   0                      0                   224 kB        Sync complete; awaiting config change                          128 bytes              blue04-em1   50002再查看mirror情况20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-Obtaining Segment details from master...20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:--------------------------------------------------------------20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:--Current GPDB mirror list and status20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:--Type = Spread20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:--------------------------------------------------------------20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   Mirror       Datadir                               Port    Status    Data Status    20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   blue03-em2   /home/gpadmin/data/gpdatam1/gpseg0    50000   Passive   Synchronized20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   blue04-em3   /home/gpadmin/data/gpdatam2/gpseg1    50001   Passive   Synchronized20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   blue05-em1   /home/gpadmin/data/gpdatam3/gpseg2    50002   Passive   Synchronized20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   blue04-em2   /home/gpadmin/data/gpdatam1/gpseg3    50000   Passive   Synchronized20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   blue05-em3   /home/gpadmin/data/gpdatam2/gpseg4    50001   Passive   Synchronized20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   blue02-em1   /home/gpadmin/data/gpdatam3/gpseg5    50002   Passive   Synchronized20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   blue05-em2   /home/gpadmin/data/gpdatam1/gpseg6    50000   Passive   Synchronized20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   blue02-em3   /home/gpadmin/data/gpdatam2/gpseg7    50001   Passive   Synchronized20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   blue03-em1   /home/gpadmin/data/gpdatam3/gpseg8    50002   Passive   Synchronized20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   blue02-em2   /home/gpadmin/data/gpdatam1/gpseg9    50000   Passive   Synchronized20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   blue03-em3   /home/gpadmin/data/gpdatam2/gpseg10   50001   Passive   Synchronized20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:-   blue04-em1   /home/gpadmin/data/gpdatam3/gpseg11   50002   Passive   Synchronized20160816:13:47:27:018297 gpstate:blue01-em1:gpadmin-[INFO]:--------------------------------------------------------------

现在查看角色已经转换过来，在用sql验证一下

select * from gp_segment_configuration order by 1; dbid | content | role | preferred_role | mode | status | port  |  hostname  |  address   | replication_port | san_mounts ------+---------+------+----------------+------+--------+-------+------------+------------+------------------+------------    1 |      -1 | p    | p              | s    | u      |  5432 | blue01-em1 | blue01-em1 |                  |     2 |       0 | p    | p              | s    | u      | 40000 | blue02-em1 | blue02-em1 |            41000 |     3 |       1 | p    | p              | s    | u      | 40001 | blue02-em1 | blue02-em2 |            41001 |     4 |       2 | p    | p              | s    | u      | 40002 | blue02-em1 | blue02-em3 |            41002 |     5 |       3 | p    | p              | s    | u      | 40000 | blue03-em1 | blue03-em1 |            41000 |     6 |       4 | p    | p              | s    | u      | 40001 | blue03-em1 | blue03-em2 |            41001 |     7 |       5 | p    | p              | s    | u      | 40002 | blue03-em1 | blue03-em3 |            41002 |     8 |       6 | p    | p              | s    | u      | 40000 | blue04-em1 | blue04-em1 |            41000 |     9 |       7 | p    | p              | s    | u      | 40001 | blue04-em1 | blue04-em2 |            41001 |    10 |       8 | p    | p              | s    | u      | 40002 | blue04-em1 | blue04-em3 |            41002 |    11 |       9 | p    | p              | s    | u      | 40000 | blue05-em1 | blue05-em1 |            41000 |    12 |      10 | p    | p              | s    | u      | 40001 | blue05-em1 | blue05-em2 |            41001 |    13 |      11 | p    | p              | s    | u      | 40002 | blue05-em1 | blue05-em3 |            41002 |    14 |       0 | m    | m              | s    | u      | 50000 | blue03-em1 | blue03-em2 |            51000 |    15 |       1 | m    | m              | s    | u      | 50001 | blue04-em1 | blue04-em3 |            51001 |    16 |       2 | m    | m              | s    | u      | 50002 | blue05-em1 | blue05-em1 |            51002 |    17 |       3 | m    | m              | s    | u      | 50000 | blue04-em1 | blue04-em2 |            51000 |    18 |       4 | m    | m              | s    | u      | 50001 | blue05-em1 | blue05-em3 |            51001 |    19 |       5 | m    | m              | s    | u      | 50002 | blue02-em1 | blue02-em1 |            51002 |    20 |       6 | m    | m              | s    | u      | 50000 | blue05-em1 | blue05-em2 |            51000 |    21 |       7 | m    | m              | s    | u      | 50001 | blue02-em1 | blue02-em3 |            51001 |    22 |       8 | m    | m              | s    | u      | 50002 | blue03-em1 | blue03-em1 |            51002 |    23 |       9 | m    | m              | s    | u      | 50000 | blue02-em1 | blue02-em2 |            51000 |    24 |      10 | m    | m              | s    | u      | 50001 | blue03-em1 | blue03-em3 |            51001 |    25 |      11 | m    | m              | s    | u      | 50002 | blue04-em1 | blue04-em1 |            51002 | 

再检查下之前停掉的blueo5的进程，ps -ef|grep postgres发现64个进程已经启动了。