112041 Rac CTSS service start failed during Gi install on AIX 7.1

记一次同事在IBM Aix 7.1上安装11.2.0.4 GI软件后，在安装RDBMS 前进行precheck时发现CTSS 服务未启动，自检不通过，故特意研究了下前因后果

CTSS概念：

Concept：
Cluster Time Synchronization Service
Cluster node times should be synchronized. With this release, Oracle Clusterware provides Cluster Time Synchronization Service (CTSS), which ensures that there is a
synchronization service in the cluster. If Network Time Protocol (NTP) is not  during cluster configuration, then CTSS is configured to ensure time synchronization
Reference: Oracle? Grid Infrastructure Installation Guide11g Release 2 (11.2) for IBM AIX on POWER Systems (64-Bit)

11gr2 Rac 时钟同步方法

在11gr2 版本的Rac 在安装 GI软件时，需要通过2种途径实现节点间时间同步：

1.操作系统 NTP 时间同步 

2.Oracle Cluster Time Synchronization Service （此次错误的主角）

这2种方式只能选择其一，也就是如果在主机层面配置了NTP ，那Oracle 就会通过NTP 方式来配置节点间的时间同步，反之亦然。

Reference：MOS Note [ID 1054006.1]
If you choose to let CTSSD handle time synchronization, de-configure the vendor time sync service.
For example, for NTP you may need to move or remove /etc/ntp.conf or /etc/xntp.conf.

Problem：
CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.

Active：

1. 分别在2个节点检查CTSS 服务状态

$grid]cluvfy comp clocksync
Verifying Clock Synchronization across the cluster nodes
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...
CTSS resource check passed


Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed

Check CTSS state started...
CTSS is in Observer state. Switching over to clock synchronization checks using NTP
Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
NTP Configuration file check passed

Checking daemon liveness...
Liveness check failed for "ntpd"
Check failed on nodes:
racbde1
PRVF-5415 : Check to see if NTP daemon is running failed
Clock synchronization check using Network Time Protocol(NTP) failed
PRVF-9652 : Cluster Time Synchronization Services check failed
Verification of Clock Synchronization across the cluster nodes was unsuccessful on all the specified nodes.

or

2. 检查$GRID_HOME/log/alert_ados.log 日志发现提示NTP service configure exist ，但是服务其实没有启动的

2009-12-23 20:06:53.974
[ctssd(13443)]CRS-2409:The clock on host racbde2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

3.检查NTP 配置及服务启停

stopsrc -s xntpd 禁用NTP服务（如果未禁用就需要禁用）
startsrc -s xntpd 启动NTP服务

Solution：

1) If you choose to use a vendor time sync service (like ntp), make sure it is configured AND running.

2) If you choose to let CTSSD handle time synchronization, de-configure the vendor time sync service.  (推荐)

For example, for NTP you may need to move or remove /etc/ntp.conf or /etc/xntp.conf.

在两个节点上分别将ntp 配置文件重命名做备份保留

#mv /etc/ntp.conf  /etc/ntp.conf.bak

重新配置CTSS服务

$ ./cluvfy comp clocksync
Verifying Clock Synchronization across the cluster nodes

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...
CTSS resource check passed

Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed
Check CTSS state started...
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Check of clock time offsets passed
Oracle Cluster Time Synchronization Services check passed
Verification of Clock Synchronization across the cluster nodes was successful.

重新检查，CTSS 正常，通过

Reference ：NoteID 1054006.1  CTSSD Runs in Observer Mode Even Though No Time Sync Software is Running