/u01/app/11.2.0/grid/lib/libclntsh.so.11.1: file too short

节点2：升级卡死，关机重启后：
[root@rac2 ~]# crsctl starat crs
/u01/app/11.2.0/grid/bin/crsctl.bin: error while loading shared libraries: /u01/app/11.2.0/grid/lib/libclntsh.so.11.1: file too short
[root@rac2 ~]# 
文件查看：文件缺失：
[root@rac2 ~]# ll /u01/app/11.2.0/grid/lib/libclntsh.so.11.1
-rw-r--r-- 1 grid oinstall 0 Sep 10 14:49 /u01/app/11.2.0/grid/lib/libclntsh.so.11.1


节点1：
rac1->  ll   /u01/app/11.2.0/grid/lib/libclntsh.so.11.1
-rwxr-xr-x 1 grid oinstall 40028734 Sep  6 04:28 /u01/app/11.2.0/grid/lib/libclntsh.so.11.1
rac1-> scp     /u01/app/11.2.0/grid/lib/libclntsh.so.11.1 rac2:/u01/app/11.2.0/grid/lib/libclntsh.so.11.1
libclntsh.so.11.1                                                                                                         100%   38MB  12.7MB/s   00:03    


节点2：
rac2-> ./opatch
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Syntax Error... No command specified.


 Usage: opatch [ -help ] [ -report ] [ command ]


            command := apply
                       lsinventory
                       lspatches
                       napply
                       nrollback
                       rollback
                       query
                       version 
                       prereq
                       util
 
 <global_arguments> := -help       Displays the help message for the command.
                       -report     Print the actions without executing.


 example:
   'opatch -help'
   'opatch -help -fmw'
   'opatch auto -help'
   'opatch apply -help'
   'opatch lsinventory -help'
   'opatch lspatches -help'
   'opatch napply -help'
   'opatch nrollback -help'
   'opatch rollback -help'
   'opatch prereq -help'
   'opatch util -help'




OPatch failed with error code 14
rac2-> ./opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.




Oracle Home       : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.1.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2014-09-12_14-06-33PM_1.log


Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2014-09-12_14-06-33PM.txt


--------------------------------------------------------------------------------
Installed Top-level Products (1): 


Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 product(s) installed in this Oracle Home.




Interim patches (2) :


Patch  9654983      : applied on Wed Sep 10 14:49:11 CST 2014
Unique Patch ID:  12651759
   Created on 17 Jun 2010, 03:31:21 hrs PST8PDT
   Bugs fixed:
     9068088, 9363384, 8865718, 8898852, 8801119, 9054253, 8725286, 8974548
     9093300, 8909984, 8755082, 8780372, 8664189, 8769569, 7519406, 8822531
     7705591, 8650719, 9637033, 8639114, 8723477, 8729793, 8919682, 8856478
     9001453, 8733749, 8565708, 8735201, 8684517, 8870559, 8773383, 8981059
     8812705, 9488887, 8813366, 9242411, 8822832, 8897784, 8760714, 8775569
     8671349, 8898589, 9714832, 8642202, 9011088, 9170608, 9369797, 9165206
     8834636, 8891037, 8431487, 8570322, 8685253, 8872096, 8718952, 8799099
     9032717, 9399090, 9546223, 9713537, 8588519, 8783738, 8834425, 9454385
     8856497, 8890026, 8721315, 8818175, 8674263, 9145541, 8720447, 9272086
     9467635, 9010222, 9197917, 8991997, 8661168, 8803762, 8769239, 9654983
     8706590, 8778277, 8815639, 9027691, 9454036, 9454037, 9454038, 9255542
     8761974, 9275072, 8496830, 8702892, 8818983, 8475069, 8875671, 9328668
     8798317, 8891929, 8774868, 8820324, 8544696, 8702535, 8268775, 9036013
     9363145, 8933870, 8405205, 9467727, 8822365, 9676419, 8761260, 8790767
     8795418, 8913269, 8717461, 8607693, 8861700, 8330783, 8780281, 8780711
     8784929, 9341448, 9015983, 9119194, 8828328, 8665189, 8717031, 8832205
     9676420, 8633358, 9321701, 9655013, 8796511, 9167285, 8782971, 8756598
     8703064, 9066116, 9007102, 9461782, 9352237, 8505803, 8753903, 9216806
     8918433, 9057443, 8790561, 8733225, 9067282, 8928276, 9210925, 8837736


Patch  9655006      : applied on Wed Sep 10 14:45:41 CST 2014
Unique Patch ID:  12651759
   Created on 6 Jul 2010, 12:02:48 hrs PST8PDT
   Bugs fixed:
     9655006, 9778840, 9343627, 9783609, 9262748, 9262722


Rac system comprising of multiple nodes
  Local node = rac2
  Remote node = rac1


--------------------------------------------------------------------------------


OPatch succeeded.


升级基本成功：
[root@rac2 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac2 ~]# crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.


[root@rac2 ~]# ps -ef|grep grid
root      5427     1  4 14:09 ?        00:00:03 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
grid      5566     1  1 14:10 ?        00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
grid      5580     1  0 14:10 ?        00:00:00 /u01/app/11.2.0/grid/bin/gipcd.bin
grid      5589     1  0 14:10 ?        00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
grid      5603     1  1 14:10 ?        00:00:00 /u01/app/11.2.0/grid/bin/gpnpd.bin
root      5617     1  5 14:10 ?        00:00:02 /u01/app/11.2.0/grid/bin/cssdmonitor
root      5639     1  6 14:10 ?        00:00:02 /u01/app/11.2.0/grid/bin/cssdagent
root      5641     1  0 14:10 ?        00:00:00 /u01/app/11.2.0/grid/bin/orarootagent.bin
grid      5675     1  5 14:10 ?        00:00:02 /u01/app/11.2.0/grid/bin/ocssd.bin 
grid      5678     1  0 14:10 ?        00:00:00 /u01/app/11.2.0/grid/bin/diskmon.bin -d -f
root      5749     1  1 14:10 ?        00:00:00 /u01/app/11.2.0/grid/bin/octssd.bin reboot
grid      5767     1  1 14:10 ?        00:00:00 /u01/app/11.2.0/grid/bin/evmd.bin
grid      5774     1 32 14:10 ?        00:00:05 oracle+ASM2 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
grid      5837     1  2 14:10 ?        00:00:00 asm_pmon_+ASM2
grid      5841     1  1 14:10 ?        00:00:00 asm_vktm_+ASM2
grid      5847     1  1 14:10 ?        00:00:00 asm_gen0_+ASM2
grid      5851     1  3 14:10 ?        00:00:00 asm_diag_+ASM2
grid      5855     1  1 14:10 ?        00:00:00 asm_ping_+ASM2
grid      5862     1  2 14:10 ?        00:00:00 asm_psp0_+ASM2
grid      5868     1  5 14:10 ?        00:00:00 asm_dia0_+ASM2
grid      5872     1  5 14:10 ?        00:00:00 asm_lmon_+ASM2
grid      5876     1  3 14:10 ?        00:00:00 asm_lmd0_+ASM2
grid      5881     1  7 14:10 ?        00:00:00 asm_lms0_+ASM2
grid      5888     1  2 14:10 ?        00:00:00 asm_lmhb_+ASM2
grid      5892     1  3 14:10 ?        00:00:00 asm_mman_+ASM2
grid      5899     1  4 14:10 ?        00:00:00 asm_dbw0_+ASM2
grid      5915     1  5 14:10 ?        00:00:00 asm_lgwr_+ASM2
grid      5919     1  5 14:10 ?        00:00:00 asm_ckpt_+ASM2
grid      5923     1  5 14:10 ?        00:00:00 asm_smon_+ASM2
grid      5927     1  5 14:10 ?        00:00:00 asm_rbal_+ASM2
grid      5932     1  5 14:10 ?        00:00:00 asm_gmon_+ASM2
grid      5936     1  4 14:10 ?        00:00:00 asm_mmon_+ASM2
grid      5940     1  4 14:10 ?        00:00:00 asm_mmnl_+ASM2
grid      5944     1 12 14:10 ?        00:00:00 /u01/app/11.2.0/grid/bin/oclskd.bin
grid      5948     1 15 14:10 ?        00:00:00 asm_lck0_+ASM2
root      5953  4564  0 14:10 pts/1    00:00:00 grep grid
[root@rac2 ~]# 

启动成功