hadoop集成kerberos错误排查-Failed to find any Kerberos tgt

hdfs分发完keytab然后启动，发现报错
hdfs GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

[hadoop@hadoop167 conf]$ kinit -k -t /opt/beh/core/hadoop/etc/hadoop/hadoop.keytab hadoop/hadoop167@BONC[hadoop@hadoop167 conf]$ klistTicket cache: KEYRING:persistent:1002:krb_ccache_cV004GdDefault principal: hadoop/hadoop167@BONCValid starting       Expires              Service principal2017-08-31T15:25:00  2017-09-01T15:25:00  krbtgt/BONC@BONC[hadoop@hadoop167 conf]$ hadoop fs -ls /Java config name: nullNative config name: /etc/krb5.confLoaded from native config>>>KinitOptions cache name is /tmp/krb5cc_100217/08/31 15:25:19 WARN security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]17/08/31 15:25:19 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]17/08/31 15:25:19 WARN security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]17/08/31 15:25:19 INFO retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over hadoop166/172.16.31.166:9000 after 1 fail over attempts. Trying to fail over after sleeping for 1350ms.java.net.ConnectException: Call From hadoop167/172.16.31.167 to hadoop166:9000 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)

经过各种google查找，没有发现严格因果关系的解决方案，

最后通过和文档配置详细对比，并与网上经典的配置进行对比，发现krd5.conf的配置中cache的格式有所不同，本来觉得这应该影响不大。
KEYRING格式的cache，kerberos日志输出只有cache name一行。

将配置文件里的default_ccache_name注掉后，还需要使用kdestroy清除缓存。

[logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log[libdefaults] dns_lookup_realm = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true rdns = false default_realm = BONC# default_ccache_name = KEYRING:persistent:%{uid}[realms]# EXAMPLE.COM = {#  kdc = kerberos.example.com#  admin_server = kerberos.example.com# } BONC = {  kdc = hadoop165  admin_server = hadoop165 }[domain_realm] .example.com = BONC

再次执行，发现问题

[hadoop@hadoop165 security]$ hadoop fs -ls /Java config name: nullNative config name: /etc/krb5.confLoaded from native config>>>KinitOptions cache name is /tmp/krb5cc_1000>>>DEBUG <CCacheInputStream>  client principal is hadoop/hadoop165@BONC>>>DEBUG <CCacheInputStream> server principal is krbtgt/BONC@BONC>>>DEBUG <CCacheInputStream> key type: 16>>>DEBUG <CCacheInputStream> auth time: Thu Aug 31 15:25:43 CST 2017>>>DEBUG <CCacheInputStream> start time: Thu Aug 31 15:25:43 CST 2017>>>DEBUG <CCacheInputStream> end time: Fri Sep 01 15:25:43 CST 2017>>>DEBUG <CCacheInputStream> renew_till time: null>>> CCacheInputStream: readFlags()  FORWARDABLE; INITIAL;>>>DEBUG <CCacheInputStream>  client principal is hadoop/hadoop165@BONC>>>DEBUG <CCacheInputStream> server principal is X-CACHECONF:/krb5_ccache_conf_data/fast_avail/krbtgt/BONC@BONC@BONC>>>DEBUG <CCacheInputStream> key type: 0>>>DEBUG <CCacheInputStream> auth time: Thu Jan 01 08:00:00 CST 1970>>>DEBUG <CCacheInputStream> start time: null>>>DEBUG <CCacheInputStream> end time: Thu Jan 01 08:00:00 CST 1970>>>DEBUG <CCacheInputStream> renew_till time: null>>> CCacheInputStream: readFlags() Found ticket for hadoop/hadoop165@BONC to go to krbtgt/BONC@BONC expiring on Fri Sep 01 15:25:43 CST 2017Entered Krb5Context.initSecContext with state=STATE_NEWFound ticket for hadoop/hadoop165@BONC to go to krbtgt/BONC@BONC expiring on Fri Sep 01 15:25:43 CST 2017Service ticket not found in the subject>>> Credentials acquireServiceCreds: same realmUsing builtin default etypes for default_tgs_enctypesdefault etypes for default_tgs_enctypes: 18 17 16 23.>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType>>> KdcAccessibility: reset>>> KrbKdcReq send: kdc=hadoop165 UDP:88, timeout=30000, number of retries =3, #bytes=635>>> KDCCommunication: kdc=hadoop165 UDP:88, timeout=30000,Attempt =1, #bytes=635>>> KrbKdcReq send: #bytes read=638>>> KdcAccessibility: remove hadoop165>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType>>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdETypeKrb5Context setting mySeqNumber to: 799966873Created InitSecContextToken:0000: 01 00 6E 82 02 43 30 82   02 3F A0 03 02 01 05 A1  ..n..C0..?......0010: 03 02 01 0E A2 07 03 05   00 20 00 00 00 A3 82 01  ......... ......0020: 53 61 82 01 4F 30 82 01   4B A0 03 02 01 05 A1 06  Sa..O0..K.......0030: 1B 04 42 4F 4E 43 A2 1E   30 1C A0 03 02 01 00 A1  ..BONC..0.......0040: 15 30 13 1B 06 68 61 64   6F 6F 70 1B 09 68 61 64  .0...hadoop..had0050: 6F 6F 70 31 36 35 A3 82   01 1A 30 82 01 16 A0 03  oop165....0.....0060: 02 01 10 A1 03 02 01 03   A2 82 01 08 04 82 01 04  ................0070: 80 2A E7 47 59 C9 2E C1   69 14 8A 2C 2A 4E 54 C5  .*.GY...i..,*NT.0080: 2D 2F DA D5 B1 83 41 40   74 68 B1 2E 71 5C D3 72  -/....A@th..q\.r0090: DF 49 EE D6 BA 2E 1B 7D   BC F0 64 3D 60 8C C1 4A  .I........d=`..J00A0: 46 70 89 25 BB 5A 41 61   00 0A BC B4 EB DF C7 80  Fp.%.ZAa........00B0: 58 07 64 D1 37 AA 7C 7A   47 1C 9F B5 E0 C9 E2 B5  X.d.7..zG.......00C0: 18 A4 4C 9E E1 F1 21 B2   55 F0 74 72 C1 11 F5 06  ..L...!.U.tr....00D0: B1 01 6B 32 5B AC 4D AB   26 33 BC F6 EA 58 95 7D  ..k2[.M.&3...X..00E0: 13 20 EE CD 6D A7 B2 D0   CC 34 3A F1 AE 74 A4 67  . ..m....4:..t.g00F0: 4B 28 19 A7 8D 17 27 2F   2C 57 A5 CF 0B 13 45 70  K(....'/,W....Ep0100: C8 FA 93 05 2B 37 11 5D   C0 9A 48 1F 0F A1 02 99  ....+7.]..H.....0110: 3D B5 09 1B F9 01 5A F7   48 1F 3A 1B 04 03 5B D0  =.....Z.H.:...[.0120: 29 7F 2E 94 F9 DB 48 8F   E7 9E 6F ED 89 73 CA B7  ).....H...o..s..0130: 36 DB 80 2A B0 3E 4C 19   86 04 5B BD 84 D7 FB 66  6..*.>L...[....f0140: 3B 2C EC DE F6 2B 77 20   F6 5D 79 FD 89 46 92 48  ;,...+w .]y..F.H0150: B3 84 05 EB 03 39 32 9D   ED 57 E3 EA B4 45 9D 82  .....92..W...E..0160: 15 8F A7 9B F0 14 9C A5   A4 B4 61 BF 1D 1C A1 5D  ..........a....]0170: D3 AF 75 F3 A4 81 D2 30   81 CF A0 03 02 01 10 A2  ..u....0........0180: 81 C7 04 81 C4 BB 96 A4   05 8E 00 A2 75 D3 27 F5  ............u.'.0190: DA FA 23 9F A0 5F 42 19   46 E7 50 6E 80 AE D8 FD  ..#.._B.F.Pn....01A0: 74 8D 23 76 5B F0 CD 83   40 6D 97 43 B5 79 6D 72  t.#v[[e-mail]01B0: 6E 55 42 91 98 22 93 C0   00 62 59 72 DD 38 85 83  nUB.."...bYr.8..01C0: 6F B7 E4 A2 95 E0 D8 58   77 8B D6 F6 58 4D 67 2B  o......Xw...XMg+01D0: A0 C4 C1 74 23 23 E5 38   BE 07 8C B9 D7 D8 3E BB  ...t##.8......>.01E0: 50 75 16 8A 08 53 58 BA   71 C1 ED 8D 67 D2 53 95  Pu...SX.q...g.S.01F0: CF 69 A6 BE B9 8A 89 62   D3 5A 21 81 21 F1 FA B1  .i.....b.Z!.!...0200: 2F F2 19 BE E4 9A 6D C7   16 41 07 79 20 6D AA F3  /.....m..A.y m..0210: 11 87 25 73 54 7E 2A E1   F3 93 29 D3 87 FB CF CA  ..%sT.*...).....0220: B8 B0 11 7B 0C 58 99 73   40 29 41 C9 2B E6 D7 69  .....X.s@)A.+..i0230: EF 45 31 BC FB 1B 79 D4   0C 76 93 46 97 E4 DB BA  .E1...y..v.F....0240: 6C EA 38 62 34 22 7C BF   88                       l.8b4"...Entered Krb5Context.initSecContext with state=STATE_IN_PROCESS>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdETypeKrb5Context setting peerSeqNumber to: 888143725Krb5Context.unwrap: token=[60 3f 06 09 2a 86 48 86 f7 12 01 02 02 02 01 04 00 ff ff ff ff 2c 83 fd 36 0e 37 46 3a 66 65 93 3f 45 13 d6 af 61 22 f8 83 f1 d7 46 d2 be 3e 84 72 e0 f4 b1 7d f3 7a 8c e8 01 01 00 00 04 04 04 04 ]Krb5Context.unwrap: data=[01 01 00 00 ]Krb5Context.wrap: data=[01 01 00 00 ]Krb5Context.wrap: token=[60 3f 06 09 2a 86 48 86 f7 12 01 02 02 02 01 04 00 ff ff ff ff 0a eb 94 41 5c ac ec 0f e8 e7 91 9c e5 da 95 e0 64 5d 85 19 4f 2e ad 4b ac 0f b9 2a a2 12 68 2b fc 92 d3 40 01 01 00 00 04 04 04 04 ]

又查了两个小时，为什么没数据呢，原来就是没数据，那个Service ticket not found in the subject的报错根本就是个假象。

曾经怀疑过jdk1.8版本的问题，结果竟然是一个小配置的问题。

觉得没这么简单，就一个default_ccache_name的配置就挡了俺一天，一定另有应请，后续再查。