hadoop集成kerberos错误排查-core dump

最近研究kerberos，刚开始安装就遇到了个棘手的问题，core dump了。
好些年不写c程序，这东西还真有点陌生。

[root@hadoop165 ~]# service krb5kdc startRedirecting to /bin/systemctl start  krb5kdc.serviceJob for krb5kdc.service failed because a fatal signal was delivered causing the control process to dump core. See "systemctl status krb5kdc.service" and "journalctl -xe" for details.[root@hadoop165 ~]# systemctl status krb5kdc.service● krb5kdc.service - Kerberos 5 KDC   Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; enabled; vendor preset: disabled)   Active: failed (Result: core-dump) since 一 2017-08-28 16:31:20 CST; 4s ago  Process: 792 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid $KRB5KDC_ARGS (code=dumped, signal=SEGV) Main PID: 9146 (code=dumped, signal=SEGV)8月 28 16:31:08 hadoop165 systemd[1]: Starting Kerberos 5 KDC...8月 28 16:31:20 hadoop165 systemd[1]: krb5kdc.service: control process exited, code=dumped status=118月 28 16:31:20 hadoop165 systemd[1]: Failed to start Kerberos 5 KDC.8月 28 16:31:20 hadoop165 systemd[1]: Unit krb5kdc.service entered failed state.8月 28 16:31:20 hadoop165 systemd[1]: krb5kdc.service failed.

根据提示又查看了下journalctl -xe。
发现一连串磁盘空间满的报错。
df查看还真是满了。

[root@hadoop165 sbin]# df文件系统                   1K-块     已用     可用 已用% 挂载点/dev/mapper/centos-root 52464612 52464592       20  100% /devtmpfs                 3995156        0  3995156    0% /devtmpfs                    4005608        0  4005608    0% /dev/shmtmpfs                    4005608   410992  3594616   11% /runtmpfs                    4005608        0  4005608    0% /sys/fs/cgroup/dev/mapper/centos-home 94919392   247052 94672340    1% /home/dev/sda1                 508588   128924   379664   26% /boottmpfs                     801124        0   801124    0% /run/user/0tmpfs                     801124        0   801124    0% /run/user/1000

清理磁盘，空间剩余42%。
重新测试，还是报原来的错误。
systemctl status的信息没有说明。
继续查看journalctl -xe的报错，根据时间点发现了具体的报错。
这里发现核心报错为db2.so的segment，怀疑是存储出了问题，结合上面的磁盘空间满了的情况怀疑数据库处于一种中间状态导致程序异常。

[root@hadoop165 logs]# journalctl -xe --no-pager8月 29 09:45:09 hadoop165 polkitd[931]: Registered Authentication Agent for unix-process:12871:411771162 (system bus name :1.193646 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale zh_CN.UTF-8)8月 29 09:45:09 hadoop165 systemd[1]: Starting Kerberos 5 KDC...-- Subject: Unit krb5kdc.service has begun start-up-- Defined-By: systemd-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel-- -- Unit krb5kdc.service has begun starting up.8月 29 09:45:10 hadoop165 kernel: krb5kdc[12917]: segfault at 7fd000053166 ip 00007fd0694cafa6 sp 00007ffd92e568e0 error 6 in db2.so[7fd0694bc000+18000]8月 29 09:45:58 hadoop165 abrt-hook-ccpp[12919]: /var/spool/abrt is 3445558898 bytes (more than 1279MiB), deleting 'ccpp-2017-08-29-09:41:58-12318'8月 29 09:45:59 hadoop165 systemd[1]: krb5kdc.service: control process exited, code=dumped status=118月 29 09:45:59 hadoop165 systemd[1]: Failed to start Kerberos 5 KDC.-- Subject: Unit krb5kdc.service has failed-- Defined-By: systemd-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel-- -- Unit krb5kdc.service has failed.-- -- The result is failed.8月 29 09:45:59 hadoop165 systemd[1]: Unit krb5kdc.service entered failed state.8月 29 09:45:59 hadoop165 systemd[1]: krb5kdc.service failed.8月 29 09:45:59 hadoop165 polkitd[931]: Unregistered Authentication Agent for unix-process:12871:411771162 (system bus name :1.193646, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale zh_CN.UTF-8) (disconnected from bus)8月 29 09:46:00 hadoop165 abrt-server[12991]: 正在生成核心转储

将数据库数据清空，并重新生成，测试问题解决。

[root@hadoop165 ~]# cd /var/kerberos/krb5kdc/[root@hadoop165 krb5kdc]# lskadm5.acl  kdc.conf  principal  principal.kadm5  principal.kadm5.lock  principal.ok[root@hadoop165 krb5kdc]# rm principal* -rf[root@hadoop165 krb5kdc]# kdb5_util create -r BONC -sLoading random dataInitializing database '/var/kerberos/krb5kdc/principal' for realm 'BONC',master key name 'K/M@BONC'You will be prompted for the database Master Password.It is important that you NOT FORGET this password.Enter KDC database master key: Re-enter KDC database master key to verify: [root@hadoop165 krb5kdc]#