问题描述

在机器上执行/usr/sbin/kadmin.local时卡住，通过 systemctl重启kadmin、kdc server也都会因为超时启动失败。

[root@master_10.0.0.58 /data/emr/krb5]$ journalctl -xe -u kadmin
-- Unit kadmin.service has begun starting up.
Jul 18 11:25:59 10.0.0.58 systemd[1]: kadmin.service start operation timed out. Terminating.
Jul 18 11:25:59 10.0.0.58 systemd[1]: Failed to start Kerberos 5 Password-changing and Administration.
-- Subject: Unit kadmin.service has failed

问题处理

使用journal -xe -u kadmin也没有详细的报错信息，由于kerberos是与openldap集成的，所以顺便看了下openldap的日志，发现有很多这样的错误：

Jul 18 11:31:21 10 slapd[11874]: daemon: accept(8) failed errno=24 (Too many open files)
Jul 18 11:31:21 10 slapd[11874]: daemon: accept(7) failed errno=24 (Too many open files)

先使用ps查看openldap进程的pid，然后查了下它能打开的文件数

[root@master_10.0.0.58 /data/emr/openldap/logs]$ cat /proc/11874/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             127308               127308               processes 
Max open files            1024                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       127308               127308               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us 

然后查看该进程已经打开的文件数

[root@master_10.0.0.58 /data/emr/openldap/logs]$ lsof -p 11874 | wc -l
1073

显而易见，已经超过了Max open files的Soft Limit，需要增加openldap能打开的文件数。在环境变量文件(/usr/lib/systemd/system/slapd.service文件的EnvironmentFile)中添加LDAP_NOFILE=5000 无效。由于systemctl也会限制进程的文件数，需要在/usr/lib/systemd/system/slapd.service的[Service]下添加

LimitNOFILE=5000

然后执行

systemctl daemon-reload
systemctl restart slapd

再次查看openldap进程的limits，发现设置生效，指令也正常了

[root@master_10.0.0.58 /data/emr/openldap/logs]$ cat /proc/557799/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             127308               127308               processes 
Max open files            5000                 5000                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       127308               127308               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us