k8s删除pod一直terminating

最近集群总是遇到某些节点出现pod一直terminating，无法正常删除的状态，查看系统日志

journalctl -xeu kubelet

发现kubelet的异常日志如下
no space left on device

image.png

根据关键字imotify_add_watch，网上介绍说是fs.inotify.max_user_watches的值不够的缘故，查看节点的该值

cat /proc/sys/fs/inotify/max_user_watches

临时调整

sysctl -w fs.inotify.max_user_watches=65536

观察到kubelet确实正常自动，为了避免重启机器配置丢失，将配置写入/etc/sysctl.conf

# 新增配置
fs.inotify.max_user_watches=65536

# 启动配置
sysctl -p

利用ansible操作多台机器

hosts.ini

[kube-all]
192.168.0.1 ansible_ssh_user=ubuntu ansible_ssh_port=22 ansible_ssh_pass="***" ansible_sudo_pass="***"

fix_watch.yaml

---
- hosts: kube-all
  become_user: root
  become: yes
  gather_facts: no

  tasks:
   - name: change watch
     lineinfile:
       path: /etc/sysctl.conf
       line: fs.inotify.max_user_watches=65536

   - name: sysctl -p
     shell: |
       sysctl -p

ansible.cfg

[defaults]
host_key_checking = False
any_errors_fatal = True
timeout = 30
forks = 10

[ssh_connection]
ssh_args=-F ansible_ssh_config
retries=10

ansible_ssh_config

Host *
ForwardAgent no
ControlMaster=auto
ControlPersist=300s

执行命令，即可完成机器的配置调整，上面的ansible.cfg在调整了一些配置，ansible会自动优先查找当前目录下的该文件

ansible-playbook -i hosts,ini fix_watch.yaml