kubernetes:pod无法删除

kubernetes:v1.12.2
docker:18.06.1-ce
helm:v2.12.0

通过helm删除服务后，pods无法删除:

~]#  helm del  --purge simp-ser           
release "simp-ser" deleted  //删除成功
~]#  helm install simp-ser --name=simp-ser
Error: release simp-ser failed: object is being deleted: persistentvolumes "simp-ser" already exists //创建时报错，pvc已经存在

突如其来的报错让我有点懵，因为定义的这个pvc是跟随helm服务一起删除，我赶紧去看了下pvc服务:

~]#   kubectl get pvc
simp-ser                   Terminating   simp-ser         300Mi      RWO,RWX                       3d21h//正在删除

pvc正在删除中，但是一直没有删除成功，因为关联了pod，我又查看了相关pod：

~]# kubectl get pods -o wide
simp-ser-6b8f54bb64-tdxlw     1/1     Terminating   0          3d21h   10.244.1.71    k8s.node1   <none>//也是正在删除
~]#  kubectl describe pods simp-ser-6b8f54bb64-tdxlw
...
 Container ID:  docker://70494faefcf334b906a112a7e7f9a9821a0aaff314f9e21f4ae949d527daf7e1
...
 Warning  FailedKillPod  13m (x324 over 21h)  kubelet, k8s.node1  error killing pod: [failed to "KillContainer" for "simp-ser" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"
, failed to "KillPodSandbox" for "0e9a9fa9-6328-11e9-bd84-000c2975aae4" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
...
] //pod内的容器是70494faefcf...

看的出来是问题出在了node节点上，继续查看node的状态：

// 查看node是否正常
~]# kubectl get node 
k8s.node1    Ready      <none>   146d   v1.12.2
//切换到node节点查看kubelet 状态
~]#systemctl status kubelet
... 
Active: active (running) since 五 2019-03-29 10:07:30 CST; 3 weeks 5 days ago.//kubelet状态也是正常的
...
//查看kubelet日志
~]# journalctl -u  kubelet -f 
···
4月 24 11:08:48 k8s.node1 kubelet[2147]: E0424 11:08:48.155788    2147 kuberuntime_manager.go:810] Failed to stop sandbox {"docker" "ffc1eccb5b0ba196fe86f9987632194ee3c2e63372adb2d283a05ac0c2e68f4d"}
4月 24 11:08:48 k8s.node1 kubelet[2147]: E0424 11:08:48.155934    2147 kubelet.go:1551] error killing pod: [failed to "KillContainer" for "simp-layui" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"
4月 24 11:08:48 k8s.node1 kubelet[2147]: , failed to "KillPodSandbox" for "c5cae9f9-6321-11e9-bd84-000c2975aae4" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
4月 24 11:08:48 k8s.node1 kubelet[2147]: ]
··· //

既然kubelet没有问题，那再去看看pod中的容器：

~]# docker ps 
70494faefcf3        f4b4fccc65bb           "sh -c 'java -jar /o…"   3 days ago          Up 3 days                               k8s_simp-ser_simp-ser-6b8f54bb64-tdxlw_jx_0e9a9fa9-6328-11e9-bd84-000c2975aae4_0 // 容器还在运行
//查看下docker日志
~]#journalctl -u docker -f 
···
msg="Container 70494faefcf334b906a112a7e7f9a9821a0aaff314f9e21f4ae949d527daf7e1 failed to exit within 30 seconds of signal 15 - using the force"
···//docker 无法 kill 掉容器

初步得出的判断是因为容器化后，由于单容器单进程，已经没有传统意义上的 init 进程了。应用进程直接占用
了 pid 1 的进程号，Linux 内核中会对 pid 1 进程发送特殊的信号量。具体可参考Docker init 进程。

最终解决办法：

~]# docker restart // 重启nodes节点

kubernetes:pod无法删除

推荐阅读更多精彩内容