背景
kubernetes: 1.16.3
master: 3台
采用kubeadm部署，在证书还有30天以上时，使用kubeadm alpha certs renew all更新所有证书，以为万无一失，但是到原有的证书过期时间，发现API异常告警。
问题
api-server日志：
E0721 08:09:28.129981 1 available_controller.go:416] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://10.244.69.146:6443/apis/custom.metrics.k8s.io/v1beta1: bad status from https://10.244.69.146:6443/apis/custom.metrics.k8s.io/v1beta1: 401
E0721 08:09:28.133091 1 available_controller.go:416] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://10.244.69.146:6443/apis/custom.metrics.k8s.io/v1beta1: bad status from https://10.244.69.146:6443/apis/custom.metrics.k8s.io/v1beta1: 401
E0721 08:09:28.133460 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.244.8.165:4443/apis/metrics.k8s.io/v1beta1: bad status from https://10.244.8.165:4443/apis/metrics.k8s.io/v1beta1: 401
E0721 08:09:28.135093 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.244.8.165:4443/apis/metrics.k8s.io/v1beta1: bad status from https://10.244.8.165:4443/apis/metrics.k8s.io/v1beta1: 401
E0721 08:09:28.139986 1 available_controller.go:416] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://10.244.69.146:6443/apis/custom.metrics.k8s.io/v1beta1: bad status from https://10.244.69.146:6443/apis/custom.metrics.k8s.io/v1beta1: 401
E0721 08:09:28.141188 1 available_controller.go:416] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://10.244.69.146:6443/apis/custom.metrics.k8s.io/v1beta1: bad status from https://10.244.69.146:6443/apis/custom.metrics.k8s.io/v1beta1: 401
E0721 08:09:28.143084 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.244.8.165:4443/apis/metrics.k8s.io/v1beta1: bad status from https://10.244.8.165:4443/apis/metrics.k8s.io/v1beta1: 401
解决思路
1.kubectl无法使用，显示证书过期或者失效，随即
$ kubectl get node
Unable to connect to the server: x509: certificate has expired or is not yet valid
2.随即将/etc/kubernetes/admin.conf复制~/.kube/conf，执行命令后，还是返回步骤1的错误，随即意识到是api-server的问题。
3.检查pki木块下的所有证书过期时间
$ for i in $(ls *.crt); do echo "===== $i ====="; openssl x509 -in $i -text -noout | grep -A 3 'Validity' ; done
===== apiserver.crt =====
Validity
Not Before: Jul 21 08:08:33 2020 GMT
Not After : Apr 14 05:58:54 2022 GMT
Subject: CN=kube-apiserver
===== apiserver-kubelet-client.crt =====
Validity
Not Before: Jul 21 08:08:33 2020 GMT
Not After : Apr 14 06:00:03 2022 GMT
Subject: O=system:masters, CN=kube-apiserver-kubelet-client
===== ca.crt =====
Validity
Not Before: Jul 21 08:08:33 2020 GMT
Not After : Jul 19 08:08:33 2030 GMT
Subject: CN=kubernetes
===== front-proxy-ca.crt =====
Validity
Not Before: Jul 21 08:08:34 2020 GMT
Not After : Jul 19 08:08:34 2030 GMT
Subject: CN=front-proxy-ca
===== front-proxy-client.crt =====
Validity
Not Before: Jul 21 08:08:34 2020 GMT
Not After : Apr 14 06:00:40 2022 GMT
Subject: CN=front-proxy-client
4.pki下证书检测均没有过期，问题就很明显，出在正在运行的container上，重启kubelet无法解决问题
5.重启kulet并不会重建container
systemctl stop kubelet
docker ps -q | xargs docker stop
df -Th | grep "docker" | awk '{print $NF}' | xargs umount
df -Th | grep "kubelet" | awk '{print $NF}' | xargs umount
docker ps -a -q | xargs docker rm
systemctl restart kubelet
6.重启后，发现除了主控制节点(kubeadm init的第一台服务器)，其余master节点都恢复正常
7.查看报错信息，主控制节点上的kubelet无法启动
Jul 21 17:53:03 master001 kubelet[23047]: E0721 17:53:03.545713 23047 bootstrap.go:250] unable to load TLS configuration from existing bootstrap client config: tls: private key does not match public key
Jul 21 17:53:03 master001 kubelet[23047]: F0721 17:53:03.545749 23047 server.go:271] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
8.对比其他集群，发现均没有/etc/kubernetes/bootstrap-kubelet.conf文件
9.随即针对kubelet进行检查
$ cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
kubelet在启动时，会优先读取kubelet.conf，当kubelet不可用时，才读取bootstrap-kubelet.conf
10.对比住控制节点上的kubelet.conf发现
主控制节点
users:
- name: system:node:master001
user:
client-certificate-data: *****
client-key-data: *****
***** 为加密值
其他主控节点
users:
- name: default-auth
user:
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
可以看到直接读取的文件，在首个master节点初始化集群时，kubelet还没加入集群，也就没法生成这个文件，所以直接用的加密值，我们只需要手动更改即可。
11.重启首台master，集群全部恢复正常
总结
1.kubeadm renew在更新证书后，api-server，controller，schedule并不会重载证书文件，需要重建container。
2.主控制节点的kubelet.conf中使用的加密值，并不是文件，而且kubeadm renew也不会更新这个文件，所以需要手动操作。
