【报错】
隔壁部门研发环境k8s集群（单master）的master上 kube-flannel 和 kube-proxy 一直pending，找我们帮忙排查。
describe 查看pod 报错说挂载问题，一个节点上所有pod挂载不上，大概率是kubelet会报错（当然也可以从journalctl命令过滤错误日志）。
kubelet 报错如下：
kubelet 报错如下：
Redirecting to /bin/systemctl status -l kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since 五 2021-10-08 12:35:54 CST; 53min ago
Docs: https://kubernetes.io/docs/
Main PID: 51270 (kubelet)
Tasks: 16
Memory: 48.3M
CGroup: /system.slice/kubelet.service
└─51270 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.1
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.379424 51270 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/secret/e2f8f8ed-6386-4f0b-b7ef-ebde580ba880-flannel-token-bfxdm\" (\"e2f8f8ed-6386-4f0b-b7ef-ebde580ba880\")" failed. No retries permitted until 2021-10-08 13:31:53.37940227 +0800 CST m=+3359.052988748 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"flannel-token-bfxdm\" (UniqueName: \"kubernetes.io/secret/e2f8f8ed-6386-4f0b-b7ef-ebde580ba880-flannel-token-bfxdm\") pod \"kube-flannel-ds-amd64-wh59m\" (UID: \"e2f8f8ed-6386-4f0b-b7ef-ebde580ba880\") : couldn't propagate object cache: timed out waiting for the condition"
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.382424 51270 reflector.go:125] object-"ingress-nginx"/"ingress-nginx-token-pnlcd": Failed to list *v1.Secret: secrets "ingress-nginx-token-pnlcd" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "ingress-nginx": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.580367 51270 reflector.go:125] object-"ingress-nginx"/"ingress-nginx-admission": Failed to list *v1.Secret: secrets "ingress-nginx-admission" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "ingress-nginx": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.780370 51270 reflector.go:125] object-"kube-system"/"flannel-token-bfxdm": Failed to list *v1.Secret: secrets "flannel-token-bfxdm" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: I1008 13:29:51.980323 51270 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
10月 08 13:29:51 XL-master kubelet[51270]: E1008 13:29:51.980714 51270 reflector.go:125] object-"kube-system"/"kube-flannel-cfg": Failed to list *v1.ConfigMap: configmaps "kube-flannel-cfg" is forbidden: User "system:node:XL-master" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:51 XL-master kubelet[51270]: I1008 13:29:51.986133 51270 kubelet_node_status.go:72] Attempting to register node xl-master
10月 08 13:29:52 XL-master kubelet[51270]: E1008 13:29:52.380341 51270 reflector.go:125] object-"kube-system"/"kube-proxy-token-74txd": Failed to list *v1.Secret: secrets "kube-proxy-token-74txd" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:52 XL-master kubelet[51270]: E1008 13:29:52.580340 51270 reflector.go:125] object-"kube-system"/"kube-proxy": Failed to list *v1.ConfigMap: configmaps "kube-proxy" is forbidden: User "system:node:XL-master" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
10月 08 13:29:52 XL-master kubelet[51270]: E1008 13:29:52.780279 51270 reflector.go:125] object-"kubesphere-monitoring-system"/"qingcloud": Failed to list *v1.Secret: secrets "qingcloud" is forbidden: User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kubesphere-monitoring-system": no relationship found between node "XL-master" and this object
【分析】
- 重点是这里
User "system:node:XL-master" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "XL-master" and this object
如上可知 system:node:XL-master用户没有该命名空间的相关权限
- 配置文件
查看/etc/kubernetes/kubelet.conf文件，kubelet 确实使用的是system:node:XL-master用户。
具当事部门运维人员说并没有更换过用户，权限绑定也没有人删除过。好吧我们是友军，不深究了。
【解决】
创建一个clusterrolebinding给system:node:XL-master用户绑定cluster-admin权限
kubectl create clusterrolebinding 10-18-XL-master --clusterrole=cluster-admin --user=system:node:XL-master
说明：10-18-XL-master是 clusterrolebinding 的名字，随便起的。
