Forbidden (user=system:node, verb=get, resource=nodes, subresource=metrics)
原因分析
当前用户(kubectl指令使用的证书代表的用户,即 system:node)没有足够的权限去获取 /metrics 端点的数据, Kubernetes 集群的权限设置(Role-Based Access Control, RBAC)阻止了这种访问。
查看现有ClusterRole的权限:kubectl describe clusterrole system:node
解决措施
新建ClusterRole和ClusterRoleBinding,添加访问 /metrics 的权限
cat <<EOF > node-metrics-access.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: node-metrics-access
rules:
- apiGroups: [""]
resources: ["nodes/metrics"]
verbs: ["get", "list", "watch"]
EOF
cat <<EOF > node-metrics-access-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: node-metrics-access-binding
subjects:
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: node-metrics-access
apiGroup: rbac.authorization.k8s.io
EOF
kubectl apply -f node-metrics-access.yaml
kubectl apply -f node-metrics-access-binding.yaml
kubectl get clusterrole | grep metrics
kubectl describe clusterrole node-metrics-access
查询指令
sudo curl -s -k --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem https://127.0.0.1:10250/metrics | grep kubelet_pleg_relist_interval