问题现象：
使用 kubectl create -f release.yaml 创建 deployment 时，启动的 pod 无法拉取镜像，报错 Unknown desc = Error response from daemon: pull access denied for cr.registry.platform.foobar.cn/prod/foobar, repository does not exist or may require 'docker login' 如下：
# kubectl describe pod foobar-699df4f77b-8zkvj
省略其他……
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 80s default-scheduler Successfully assigned default/foobar-699df4f77b-8zkvj to cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc8
Normal Pulling 72s (x3 over 108s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc8 Pulling image "cr.registry.platform.foobar.cn/prod/foobar:1.5.11"
Warning Failed 72s (x3 over 108s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc8 Failed to pull image "cr.registry.platform.foobar.cn/prod/foobar:1.5.11": rpc error: code = Unknown desc = Error response from daemon: pull access denied for cr.registry.platform.foobar.cn/prod/foobar, repository does not exist or may require 'docker login'
Warning Failed 72s (x3 over 108s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc8 Error: ErrImagePull
Normal BackOff 37s (x6 over 107s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc8 Back-off pulling image "cr.registry.platform.foobar.cn/prod/foobar:1.5.11"
Warning Failed 37s (x6 over 107s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc8 Error: ImagePullBackOff
直接在 node 节点上 docker pull 可以拉取镜像。
# docker pull cr.registry.platform.foobar.cn/prod/foobar:1.5.11
1.5.11: Pulling from prod/foobar
840caab23da4: Pull complete
c73e835deefd: Pull complete
2eec68d27246: Pull complete
c5cfd8d4e231: Pull complete
db1ee2b864ce: Pull complete
8d01cb45f656: Pull complete
b47d363c0228: Pull complete
0dd590ba0e0b: Pull complete
b15216aaf651: Pull complete
da20bc470edc: Pull complete
ae966cba2cfc: Pull complete
642b4b98622b: Pull complete
Digest: sha256:f523eb8b8f1c1da7c659a6a2a60bfc611ef7287e42e724eb4e99e8a37076f53e
Status: Downloaded newer image for cr.registry.platform.foobar.cn/prod/foobar:1.5.11
原因分析：
cr.registry.platform.foobar.cn/prod/foobar:1.5.11 存放在自建的私有镜像仓库中。kubernetes 并不能直接使用 docker 的 /root/.docker/config.json 凭证拉取这个镜像，需要单独配置自己的凭证来拉取镜像。
解决办法：
方式一：使用密钥编排容器，在 Pod 上指定 ImagePullSecrets。
// 查看 kubernetes 的 secret
# kubectl get secret -o wide
NAME TYPE DATA AGE
aliyun-acr-credential-a kubernetes.io/dockerconfigjson 1 24d
aliyun-acr-credential-b kubernetes.io/dockerconfigjson 1 24d
default-token-x9m7q kubernetes.io/service-account-token 3 24d
// 创建 kubernetes 拉取私有镜像仓库的凭证 foobar-registry。
// kubectl create secret generic aliyun-registry-ee \
// --from-file=.dockerconfigjson=/root/.docker/config.json \
// --type=kubernetes.io/dockerconfigjson
# kubectl create secret docker-registry foobar-registry --docker-server=私有镜像仓库地址 --docker-username=用户名 --docker-password=密码 --docker-email=邮箱
secret/foobar-registry created
# kubectl get secret -o wide
NAME TYPE DATA AGE
aliyun-acr-credential-a kubernetes.io/dockerconfigjson 1 24d
aliyun-acr-credential-b kubernetes.io/dockerconfigjson 1 24d
default-token-x9m7q kubernetes.io/service-account-token 3 24d
foobar-registry kubernetes.io/dockerconfigjson 1 2s
// 在 Pod 上指定 ImagePullSecrets 为 foobar-registry 来拉取镜像。
# vim release.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: foobar
labels:
app: foobar
spec:
replicas: 2
selector:
matchLabels:
app: foobar
template:
metadata:
labels:
app: foobar
spec:
containers:
- name: foobar
image: cr.registry.platform.foobar.cn/prod/foobar:1.5.11
imagePullPolicy: Always
imagePullSecrets:
- name: foobar-registry
# kubectl create -f release.yaml
// 查看 kubernetes pod 事件，可以知道成功拉取了私有镜像仓库的镜像
# kubectl describe pod foobar-699df4f77b-8zkvj
省略其他……
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 31s (x3 over 45s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc9 Pulling image "cr.registry.platform.foobar.cn/prod/foobar:1.5.11"
Normal Pulled 31s (x3 over 45s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc9 Successfully pulled image "cr.registry.platform.foobar.cn/prod/foobar:1.5.11"
Normal Created 31s (x3 over 45s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc9 Created container foobar
Normal Started 31s (x3 over 45s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc9 Started container foobar
方式二：实现无密钥编排，为服务账户添加 ImagePullSecrets。
// 查看 kubernetes 的 serviceaccounts
# kubectl get serviceaccounts default -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2020-12-04T10:04:26Z"
name: default
namespace: default
resourceVersion: "344"
selfLink: /api/v1/namespaces/default/serviceaccounts/default
uid: 1740b59c-3618-11eb-848c-00163e01064f
secrets:
- name: default-token-x9m7q
// 导出 服务账户 的配置文件
# kubectl get serviceaccounts default -o yaml > /tmp/sa.yaml
# cat /tmp/sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2020-12-04T10:04:26Z"
name: default
namespace: default
resourceVersion: "344"
selfLink: /api/v1/namespaces/default/serviceaccounts/default
uid: 1740b59c-3618-11eb-848c-00163e01064f
secrets:
- name: default-token-x9m7q
// 删除了 resourceVersion 字段，添加了 imagePullSecrets 字段，name 为上述添加的 secret foobar-registry
# vim /tmp/sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2020-12-04T10:04:26Z"
name: default
namespace: default
selfLink: /api/v1/namespaces/default/serviceaccounts/default
uid: 1740b59c-3618-11eb-848c-00163e01064f
secrets:
- name: default-token-x9m7q
imagePullSecrets:
- name: foobar-registry
// 替换现有的 服务账户 的配置文件
# kubectl replace serviceaccount default -f /tmp/sa.yaml
serviceaccount/default replaced
# kubectl get serviceaccounts default -o yaml
apiVersion: v1
imagePullSecrets:
- name: foobar-registry
kind: ServiceAccount
metadata:
creationTimestamp: "2020-12-04T10:04:26Z"
name: default
namespace: default
resourceVersion: "5918451"
selfLink: /api/v1/namespaces/default/serviceaccounts/default
uid: 1740b59c-3618-11eb-848c-00163e01064f
secrets:
- name: default-token-x9m7q
// 在 Pod 上不用单独指定 ImagePullSecrets 来拉取镜像。因为配置了 serviceaccounts 的 ImagePullSecrets，可以直接拉取镜像。
# vim release.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: foobar
labels:
app: foobar
spec:
replicas: 2
selector:
matchLabels:
app: foobar
template:
metadata:
labels:
app: foobar
spec:
containers:
- name: foobar
image: cr.registry.platform.foobar.cn/prod/foobar:1.5.11
imagePullPolicy: Always
# kubectl create -f release.yaml
// 查看 kubernetes pod 事件，可以知道成功拉取了私有镜像仓库的镜像
# kubectl describe pod foobar-699df4f77b-ngg7f
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 26s (x3 over 41s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc9 Pulling image "cr.registry.platform.foobar.cn/prod/foobar:1.5.11"
Normal Pulled 26s (x3 over 41s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc9 Successfully pulled image "cr.registry.platform.foobar.cn/prod/foobar:1.5.11"
Normal Created 26s (x3 over 41s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc9 Created container foobar
Normal Started 26s (x3 over 40s) kubelet, cn-shanghai-foobar-d01.i-a0z01nzs2pkled5iikc9 Started container foobar
Kubernetes 查看 docker secret 明文账号密码
kubectl get secrets foobar-registry -n default --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode
参考：
- 使用私有仓库：https://kubernetes.io/zh/docs/concepts/containers/images/#using-a-private-registry
- 在 Pod 上指定 ImagePullSecrets：https://kubernetes.io/zh/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
- 为服务账户添加 ImagePullSecrets：https://kubernetes.io/zh/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account
- 在 Kubernetes 集群如何支持私有镜像：https://help.aliyun.com/knowledge_detail/86562.html
