一，节点维护
标记需要切换维护的节点为维护模式，强制驱逐节点上正在运行的pods，这样可以最大程度降低切换过程中影响正常运行的应用。
使用kubectl cordon 命令将node1节点标记为 unschedulable 不可调度状态
[root@master ~]# kubectl cordon node1
node/node1 cordoned
[root@master ~]#
[root@master ~]#
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 8d v1.20.0
node1 Ready,SchedulingDisabled <none> 6d23h v1.20.0
执行完上面的命令后node节点就会变成一个 SchedulingDisabled状态,表示不可调度，这样创建新的pod就不会被调度到这上面来了，
接下来维护Node1节点，使用kubectl drain命令来维护节点并驱逐节点上的pod
[root@master metrics-server]# kubectl drain node1 --ignore-daemonsets
node/node1 already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-mon/node-exporter-fclfj, kube-system/calico-node-qphlw, kube-system/kube-proxy-v6527
evicting pod kube-system/kube-state-metrics-b888767c4-tgjs5
evicting pod kube-mon/grafana-6c464596b8-pvcll
evicting pod kube-mon/prometheus-7947cdcf77-6kzbs
evicting pod kube-mon/redis-7fb8ff6779-zvxxw
pod/kube-state-metrics-b888767c4-tgjs5 evicted
pod/redis-7fb8ff6779-zvxxw evicted
pod/prometheus-7947cdcf77-6kzbs evicted
pod/grafana-6c464596b8-pvcll evicted
node/node1 evicted
上面的命令我们会强制将node1节点上的pods进行驱逐，我们加了一个--ignore-daemonsets的参数可以用来忽略Daemonset控制器管理的pods，因为这些pod不去要驱逐到其他节点，当节点驱逐完成后接下来我们就可以来对节点进行维护操作了，除了切换容器运行时可以这样操作，比如我们需要变更节点配置、升级内核等操作的时候都可以先将节点进行驱逐，然后再进行维护。
二，切换containerd
停掉docker，containerd和kubelet
systemctl stop kubelet
systemctl stop docker
systemctl stop containerd
之前安装的docker默认安装使用了containerd作为后端的容器运行时，所以不需要单独安全containerd，当然你也可以将docker和containerd完全卸载掉重新安装，这里我就使用之前安装的containerd。
因为containerd中已经实现了CRI，但是是以plugin的形式配置，以前docker中自带的containerd默认是将CRI这个插件禁掉的（使用配置 disabled_plugins = ["cri"]），这里我们重新生成默认配置文件覆盖掉
[root@node1 ~]# containerd config default >/etc/containerd/config.toml
替换默认的pause镜像为国内镜像源地址[plugins."io.containerd.grpc.v1.cri"] 下面的 sandbox_image：
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.aliyuncs.com/k8sxio/pause:3.2"
配置镜像仓库加速地址
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://bqr1dr1n.mirror.aliyuncs.com"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
endpoint = ["https://registry.aliyuncs.com/k8sxio"]
接下来修改kubelet配置，将容器运行时配置为containerd
[root@node1 ~]# cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
上面的配置中增加了两个参数--container-runtime参数是用来指定使用的容器运行时的，可选值为docker或者remote，默认就是docker，由于我们这里是用的是containerd作为容器运行时所以配置为remote值，（也就是说除了docker外的容器运行时都是配置remote值）第二个参数--container-runtime-endpoint是用来指定远程运行服务的endpoint地址在linux中一般都是使用unix套接字的形式，这里我使用的是指定containerd的套接字地址：unix:///run/containerd/containerd.sock
其实这里还应该配置一个 --image-service-endpoint 参数来指定CRI镜像服务地址如果没有指定则默认使用 --container-runtime-endpoint 的值，因为CRI都会实容器和镜像服务的
完成配置后重启containerd和kubelet
[root@node1 ~]# systemctl daemon-reload
[root@node1 ~]# systemctl restart kubelet.service
[root@node1 ~]# systemctl restart containerd
重启完成后查看检点是否正常
[root@master metrics-server]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready control-plane,master 8d v1.20.0 10.0.20.10 <none> CentOS Linux 7 (Core) 3.10.0-1160.45.1.el7.x86_64 docker://20.10.14
node1 Ready,SchedulingDisabled <none> 7d v1.20.0 10.0.20.6 <none> CentOS Linux 7 (Core) 3.10.0-1160.45.1.el7.x86_64 containerd://1.5.11
获取节点信息的时候加上-o wide可以查看到节点的更多信息，这里的节点1已经切换成containerd1.5.11了
把node1节点重新加入到集群中
kubectl uncordon node1
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 9d v1.20.0
node1 Ready <none> 7d5h v1.20.0
同样的方法处理其他的节点就可以将整个集群切换成容器运行时containerd了
三，crictl
现在可以用ctr命令来管理containerd了，查看一个名为k8s.io的命名空间
[root@master ~]# ctr ns ls
NAME LABELS
k8s.io
moby
kubernetes集群对接的containerd所有资源都在k8s.io的命名空间下面，而docker的则默认在moby的命名空间下面。
[root@master ~]# ctr -n moby c ls
CONTAINER IMAGE RUNTIME
[root@master ~]# ctr -n moby i ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
[root@master ~]# ctr -n moby t ls
TASK PID STATUS
[root@master ~]# ctr -n k8s.io c ls
CONTAINER IMAGE RUNTIME
1b26b063100dcacc6edde9e16f1d1da8453b0a5ee9477278e77b1b10205c2e79 registry.aliyuncs.com/k8sxio/pause:3.2 io.containerd.runc.v2
26895b3fc91fc391f207b96ebec3b96d7d77abbcc0b052469260898a89a9d768 docker.io/calico/pod2daemon-flexvol:v3.9.6 io.containerd.runc.v2
308a91ff15484bf92219a310b9cc07f5170c8adcdecb26f981666c96ecba833b docker.io/calico/kube-controllers:v3.9.6 io.containerd.runc.v2
4f9b550c86c0b8b5e28dfb913db80ca0e5fd5a63cca5426041cd793e7509d0d8 registry.aliyuncs.com/k8sxio/pause:3.2 io.containerd.runc.v2
5ebce69a9d6c198cdeada97e679514e43b65f7982a709c885002972efab445bd docker.io/prom/node-exporter:v1.1.1 io.containerd.runc.v2
5f1db822b2d86ca8f9e560cf3f44338d431222ff074597de197d9761f4ea96f8 docker.io/calico/cni:v3.9.6 io.containerd.runc.v2
60f2f7625993be7d05da632e347f2834c5ecd9f7d1d8695cf48ec6c1222f2205 registry.aliyuncs.com/k8sxio/pause:3.2 io.containerd.runc.v2
657abb53bb90451ed70b2c0dfd619e0668de940d8c8503af4c2613064875dc73 registry.aliyuncs.com/k8sxio/pause:3.2 io.containerd.runc.v2
76f3a9bb5e96f1a40df45a30f66d7ebe23cdc049e151b9b5d3fd58dea47db1bb registry.aliyuncs.com/google_containers/kube-scheduler:v1.20.0 io.containerd.runc.v2
86cb31d730810c2566b96baa4e1fdfb37ae41b835543f9b977d389072a5780d0 registry.aliyuncs.com/google_containers/coredns:1.7.0 io.containerd.runc.v2
a48ab29a7470980742c1dd3d7d1f45e3664ff1056c0fe4c0591192ef1b96a876 registry.aliyuncs.com/k8sxio/pause:3.2 io.containerd.runc.v2
ab33456e2c6b2e619470b962b6b12a3de2559cd7183042e3527104d96d98355f registry.aliyuncs.com/k8sxio/pause:3.2 io.containerd.runc.v2
b7dec7b4322a9d08929ddde6201d342284879aa3c77d6c8839315acd7430b3ac registry.aliyuncs.com/k8sxio/pause:3.2 io.containerd.runc.v2
b899928b2318abdd9d74950131172d634f3b15b7567f53ddc031bb04b2f24820 registry.aliyuncs.com/google_containers/coredns:1.7.0 io.containerd.runc.v2
b89f82f08a459aca3babdc094f8359c5aff435d2c61316da32e2d5246ec72430 registry.aliyuncs.com/k8sxio/pause:3.2 io.containerd.runc.v2
bbd3f5fdcb3cceac9348ea5599e7a89e6a7d3b1938649956e5873461cef21e51 registry.aliyuncs.com/google_containers/kube-proxy:v1.20.0 io.containerd.runc.v2
c3a43c0aabcabef5e19527778dc53b92b3dfa0d72df88069b1ea99d176d2ad09 docker.io/calico/node:v3.9.6 io.containerd.runc.v2
cb67862b84771493777f7f4c38f4cff29af55b7337632be07212fee5b37f15dc registry.aliyuncs.com/k8sxio/pause:3.2 io.containerd.runc.v2
d3f2ef50eeb3d30e6526887c56dd1f823fb59a314bb43d4b255c56170cf897ac registry.aliyuncs.com/google_containers/etcd:3.4.13-0 io.containerd.runc.v2
d8cef6b330437b8b7153ff03b1b5b5b351ead446f3ba9c2b6a2e686a4a01cc03 registry.aliyuncs.com/google_containers/kube-apiserver:v1.20.0 io.containerd.runc.v2
f5b0c96c833962915862ef2f6d0c854ed971143e84a681aa349e98220bdcf7d9 registry.aliyuncs.com/google_containers/kube-controller-manager:v1.20.0 io.containerd.runc.v2
f9d432a9a547e410114ed99697b4ce469a68c7a2293fac2162c6cf6032a714c9 docker.io/calico/cni:v3.9.6 io.containerd.runc.v2
fe1e663b8e50ed26b96f77462d4f6f9237cf7d0205415ca0ed824d72a9dc14d0 registry.aliyuncs.com/k8sxio/pause:3.2 io.containerd.runc.v2
可以直接使用ctr命令来直接管理镜像或者容器资源但是，但是我们使用过程中明显感觉这个工具没有docker CLI方便，从使用便捷性和功能性上考虑，更推荐crictl工具作为管理工具,crictl为CRI兼容的容器运行时提供CLI,这允许CRI开发人员无需设置kubernetes组件的情况下调试他们的运行时。
crictl安装
从github上下载相应的二进制包放在/usr/local/bin下
VERSION="v1.22.0"
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
# 如果有限制，也可以替换成下面的 URL 加速下载
wget https://download.fastgit.org/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
[root@master ~]# crictl -v
crictl version v1.22.0
使用方法
首先修改一下默认配置文件/etc/crictl.yaml在文件中指定容器运行时和镜像的endpoint地址
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
debug: false
pull-image-on-create: false
disable-pull-on-run: false
配置完成后就可以是用crictl了
1，获取pod列表
[root@master ~]# crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
b7dec7b4322a9 2 hours ago Ready calico-kube-controllers-56b44cd6d5-72g7z kube-system 0 (default)
ab33456e2c6b2 2 hours ago Ready node-exporter-mg8x5 kube-mon 0 (default)
b89f82f08a459 2 hours ago Ready coredns-7f89b7bc75-jbmv7 kube-system 0 (default)
1b26b063100dc 2 hours ago Ready coredns-7f89b7bc75-j57wv kube-system 0 (default)
4f9b550c86c0b 2 hours ago Ready calico-node-hjkjb kube-system 0 (default)
60f2f7625993b 2 hours ago Ready kube-proxy-xdbzj kube-system 0 (default)
657abb53bb904 2 hours ago Ready etcd-master kube-system 0 (default)
fe1e663b8e50e 2 hours ago Ready kube-apiserver-master kube-system 0 (default)
cb67862b84771 2 hours ago Ready kube-controller-manager-master kube-system 0 (default)
a48ab29a74709 2 hours ago Ready kube-scheduler-master kube-system 0 (default)
2，使用--name参数获取指定的pod
[root@master ~]# crictl pods --name calico
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
b7dec7b4322a9 2 hours ago Ready calico-kube-controllers-56b44cd6d5-72g7z kube-system 0 (default)
4f9b550c86c0b 2 hours ago Ready calico-node-hjkjb kube-system 0 (default)
3，可以使用标签来筛选pod
[root@master ~]# crictl pods --label k8s-app=calico-node
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
4f9b550c86c0b 2 hours ago Ready calico-node-hjkjb kube-system 0 (default)
4，获取镜像列表
[root@master ~]# crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/calico/cni v3.9.6 0ce7550069ed9 58MB
docker.io/calico/kube-controllers v3.9.6 081a5bf738add 22.9MB
docker.io/calico/node v3.9.6 15f795b449d29 88.9MB
docker.io/calico/pod2daemon-flexvol v3.9.6 63fbf227cf100 4.91MB
docker.io/prom/node-exporter v1.1.1 15a32669b6c21 12.7MB
registry.aliyuncs.com/google_containers/coredns 1.7.0 bfe3a36ebd252 14MB
registry.aliyuncs.com/google_containers/etcd 3.4.13-0 0369cf4303ffd 86.7MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.20.0 ca9843d3b5454 30.4MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.20.0 b9fa1895dcaa6 29.4MB
registry.aliyuncs.com/google_containers/kube-proxy v1.20.0 10cc881966cfd 49.5MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.20.0 3138b6e3d4712 14MB
registry.aliyuncs.com/k8sxio/pause 3.2 80d28bedfe5de 300kB
可以加上-v来显示详细信息
[root@master ~]# crictl images -v
ID: sha256:0ce7550069ed9c806ed28c0f83b6881191ce94390e23f67aaf94e539978992b8
RepoTags: docker.io/calico/cni:v3.9.6
RepoDigests: docker.io/calico/cni@sha256:fc1a5a09d4dbef71d401d620c22d4a2f064b70a9c6d52072f82fc912489a2c60
Size: 57987657
ID: sha256:081a5bf738add7b38848b86efb193f69a3f90cb943eb0a34f12981953408228a
RepoTags: docker.io/calico/kube-controllers:v3.9.6
RepoDigests: docker.io/calico/kube-controllers@sha256:e1d02ae0716e5a9ab5c426e25e54a00feb9d70cf577b5dba1c1ccbf956306fab
Size: 22942939
5，获取容器列表
[root@master ~]# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
c3a43c0aabcab 15f795b449d29 2 hours ago Running calico-node 0 4f9b550c86c0b
308a91ff15484 081a5bf738add 2 hours ago Running calico-kube-controllers 0 b7dec7b4322a9
5ebce69a9d6c1 15a32669b6c21 2 hours ago Running node-exporter 0 ab33456e2c6b2
86cb31d730810 bfe3a36ebd252 2 hours ago Running coredns 0 b89f82f08a459
b899928b2318a bfe3a36ebd252 2 hours ago Running coredns 0 1b26b063100dc
bbd3f5fdcb3cc 10cc881966cfd 2 hours ago Running kube-proxy 0 60f2f7625993b
d3f2ef50eeb3d 0369cf4303ffd 2 hours ago Running etcd 0 657abb53bb904
d8cef6b330437 ca9843d3b5454 2 hours ago Running kube-apiserver 0 fe1e663b8e50e
f5b0c96c83396 b9fa1895dcaa6 2 hours ago Running kube-controller-manager 0 cb67862b84771
76f3a9bb5e96f 3138b6e3d4712 2 hours ago Running kube-scheduler 0 a48ab29a74709
加上-s参数按照状态过滤
[root@master ~]# crictl ps -s running
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
c3a43c0aabcab 15f795b449d29 2 hours ago Running calico-node 0 4f9b550c86c0b
308a91ff15484 081a5bf738add 2 hours ago Running calico-kube-controllers 0 b7dec7b4322a9
5ebce69a9d6c1 15a32669b6c21 2 hours ago Running node-exporter 0 ab33456e2c6b2
86cb31d730810 bfe3a36ebd252 2 hours ago Running coredns 0 b89f82f08a459
b899928b2318a bfe3a36ebd252 2 hours ago Running coredns 0 1b26b063100dc
bbd3f5fdcb3cc 10cc881966cfd 2 hours ago Running kube-proxy 0 60f2f7625993b
d3f2ef50eeb3d 0369cf4303ffd 2 hours ago Running etcd 0 657abb53bb904
d8cef6b330437 ca9843d3b5454 2 hours ago Running kube-apiserver 0 fe1e663b8e50e
f5b0c96c83396 b9fa1895dcaa6 2 hours ago Running kube-controller-manager 0 cb67862b84771
76f3a9bb5e96f 3138b6e3d4712 2 hours ago Running kube-scheduler 0 a48ab29a74709
6，在容器中执行命令
crictl也有类似exec的命令，比如在容器ID为c3a43c0aabcab的容器中执行一个date命令
[root@master ~]# crictl exec -it c3a43c0aabcab date
Thu May 5 09:10:48 UTC 2022
7，查看容器的日志
[root@master ~]# crictl logs c3a43c0aabcab |tail
2022-05-05 09:14:01.977 [INFO][41] ipsets.go 356: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=1.230059ms
2022-05-05 09:14:01.977 [INFO][41] int_dataplane.go 978: Finished applying updates to dataplane. msecToApply=1.45452
2022-05-05 09:14:12.408 [INFO][41] int_dataplane.go 964: Applying dataplane updates
和kubectl logs 类似，还可以使用-f参数来follow日志输出，--tail -N也可以指定输最近的N行日志
8，资源统计
[root@master ~]# crictl stats
CONTAINER CPU % MEM DISK INODES
308a91ff15484 0.00 6.984MB 45.06kB 14
5ebce69a9d6c1 0.00 2.531MB 53.25kB 16
76f3a9bb5e96f 0.08 29.29MB 12.29kB 7
86cb31d730810 0.16 9.04MB 45.06kB 14
b899928b2318a 0.15 8.757MB 45.06kB 14
bbd3f5fdcb3cc 0.01 16.13MB 94.21kB 25
c3a43c0aabcab 0.51 24.83MB 225.3kB 65
d3f2ef50eeb3d 0.64 66.36MB 36.86kB 11
d8cef6b330437 4.64 318.4MB 20.48kB 8
f5b0c96c83396 1.45 80.05MB 45.06kB 15
此外镜像和容器相关的一些操作也支持，：
拉取镜像： crictl pull
运行 Pod： crictl runp
运行容器： crictl run
启动容器： crictl start
删除容器： crictl rm
删除镜像： crictl rmi
删除 Pod： crictl rmp
停止容器： crictl stop
停止 Pod： crictl stopp
可以参考：https://github.com/kubernetes-sigs/cri-tools
四，CLI对比
镜像容器和pod可以使用docker，ctr，crictl 这些命令行工具进行管理，比价一下常用命令的区别
1，镜像相关
图片.png
2，容器相关
图片.png
需要注意的是通过ctr container create命令创建的容器只是创建了一个静态容器，所以还需要通过ctr task start来启动容器进程，也可以直接使用ctr run命令来直接创建并运行容器，在进入容器操作时与docker不同的是必须在ctr task exec命令后指定--exec-id参数。这个id可以随便写，只要唯一就行，另外ctr没有stop容器的功能只能ctr task pause或者杀死容器ctr task kill
3，pod相关
图片.png
说明： ctictl pods列出的是pods的信息，包括pod所在的命名空间以及状态，crictl ps列出的是相应的容器的信息，而docker ps列出的是初始化容器（pause）以及应用容器的信息，初始化容器在每个pod创建时都会创建通常不会关注，所以crictl使用起来更加简洁。
