本文基于k8s 1.22版本。
流程图
kubelet root dir
[root@glusterfs-2 kubelet]# tree /var/lib/kubelet/plugins*
/var/lib/kubelet/plugins
└── kubernetes.io
└── csi
└── pv
├── pvc-1dfaf0ee-d88e-4b94-8a36-4e7bde2e10ab
│ ├── globalmount
│ └── vol_data.json
├── pvc-61a403de-cb24-4d2d-ab7e-df75f643366a
│ ├── globalmount
│ └── vol_data.json
└── pvc-8b86f8b4-cc62-4c3c-ba48-1e235215dbd8
├── globalmount
└── vol_data.json
/var/lib/kubelet/plugins_registry
├── kadalu
│ └── csi.sock
└── kadalu-reg.sock
10 directories, 5 files
CSI plugin register相关代码
pkg/kubelet/kubelet.go 1411 initializeRuntimeDependentModules()
pkg/kubelet/kubelet.go 784 klet.pluginManager
pkg/kubelet/pluginmanager/reconciler/reconciler.go 110 reconcile()
pkg/kubelet/pluginmanager/operationexecutor/operation_generator.go 74 GenerateRegisterPluginFunc()
pkg/kubelet/pluginmanager/cache/types.go
pkg/volume/csi/csi_plugin.go
pkg/volume/csi/csi_client.go
pkg/kubelet/pluginmanager/plugin_manager.go 108 Run()
staging/src/k8s.io/kubelet/pkg/apis/pluginregistration/v1/api.proto
plugin的注册逻辑
遍历kubelet root dir下的plugins_registry目录中的所有socket文件，其实只需要调用<driver>-reg.sock文件即可，其他的socket文件在dial之后调用GetInfo()会报错"Method not found!"。
-
reconciler对象会rpc调用node-driver-registrar sidecar container中rpc server提供的的GetInfo。 - 然后根据返回字段的type，从最开始注册的
pluginHandlers中查找对应的handler，这里就是上文说的CSIPlugintype的csi.RegistrationHandler{}对象。 - 调用该对象的
ValidatePlugin和RegisterPlugin来注册插件，这里的注册插件其实就是设置node annotation和label, 以及创建/更新CSINode对象(CSINodeDrivers)。 - 最后rpc调用
NotifyRegistrationStatus告知注册结果。
对于上述第三点，解释如下：
更新node annotation；创建更新CSINode对象。
- 更新node annotation主要是往当前Node中增加一个annotation
csi.volume.kubernetes.io/nodeid:{"$csiDriverName":"$driverNodeID"}，csiDriverName是之前rpc调用node-driver-registrar sidecar container的GetInfo获得的，
driverNodeID是直接rpc调用自定义csi-plugin的node serviceNodeGetInfo获得的 。后面会详细介绍。
(dlv) p socketPath
"/var/lib/kubelet/plugins_registry/kadalu-reg.sock"
(dlv) p infoResp
*k8s.io/kubelet/pkg/apis/pluginregistration/v1.PluginInfo {
Type: "CSIPlugin",
Name: "kadalu",
Endpoint: "/var/lib/kubelet/plugins_registry/kadalu/csi.sock",
SupportedVersions: []string len: 1, cap: 1, ["1.0.0"],}
(dlv) p c
*k8s.io/kubernetes/pkg/volume/csi.csiDriverClient {
driverName: "kadalu",
addr: "/var/lib/kubelet/plugins_registry/kadalu/csi.sock",
metricsManager: *k8s.io/kubernetes/pkg/volume/csi.MetricsManager {driverName: "kadalu"},
nodeV1ClientCreator: k8s.io/kubernetes/pkg/volume/csi.newV1NodeClient,}
(dlv) p res
*github.com/container-storage-interface/spec/lib/go/csi.NodeGetInfoResponse {
NodeId: "glusterfs-2.deeproute.ai",
MaxVolumesPerNode: 0,
AccessibleTopology: *github.com/container-storage-interface/spec/lib/go/csi.Topology nil,
XXX_NoUnkeyedLiteral: struct {} {},
XXX_unrecognized: []uint8 len: 0, cap: 0, nil,
XXX_sizecache: 0,}
- 更新node label主要是将topology信息保存在metadata.labels中。
- apiserver中创建/更新CSINode对象，创建CSINode对象逻辑可见
CreateCSINode()，更新CSINode对象逻辑可见installDriverToCSINode()。-
CreateCSINode()会创建CSINode，设置metadata中的name和ownerReferences，以及创建空的spec.drivers，然后设置annotation，storage.alpha.kubernetes.io/migrated-plugins:<migratedPlugin1, migratedPlugin2> -
installDriverToCSINode()主要是更新spec.driver信息，其中包括name，nodeID,topologyKeys,allocatable。如果migration annotation有变化的话也会做更新。
-
🍺 /root/go/src/k8s.io/kubernetes ☞ git:(master) ✗ kubectl get nodes glusterfs-2.deeproute.ai -o yaml | yq e '.metadata' -
annotations:
csi.volume.kubernetes.io/nodeid: '{"kadalu":"glusterfs-2.deeproute.ai"}'
flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"9e:b1:47:3f:c2:cd"}'
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: "true"
flannel.alpha.coreos.com/public-ip: 10.9.9.151
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: "0"
volumes.kubernetes.io/controller-managed-attach-detach: "true"
creationTimestamp: "2021-08-11T06:50:02Z"
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
kubernetes.io/arch: amd64
kubernetes.io/hostname: glusterfs-2.deeproute.ai
kubernetes.io/os: linux
name: glusterfs-2.deeproute.ai
resourceVersion: "7888711"
uid: ab7b5284-984c-4259-978d-f954151e300a
🍺 /root/go/src/k8s.io/kubernetes ☞ git:(master) ✗ kubectl get csinodes.storage.k8s.io glusterfs-2.deeproute.ai -o yaml
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
annotations:
storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/azure-disk,kubernetes.io/cinder
creationTimestamp: "2021-08-11T06:50:03Z"
name: glusterfs-2.deeproute.ai
ownerReferences:
- apiVersion: v1
kind: Node
name: glusterfs-2.deeproute.ai
uid: ab7b5284-984c-4259-978d-f954151e300a
resourceVersion: "7878094"
uid: 51efb47e-340d-4b28-98fd-485164259dc9
spec:
drivers:
- name: kadalu
nodeID: glusterfs-2.deeproute.ai
topologyKeys: null
node-driver-registrar
前面提到过GetInfo是通过node-driver-registrar sidecar container的GetInfo获得的，通过与/var/lib/kubelet/plugins_registry/<csi-driver>-reg.sock进行通信来获取的。
node-driver-registrar主要有以下两个作用：
- 连接到
/var/lib/kubelet/plugins_registry/<csi-driver>/csi.sockUnix domain socket
用于获取csi driver name。其会调用csi driver的GetPluginInfo来获取。 - 创建
/var/lib/kubelet/plugins_registry/<csi-driver>-reg.sockUnix domain socket来监听kubelet的GetInfo和NotifyRegistrationStatus请求，其实现了RegistrationServer接口。
