本文基于k8s 1.22版本。

流程图

kubelet root dir

[root@glusterfs-2 kubelet]# tree /var/lib/kubelet/plugins*
/var/lib/kubelet/plugins
└── kubernetes.io
    └── csi
        └── pv
            ├── pvc-1dfaf0ee-d88e-4b94-8a36-4e7bde2e10ab
            │   ├── globalmount
            │   └── vol_data.json
            ├── pvc-61a403de-cb24-4d2d-ab7e-df75f643366a
            │   ├── globalmount
            │   └── vol_data.json
            └── pvc-8b86f8b4-cc62-4c3c-ba48-1e235215dbd8
                ├── globalmount
                └── vol_data.json
/var/lib/kubelet/plugins_registry
├── kadalu
│   └── csi.sock
└── kadalu-reg.sock

10 directories, 5 files

CSI plugin register相关代码

pkg/kubelet/kubelet.go 1411 initializeRuntimeDependentModules()
pkg/kubelet/kubelet.go 784 klet.pluginManager
pkg/kubelet/pluginmanager/reconciler/reconciler.go 110 reconcile()
pkg/kubelet/pluginmanager/operationexecutor/operation_generator.go 74 GenerateRegisterPluginFunc()
pkg/kubelet/pluginmanager/cache/types.go
pkg/volume/csi/csi_plugin.go
pkg/volume/csi/csi_client.go
pkg/kubelet/pluginmanager/plugin_manager.go 108 Run()
staging/src/k8s.io/kubelet/pkg/apis/pluginregistration/v1/api.proto

plugin的注册逻辑

遍历kubelet root dir下的plugins_registry目录中的所有socket文件，其实只需要调用<driver>-reg.sock文件即可，其他的socket文件在dial之后调用GetInfo()会报错"Method not found!"。

1. reconciler对象会rpc调用node-driver-registrar sidecar container中rpc server提供的的GetInfo。
2. 然后根据返回字段的type，从最开始注册的pluginHandlers中查找对应的handler，这里就是上文说的CSIPlugin type的csi.RegistrationHandler{}对象。
3. 调用该对象的ValidatePlugin和RegisterPlugin来注册插件，这里的注册插件其实就是设置node annotation和label, 以及创建/更新CSINode对象(CSINodeDrivers)。
4. 最后rpc调用NotifyRegistrationStatus告知注册结果。

对于上述第三点，解释如下：

更新node annotation；创建更新CSINode对象。

1. 更新node annotation主要是往当前Node中增加一个annotation csi.volume.kubernetes.io/nodeid:{"$csiDriverName":"$driverNodeID"} ，csiDriverName是之前rpc调用node-driver-registrar sidecar container的GetInfo获得的，
   driverNodeID是直接rpc调用自定义csi-plugin的node service NodeGetInfo获得的 。后面会详细介绍。

(dlv) p socketPath
"/var/lib/kubelet/plugins_registry/kadalu-reg.sock"
(dlv) p infoResp
*k8s.io/kubelet/pkg/apis/pluginregistration/v1.PluginInfo {
    Type: "CSIPlugin",
    Name: "kadalu",
    Endpoint: "/var/lib/kubelet/plugins_registry/kadalu/csi.sock",
    SupportedVersions: []string len: 1, cap: 1, ["1.0.0"],}

(dlv) p c
*k8s.io/kubernetes/pkg/volume/csi.csiDriverClient {
    driverName: "kadalu",
    addr: "/var/lib/kubelet/plugins_registry/kadalu/csi.sock",
    metricsManager: *k8s.io/kubernetes/pkg/volume/csi.MetricsManager {driverName: "kadalu"},
    nodeV1ClientCreator: k8s.io/kubernetes/pkg/volume/csi.newV1NodeClient,}
(dlv) p res
*github.com/container-storage-interface/spec/lib/go/csi.NodeGetInfoResponse {
    NodeId: "glusterfs-2.deeproute.ai",
    MaxVolumesPerNode: 0,
    AccessibleTopology: *github.com/container-storage-interface/spec/lib/go/csi.Topology nil,
    XXX_NoUnkeyedLiteral: struct {} {},
    XXX_unrecognized: []uint8 len: 0, cap: 0, nil,
    XXX_sizecache: 0,}

2. 更新node label主要是将topology信息保存在metadata.labels中。
3. apiserver中创建/更新CSINode对象，创建CSINode对象逻辑可见CreateCSINode() ，更新CSINode对象逻辑可见 installDriverToCSINode() 。
   • CreateCSINode()会创建CSINode，设置metadata中的name和ownerReferences，以及创建空的spec.drivers，然后设置annotation，storage.alpha.kubernetes.io/migrated-plugins:<migratedPlugin1, migratedPlugin2>
   • installDriverToCSINode()主要是更新spec.driver信息，其中包括name，nodeID, topologyKeys, allocatable。如果migration annotation有变化的话也会做更新。

🍺 /root/go/src/k8s.io/kubernetes ☞ git:(master) ✗ kubectl get nodes glusterfs-2.deeproute.ai -o yaml | yq e '.metadata' -            
annotations:
  csi.volume.kubernetes.io/nodeid: '{"kadalu":"glusterfs-2.deeproute.ai"}'
  flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"9e:b1:47:3f:c2:cd"}'
  flannel.alpha.coreos.com/backend-type: vxlan
  flannel.alpha.coreos.com/kube-subnet-manager: "true"
  flannel.alpha.coreos.com/public-ip: 10.9.9.151
  kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
  node.alpha.kubernetes.io/ttl: "0"
  volumes.kubernetes.io/controller-managed-attach-detach: "true"
creationTimestamp: "2021-08-11T06:50:02Z"
labels:
  beta.kubernetes.io/arch: amd64
  beta.kubernetes.io/os: linux
  kubernetes.io/arch: amd64
  kubernetes.io/hostname: glusterfs-2.deeproute.ai
  kubernetes.io/os: linux
name: glusterfs-2.deeproute.ai
resourceVersion: "7888711"
uid: ab7b5284-984c-4259-978d-f954151e300a
🍺 /root/go/src/k8s.io/kubernetes ☞ git:(master) ✗ kubectl get csinodes.storage.k8s.io glusterfs-2.deeproute.ai -o yaml               
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
  annotations:
    storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/azure-disk,kubernetes.io/cinder
  creationTimestamp: "2021-08-11T06:50:03Z"
  name: glusterfs-2.deeproute.ai
  ownerReferences:
  - apiVersion: v1
    kind: Node
    name: glusterfs-2.deeproute.ai
    uid: ab7b5284-984c-4259-978d-f954151e300a
  resourceVersion: "7878094"
  uid: 51efb47e-340d-4b28-98fd-485164259dc9
spec:
  drivers:
  - name: kadalu
    nodeID: glusterfs-2.deeproute.ai
    topologyKeys: null

node-driver-registrar

前面提到过GetInfo是通过node-driver-registrar sidecar container的GetInfo获得的，通过与/var/lib/kubelet/plugins_registry/<csi-driver>-reg.sock进行通信来获取的。

node-driver-registrar主要有以下两个作用：

1. 连接到/var/lib/kubelet/plugins_registry/<csi-driver>/csi.sock Unix domain socket
   用于获取csi driver name。其会调用csi driver的GetPluginInfo来获取。
2. 创建/var/lib/kubelet/plugins_registry/<csi-driver>-reg.sock Unix domain socket来监听kubelet的GetInfo和NotifyRegistrationStatus请求，其实现了RegistrationServer接口。

链接

• Container Storage Interface (CSI)
• CSI Volume Plugins in Kubernetes Design Doc
• external provisioner design
• Kubernetes Container Storage Interface (CSI) Documentation
• Plugin Registration Service
• node-driver-registrar
• Kubernetes必备知识： kubernetes CSI
• Kubernetes学习笔记之CSI Plugin注册机制源码解析
• 一文读懂 K8s 持久化存储流程