背景介绍

需要访问ceph创建subvolume卷，因为在ceph quincy版本rest api不支持subvolume的创建（reef版本已经支持）。所以需要引入ceph的sdk库进行subvlume的创建。我选择的是go-ceph:github.com/ceph/go-ceph

前提条件

1. 在ceph已创建好volume和subvolumegroup

ceph fs volume create xxx
ceph fs subvolumegroup create <vol_name> <group_name>

2. 创建对接账号

ceph auth get-or-create client.test mon "allow r" mds "allow rw" osd 'allow rwx pool=cephfs.testfs.data'
#生成密钥对
ceph auth get client.test -o ceph.client.test.keyring
ceph auth print-key client.test> test.key

创建subvolume

创建go-ceph客户端(参考csi代码修改)

import (
    "fmt"
    "github.com/ceph/go-ceph/rados"
)
const (
    cephConfigRoot = "/etc/ceph"
    // CephConfigPath ceph configuration file.
    CephConfigPath = "/etc/ceph/ceph.conf"
    keyRing = "/etc/ceph/keyring"
)
func GetConn(monitors, user, keyfile string) (*rados.Conn, error) {
    // construct and connect a new rados.Conn
    args := []string{"--keyfile=" + keyfile}
    conn, err := rados.NewConnWithUser(user)
    if err != nil {
        return nil, fmt.Errorf("creating a new connection failed: %w", err)
    }
    err = conn.ParseCmdLineArgs(args)
    if err != nil {
        return nil, fmt.Errorf("parsing cmdline args (%v) failed: %w", args, err)
    }
    if err = conn.ReadConfigFile(CephConfigPath); err != nil {
        return nil, fmt.Errorf("failed to read config file %q: %w", CephConfigPath, err)
    }
    err = conn.Connect()
    if err != nil {
        return nil, fmt.Errorf("connecting failed: %w", err)
    }
    return conn, nil
}
conn, err := GetConn("", "slurm", "/etc/ceph/slurm.key")

创建ceph fs客户端, 并创建subvolume

import ca "github.com/ceph/go-ceph/cephfs/admin"

fsadmin := ca.NewFromConn(conn)
if err := fsadmin.CreateSubVolume("testfs", "data", fmt.Sprintf("mount%d", rand.Intn(100)), &ca.SubVolumeOptions{Size: 10}); err != nil {
    fmt.Printf("create subvolume err, %v", err)
    eturn err
}
return nil

编译

因为go-ceph 引入了ceph的c语言动态链接库，需要找一台linux机器，安装对应的动态链接库之后才能正常编译。ceph-go中已经做了相关说明GitHub - ceph/go-ceph: Go bindings for Ceph :octopus: :octopus:

The code in go-ceph is purely a library module. Typically, one will import go-ceph in another Go based project. When building the code the native RADOS, RBD, & CephFS library and development headers are expected to be installed.
On debian based systems (apt) these may be:

libcephfs-dev librbd-dev librados-dev
On rpm based systems (dnf, yum, etc) these may be:

libcephfs-devel librbd-devel librados-devel
On MacOS you can use brew to install the libraries:

brew tap mulbc/ceph-client
brew install ceph-client