Ceph作为已经发展了10年的分布式存储，目前已经有很多生产系统的实例，当中对OpenStack提供后端存储这块也是时下最流行也比较成熟的解决方案，经过一段时间的使用，对其结合自己实际系统做了一个简单的应用总结。

现有的OpenStack环境当中Ceph只对接Cinder和部分Nova，根据需求不同，有的虚拟机跑在本地文件系统，有的则放在Ceph存储当中。数据卷则统一由Ceph提供块存储。

与OpenStack集成

基层这块Ceph官网已经提供健全的文档，我这里就不多讲了，直接贴上我的操作和配置。

• 安装Ceph客户端

$ yum install -y centos-release-ceph-jewel
$ yum clean all && yum makecache
$ yum install ceph python-rbd

• 配置客户端认证

ceph auth get-or-create client.oepnstack mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=instances  -o /etc/ceph/ceph.client.openstack.keyring'

我这里就统一对openstack账号授权，官方建议是每个服务创建一个账号。将ceph.conf和ceph.client.openstack.keyring这两个文件拷贝到需要对接Ceph的Nova-compute和Cinder-volume节点的/etc/ceph目录下

• 配置Libvirt

libvirt这块可以参考之前的文章《Ceph in Libvirt and Kubernetes》,为保证平台一致性，最好同步所有计算节点secret的UUID。

配置Cinder

cinder

在配置文件cinder.conf结尾追加一下配置

[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
volume_clear_size = 100
rbd_user = openstack
rbd_secret_uuid = 881e72de-961f-4556-9e8c-0b909408186b #libvirt的secret id

同时修改cinder.conf中的

enabled_backends = ceph

nova

为了挂载 Cinder 块设备（块设备或者启动卷），必须告诉 Nova 挂载设备时使用的用户和 uuid 。libvirt会使用该用户来和 Ceph 集群进行连接和认证。

rbd_user = openstack
rbd_secret_uuid = 881e72de-961f-4556-9e8c-0b909408186b

配置Nova

ceph

客户端配置文件ceph.conf开启RBD缓存和套接字，对于故障排查来说大有好处，给每个使用 Ceph 块设备的虚拟机分配一个套接字有助于调查性能和/或异常行为。

[client]
        rbd cache = true
        rbd cache writethrough until flush = true
        admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok
        log file = /var/log/qemu/qemu-guest-$pid.log
        rbd concurrent management ops = 20

调整目录权限

mkdir -p /var/run/ceph/guests/ /var/log/qemu/
chown qemu:libvirtd /var/run/ceph/guests /var/log/qemu/

nova

调整nova.conf文件的[libvirt]域下配置

#确保热迁移能顺利进行，要使用如下标志
live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED"
#禁止文件注入
inject_password = false
inject_key = false
inject_partition = -2
disk_cachemodes ="network=writeback"
images_type=rbd
images_rbd_pool=instances
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user=openstack
rbd_secret_uuid=881e72de-961f-4556-9e8c-0b909408186b

玩法

• 通过镜像创建Cinder卷

usage: cinder create [--consisgroup-id <consistencygroup-id>]
                     [--snapshot-id <snapshot-id>]
                     [--source-volid <source-volid>]
                     [--source-replica <source-replica>]
                     [--image-id <image-id>] [--image <image>] [--name <name>]
                     [--description <description>]
                     [--volume-type <volume-type>]
                     [--availability-zone <availability-zone>]
                     [--metadata [<key=value> [<key=value> ...]]]
                     [--hint <key=value>] [--allow-multiattach]
                     [<size>]


$ cinder create --image-id a02c0829-b198-4650-a9c6-7cc6b0b94018 --name jcloud.v1.0 60 
+--------------------------------------+-------------+--------------------+------+-------------+----------+--------------------------------------+
|                  ID                  |    Status   |        Name        | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-------------+--------------------+------+-------------+----------+--------------------------------------+
| ed63922f-f88c-4dab-9e8d-9670db8ee7b2 | downloading |    jcloud.v1.0     |  60  |      -      |  false   |                                      |
+--------------------------------------+-------------+--------------------+------+-------------+----------+--------------------------------------+
可以看到刚刚创建的卷，会从glance中下线镜像再导入到rbd块中，等导入完成之后这个卷就变成可一个引导的卷了，如下显示：
+--------------------------------------+-----------+--------------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  |        Name        | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+--------------------+------+-------------+----------+--------------------------------------+
| ed63922f-f88c-4dab-9e8d-9670db8ee7b2 | available |    jcloud.v1.0     |  60  |      -      |   true   |                                      |
+--------------------------------------+-----------+--------------------+------+-------------+----------+--------------------------------------+

• 对镜像做快照

usage: cinder snapshot-create [--force [<True|False>]] [--name <name>]
                              [--description <description>]
                              [--metadata [<key=value> [<key=value> ...]]]
                              <volume>

$  cinder snapshot-create --name jcloud.v1.0_snapshot ed63922f-f88c-4dab-9e8d-9670db8ee7b2
+-------------+--------------------------------------+
|   Property  |                Value                 |
+-------------+--------------------------------------+
|  created_at |      2017-03-22T07:52:28.887130      |
| description |                 None                 |
|      id     | 6be98fec-bfee-4550-99a6-358e2b8b6609 |
|   metadata  |                  {}                  |
|     name    |         jcloud.v1.0_snapshot         |
|     size    |                  60                  |
|    status   |               creating               |
|  updated_at |                 None                 |
|  volume_id  | ed63922f-f88c-4dab-9e8d-9670db8ee7b2 |
+-------------+--------------------------------------+

• 通过快照创建虚拟机

usage: nova boot [--flavor <flavor>] [--image <image>]
                 [--image-with <key=value>] [--boot-volume <volume_id>]
                 [--snapshot <snapshot_id>] [--min-count <number>]
                 [--max-count <number>] [--meta <key=value>]
                 [--file <dst-path=src-path>] [--key-name <key-name>]
                 [--user-data <user-data>]
                 [--availability-zone <availability-zone>]
                 [--security-groups <security-groups>]
                 [--block-device-mapping <dev-name=mapping>]
                 [--block-device key1=value1[,key2=value2...]]
                 [--swap <swap_size>]
                 [--ephemeral size=<size>[,format=<format>]]
                 [--hint <key=value>]
                 [--nic <net-id=net-uuid,net-name=network-name,v4-fixed-ip=ip-addr,v6-fixed-ip=ip-addr,port-id=port-uuid>]
                 [--config-drive <value>] [--poll] [--admin-pass <value>]
                 [--access-ip-v4 <value>] [--access-ip-v6 <value>]
                 <name>

--block-device 参数

• source=images|napshot|volume|blank

• dest=volume|local

• id=XXXXXX (a volume|image|snapshot UUID if using source=volume|snapshot|image)

• format=swap|ext4|...|none (to format the image/volume/ephemeral file; defaults to 'none' if omitted)

• bus=ide|usb|virtio|scsi (hypervisor driver chooses a suitable default if omitted)

• device=the desired device name (e.g. /dev/vda, /dev/xda, ...)

• type=disk|cdrom|floppy|mmc (defaults to 'disk' if omitted)

• bootindex=N (where N is any number >= 0, controls the order in which disks are looked at for booting)

• size=NN (where NN is number of GB to create type=emphemeral image, or the size to re-size to for type=glance|cinder)

• shutdown=preserve|remove

这里面只有 source 和 id 是必须的，别的都有默认值。比如：

• --block-device source=image,dest=volume,id=XXXXXXX,bus=ide,bootindex=2

• --block-device source=volume,dest=volume,id=XXXXXXX,bus=ide,type=cdrom,bootdex=1

• --block-device source=blank,dest=local,format=swap,size=50,bus=ide,type=floppy

dest 会指定source 的 destination，包括本地的（local）和 Cinder 卷 （volume）。

通过快照创建虚拟机

nova boot --flavor ff29c42b-754d-4230-9e1f-9bdaba800f5e --snapshot 6be98fec-bfee-4550-99a6-358e2b8b6609 --security-groups default --nic net-id=163df3b0-13f2-4f2e-8401-e82088e8dc07 test

#或者

nova boot --flavor ff29c42b-754d-4230-9e1f-9bdaba800f5e --block-device source=snapshot,dest=volume,id=6be98fec-bfee-4550-99a6-358e2b8b6609,bootindex=0 --security-groups default --nic net-id=163df3b0-13f2-4f2e-8401-e82088e8dc07  test

跟踪虚拟机状态如下

#映射设备
+--------------------------------------+-------+--------+----------------------+-------------+--------------------------+
| ID                                   | Name  | Status | Task State           | Power State | Networks                 |
+--------------------------------------+-------+--------+----------------------+-------------+--------------------------+
| 52369db0-f7db-4eb0-a708-d77e250e3ecc | test  | BUILD  | block_device_mapping | NOSTATE     | privite01=192.168.17.251 |
+--------------------------------------+-------+--------+----------------------+-------------+--------------------------+

#孵化
+--------------------------------------+-------+--------+------------+-------------+--------------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks                 |
+--------------------------------------+-------+--------+------------+-------------+--------------------------+
| 52369db0-f7db-4eb0-a708-d77e250e3ecc | test  | BUILD  | spawning   | NOSTATE     | privite01=192.168.17.251 |
+--------------------------------------+-------+--------+------------+-------------+--------------------------+

#启动成功
+--------------------------------------+-------+--------+------------+-------------+--------------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks                 |
+--------------------------------------+-------+--------+------------+-------------+--------------------------+
| 52369db0-f7db-4eb0-a708-d77e250e3ecc | test  | ACTIVE | -          | Running     | privite01=192.168.17.251 |
+--------------------------------------+-------+--------+------------+-------------+--------------------------+

快照创建成功后我们可以通过rbd命令去ceph的pool里面查询volumes状态。

• 首先看下cinder-volume的挂载

+--------------------------------------+-----------+--------------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  |        Name        | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+--------------------+------+-------------+----------+--------------------------------------+
| 7398e12b-333b-4610-b75c-e237d164781d |   in-use  |                    |  60  |      -      |   true   | 52369db0-f7db-4eb0-a708-d77e250e3ecc |
+--------------------------------------+-----------+--------------------+------+-------------+----------+--------------------------------------+

7398e12b-333b-4610-b75c-e237d164781d这个便是虚拟机系统盘所用的volumes了，有了这个uuid就可以在底层Ceph上对应上rbd块了。

• 查下系统卷的信息

#根据volume的uuid查找rbd块
$ rbd ls volumes --name client.libvirt |grep 7398e12b-333b-4610-b75c-e237d164781d
volume-7398e12b-333b-4610-b75c-e237d164781d

#查看块信息
$ rbd  info volumes/volume-7398e12b-333b-4610-b75c-e237d164781d --name client.libvirt
rbd image 'volume-7398e12b-333b-4610-b75c-e237d164781d':
    size 61440 MB in 15360 objects
    order 22 (4096 kB objects)
    block_name_prefix: rbd_data.872a41ece775
    format: 2
    features: layering, striping
    flags: 
    parent: volumes/volume-ed63922f-f88c-4dab-9e8d-9670db8ee7b2@snapshot-6be98fec-bfee-4550-99a6-358e2b8b6609
    overlap: 61440 MB
    stripe unit: 4096 kB
    stripe count: 1

#快照的信息  
$ rbd  info volumes/volume-ed63922f-f88c-4dab-9e8d-9670db8ee7b2@snapshot-6be98fec-bfee-4550-99a6-358e2b8b6609 --name client.libvirt
rbd image 'volume-ed63922f-f88c-4dab-9e8d-9670db8ee7b2':
    size 61440 MB in 15360 objects
    order 22 (4096 kB objects)
    block_name_prefix: rbd_data.86c4238e1f29
    format: 2
    features: layering
    flags: 
    protected: True

这里可以看到volume-7398e12b-333b-4610-b75c-e237d164781d实际上是通过volumes/volume-ed63922f-f88c-4dab-9e8d-9670db8ee7b2@snapshot-6be98fec-bfee-4550-99a6-358e2b8b6609这个快照克隆而来。cinder-volumes
服务在处理快照前已经对其做了protected保护，那么按照这个特性，之后创建虚拟机或者批量创建虚拟机就可以通过这个快照克隆而来。我这里测试了下确实是可行的。

通过快照批量创建虚拟机

$ nova boot --flavor ff29c42b-754d-4230-9e1f-9bdaba800f5e --block-device source=snapshot,dest=volume,id=6be98fec-bfee-4550-99a6-358e2b8b6609,bootindex=0 --security-groups default --nic net-id=163df3b0-13f2-4f2e-8401-e82088e8dc07 --min-count 2 --max-count 3  test

对于--min-count和--max-count的处理逻辑是，在批量创建虚拟机是，nova会根据--max-count的数量计算需要的配额，如果超过配额就按照--min-count的数量计算分配，如果仍然超过配额就返回失败。

创建成功后就如下

#nova 虚拟机状态
+--------------------------------------+--------+--------+------------+-------------+--------------------------+
| ID                                   | Name   | Status | Task State | Power State | Networks                 |
+--------------------------------------+--------+--------+------------+-------------+--------------------------+
| 66e5bf31-6ad0-496b-9dbc-77a1c2b8fa32 | test-1 | ACTIVE | -          | Running     | privite01=192.168.17.246 |
| 6ebad2a4-b844-4662-9b1b-862a726c1873 | test-2 | ACTIVE | -          | Running     | privite01=192.168.17.248 |
| 5f5d7088-cfb3-4e8a-9b38-7e128b4fc8eb | test-3 | ACTIVE | -          | Running     | privite01=192.168.17.249 |
+--------------------------------------+--------+--------+------------+-------------+--------------------------+

#cinder 数据卷状态
+--------------------------------------+-----------+--------------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  |        Name        | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+--------------------+------+-------------+----------+--------------------------------------+
| 77835304-1324-4063-9d25-211065fe406f |   in-use  |                    |  60  |      -      |   true   | 5f5d7088-cfb3-4e8a-9b38-7e128b4fc8eb |
| 85f66862-1fad-4634-b96f-cdb51558c422 |   in-use  |                    |  60  |      -      |   true   | 6ebad2a4-b844-4662-9b1b-862a726c1873 |
| b457befb-490a-402e-8b6f-215945ab6248 |   in-use  |                    |  60  |      -      |   true   | 66e5bf31-6ad0-496b-9dbc-77a1c2b8fa32 |
| ed63922f-f88c-4dab-9e8d-9670db8ee7b2 | available |    jcloud.v1.0     |  60  |      -      |   true   |                                      |
+--------------------------------------+-----------+--------------------+------+-------------+----------+--------------------------------------+

#ceph卷信息
$ rbd ls volumes --name client.libvirt
volume-77835304-1324-4063-9d25-211065fe406f
volume-85f66862-1fad-4634-b96f-cdb51558c422
volume-b457befb-490a-402e-8b6f-215945ab6248
volume-ed63922f-f88c-4dab-9e8d-9670db8ee7b2

本文参考：
http://docs.ceph.org.cn/
http://www.cnblogs.com/sammyliu/p/4462718.html