1. RGW Storage Class用法
在ceph和对象存储中，桶（bucket）是用户侧的概念，就是说客户使用对象存储服务的方式一般是先创建一个桶，把数据存放在对象存储的桶上，访问的时候也是通过指定桶的名字来获取相应的数据。言而，从ceph的角度来说，ceph是以存储池（pool）的形式对外提供服务。用户把一个文件上传到桶，它的数据是如何分布在存储池上的，中间涉及到一个placement的概念，它负责处理桶和存储池的映射关系，将不同storage class的数据分配到相应的存储池上。除了placement，在ceph RGW中还有placement_rule和placement_target这一说法，它们的异同在这里不做深入讨论，本文暂且将它们作为同一个概念看待。
在RGW上可以为placement-rule设置stroage class，如果不设置默认会有一个STANDARD的storage class，对应该类型的对象会存放在唯一的data pool上。用户使用S3协议上传对象的时候，通过设置http的x-amz-storage-class header字段来指定storage class，如果不指定该header，则RGW默认其为storage class为STANDARD。
1.1 自定义Storage Class
如果需要设置额外的storage class，例如创建一个名为COLD的storage class，可以通过以下步骤实现。
-
首先创建一个pool用于存放COLD类型的对象：
store@ceph1-1 ~$ sudo ceph osd pool create default.rgw.buckets.data3 32 32 pool 'default.rgw.buckets.data3' created这时候集群的pool情况如下：
store@ceph1-1 ~$ sudo ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 270 GiB 259 GiB 1.8 GiB 11 GiB 3.99 TOTAL 270 GiB 259 GiB 1.8 GiB 11 GiB 3.99 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .rgw.root 1 32 6.8 KiB 21 3.8 MiB 0 82 GiB default.rgw.control 2 32 0 B 8 0 B 0 82 GiB default.rgw.meta 3 32 2.4 KiB 10 1.5 MiB 0 82 GiB default.rgw.log 4 32 14 KiB 210 398 KiB 0 82 GiB default.rgw.buckets.index 5 32 39 KiB 1 39 KiB 0 82 GiB default.rgw.buckets.data 6 32 125 MiB 45 375 MiB 0.15 82 GiB default.rgw.buckets.non-ec 7 32 492 B 2 384 KiB 0 82 GiB default.rgw.buckets.indexless 9 32 0 B 0 0 B 0 82 GiB default.rgw.buckets.data2 10 32 16 MiB 5 48 MiB 0.02 82 GiB default.rgw.buckets.data3 11 32 0 B 0 0 B 0 82 GiB -
为placement-target添加COLD的storage class：
storage class作为placement-target的属性之一，在创建COLD这个storage class之前需要先创建一个placement_target，也可以用现有的placement_target。由于集群一般会有一个默认的placement_target，方便起见，这里直接用它。当前集群的zonegroup配置如下：store@ceph1-1 ~$ sudo radosgw-admin zonegroup get { "id": "cfef7aa0-adce-4d00-b314-20be40c76013", "name": "default-zonegroup", "api_name": "default-zonegroup", "is_master": "true", "endpoints": [ "http://192.168.5.11:8080" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "70a9dd7a-e126-4ee2-b803-79b433bf0896", "zones": [ { "id": "70a9dd7a-e126-4ee2-b803-79b433bf0896", "name": "default", "endpoints": [ "http://192.168.5.11:8080", "http://192.168.5.12:8080" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] }, { "name": "indexless-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "3b4d8a22-8b32-4e2a-b54d-a9a49a564a24" }为名为default-placement的placement_target添加COLD的storage class：
store@ceph1-1 ~$ sudo radosgw-admin zonegroup placement add --rgw-zonegroup default-zonegroup --placement-id default-placement --storage-class COLD [ { "key": "default-placement", "val": { "name": "default-placement", "tags": [], "storage_classes": [ "COLD", "STANDARD" ] } }, { "key": "indexless-placement", "val": { "name": "indexless-placement", "tags": [], "storage_classes": [ "STANDARD" ] } } ]添加完之后的zone情况如下：
store@ceph1-1 ~$ sudo radosgw-admin zone get { "id": "70a9dd7a-e126-4ee2-b803-79b433bf0896", "name": "default", "domain_root": "default.rgw.meta:root", "control_pool": "default.rgw.control", "gc_pool": "default.rgw.log:gc", "lc_pool": "default.rgw.log:lc", "log_pool": "default.rgw.log", "intent_log_pool": "default.rgw.log:intent", "usage_log_pool": "default.rgw.log:usage", "reshard_pool": "default.rgw.log:reshard", "user_keys_pool": "default.rgw.meta:users.keys", "user_email_pool": "default.rgw.meta:users.email", "user_swift_pool": "default.rgw.meta:users.swift", "user_uid_pool": "default.rgw.meta:users.uid", "otp_pool": "default.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "default.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "default.rgw.buckets.data" } }, "data_extra_pool": "default.rgw.buckets.non-ec", "index_type": 0 } }, { "key": "indexless-placement", "val": { "index_pool": "default.rgw.buckets.indexless", "storage_classes": { "STANDARD": { "data_pool": "default.rgw.buckets.data2" } }, "data_extra_pool": "default.rgw.buckets.non-ec", "index_type": 1 } } ], "metadata_heap": "", "realm_id": "3b4d8a22-8b32-4e2a-b54d-a9a49a564a24" }可以看到虽然COLD这个storage class是添加到default-placement了，但是由于没有为其指定data pool，所以在zone上还看不见它的信息。而在这里可以发现STANDARD的storage class及其对应的data pool：default.rgw.buckets.data。
-
为COLD这个storage class指定其data pool：
store@ceph1-1 ~$ sudo radosgw-admin zone placement add --rgw-zonegroup default-zonegroup --placement-id default-placement --storage-class COLD --data-pool default.rgw.buckets.data3 --compression lz4 { "id": "70a9dd7a-e126-4ee2-b803-79b433bf0896", "name": "default", "domain_root": "default.rgw.meta:root", "control_pool": "default.rgw.control", "gc_pool": "default.rgw.log:gc", "lc_pool": "default.rgw.log:lc", "log_pool": "default.rgw.log", "intent_log_pool": "default.rgw.log:intent", "usage_log_pool": "default.rgw.log:usage", "reshard_pool": "default.rgw.log:reshard", "user_keys_pool": "default.rgw.meta:users.keys", "user_email_pool": "default.rgw.meta:users.email", "user_swift_pool": "default.rgw.meta:users.swift", "user_uid_pool": "default.rgw.meta:users.uid", "otp_pool": "default.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "default.rgw.buckets.index", "storage_classes": { "COLD": { "data_pool": "default.rgw.buckets.data3", "compression_type": "lz4" }, "STANDARD": { "data_pool": "default.rgw.buckets.data" } }, "data_extra_pool": "default.rgw.buckets.non-ec", "index_type": 0 } }, { "key": "indexless-placement", "val": { "index_pool": "default.rgw.buckets.indexless", "storage_classes": { "STANDARD": { "data_pool": "default.rgw.buckets.data2" } }, "data_extra_pool": "default.rgw.buckets.non-ec", "index_type": 1 } } ], "metadata_heap": "", "realm_id": "3b4d8a22-8b32-4e2a-b54d-a9a49a564a24" }最后保存配置变更：
store@ceph1-1 ~$ sudo radosgw-admin period update --commit
1.2 指定对象的Storage Class
使用S3上传对象（PUT Object, POST Object, Initiate Multipart Upload）的时候，可以通过请求的x-amz-storage-class header来指定上传对象的storage class。以AWS S3 Go SDK为例，上传一个14.2MB的文件，并指定其storage class为COLD，代码如下：
// 上传文件
func PutObject(svc *s3.S3) {
bucket := "test01-bucket1"
key := "flower.zip"
filePath := "D:/flower.zip"
file, err := os.Open(filePath)
if err != nil {
fmt.Printf("Unable to open file:%v", err)
os.Exit(1)
}
defer func() {
err := file.Close()
if err != nil {
fmt.Printf("fail to close file. %v\n", err)
}
}()
putObjectInput := &s3.PutObjectInput{
Body: file,
Bucket: aws.String(bucket),
Key: aws.String(key),
StorageClass: aws.String("COLD"), //指定存储类别
}
putObjectOutput, err := svc.PutObject(putObjectInput)
if err != nil {
fmt.Printf("Failed to put object %q, %v", key, err)
} else {
fmt.Println(putObjectOutput)
}
}
flower.zip这个文件所在的桶是test01-bucket1，该桶对应的default-placement有两个data pool，分别是default.rgw.buckets.data和default.rgw.buckets.data3，前者用于存放STANDARD类型的对象，后者用于存放COLD类型的对象。
遍历default.rgw.buckets.data这个pool的对象可以发现flower.zip这个文件的头对象是在这个pool：
store@ceph1-1 ~$ sudo rados -p default.rgw.buckets.data ls
bef1cd7f-58ed-439e-af4e-1db304368eaf.104181.1157__shadow_mymultipart.2~hUckQCJ6IpGxov_xo6V-RlnmO48m_Zw.1_1
70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338_flower2.txt
bef1cd7f-58ed-439e-af4e-1db304368eaf.104181.289__shadow_mymultipart.2~NoisJXP5-qnRqiDWdJ5pU-UyP004nij.1_1
70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338_flower.zip //flower.zip文件的头对象
70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338_flower3.txt
70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2038__shadow_mymultipart.2~jcjAXuInO4EgYHYrtf3usPaAZT-Jvpe.1_1
70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2098__shadow_mymultipart.2~EqpqCdGuvsW0Fi0JBq90gMzmjn0C2Wl.1_1
查看它的manifest：
store@ceph1-1 ~$ sudo rados -p default.rgw.buckets.data getxattr 70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338_flower.zip user.rgw.manifest > flower.zip.manifest
store@ceph1-1 ~$ sudo ceph-dencoder type RGWObjManifest import flower.zip.manifest decode dump_json
{
"objs": [],
"obj_size": 14963287,
"explicit_objs": "false",
"head_size": 0,
"max_head_size": 0,
"prefix": ".0jDvc6Ooiyf9z0muHicflmfxxDJOuQx_",
"rules": [
{
"key": 0,
"val": {
"start_part_num": 0,
"start_ofs": 0,
"part_size": 0,
"stripe_max_size": 4194304,
"override_prefix": ""
}
}
],
"tail_instance": "",
"tail_placement": {
"bucket": {
"name": "test01-bucket1",
"marker": "70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338",
"bucket_id": "70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338",
"tenant": "",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
}
},
"placement_rule": "default-placement/COLD"
},
"begin_iter": {
"part_ofs": 0,
"stripe_ofs": 0,
"ofs": 0,
"stripe_size": 4194304,
"cur_part_id": 0,
"cur_stripe": 0,
"cur_override_prefix": "",
"location": {
"placement_rule": "default-placement/COLD",
"obj": {
"bucket": {
"name": "test01-bucket1",
"marker": "70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338",
"bucket_id": "70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338",
"tenant": "",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
}
},
"key": {
"name": ".0jDvc6Ooiyf9z0muHicflmfxxDJOuQx_0",
"instance": "",
"ns": "shadow"
}
},
"raw_obj": {
"pool": "",
"oid": "",
"loc": ""
},
"is_raw": false
}
},
"end_iter": {
"part_ofs": 0,
"stripe_ofs": 12582912,
"ofs": 14963287,
"stripe_size": 2380375,
"cur_part_id": 0,
"cur_stripe": 3,
"cur_override_prefix": "",
"location": {
"placement_rule": "default-placement/COLD",
"obj": {
"bucket": {
"name": "test01-bucket1",
"marker": "70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338",
"bucket_id": "70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338",
"tenant": "",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
}
},
"key": {
"name": ".0jDvc6Ooiyf9z0muHicflmfxxDJOuQx_3",
"instance": "",
"ns": "shadow"
}
},
"raw_obj": {
"pool": "",
"oid": "",
"loc": ""
},
"is_raw": false
}
}
}
再遍历default.rgw.buckets.data3这个pool的对象就能找到flower.zip文件对应的数据部分的对象：
store@ceph1-1 ~$ sudo rados -p default.rgw.buckets.data3 ls
70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338__shadow_.0jDvc6Ooiyf9z0muHicflmfxxDJOuQx_3
70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338__shadow_.-r6yit2qf8P0cQNSAY1ZGpo_ewZwS2T_0
70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338__shadow_.0jDvc6Ooiyf9z0muHicflmfxxDJOuQx_0
70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338__shadow_.0jDvc6Ooiyf9z0muHicflmfxxDJOuQx_1
70a9dd7a-e126-4ee2-b803-79b433bf0896.134376.2338__shadow_.0jDvc6Ooiyf9z0muHicflmfxxDJOuQx_2
可以看到default.rgw.buckets.data3中带有0jDvc6Ooiyf9z0muHicflmfxxDJOuQx标识的对象就是flower.zip文件对应的数据部分的对象，该标识的内容和头对象manifest中的prefix一致，说明flower.zip这个文件作为COLD类型，其数据确实是存放在了storage class为COLD的data pool上。
1.3 只有一种Storage Class的placement
如果由于业务需要，例如在分层存储的情况下，集群会由不同的存储介质（固态硬盘、机械硬盘、磁带等）组成，这时候不同的集群存储池的存储类别会不一样，这里需要考虑的问题是能不能指定一个存储池的storage class为非STANDARD的，这个存储池还能单独地对外提供服务？
这里可以尝试为default.rgw.buckets.data3这个存储池建立一个placement，只让其的storage class为HOT。
先建立placement，并执行其storage class为HOT：
store@ceph1-1 ~$ sudo radosgw-admin zonegroup placement add --placement-id=hot-placement --storage-class HOT
[
{
"key": "default-placement",
"val": {
"name": "default-placement",
"tags": [],
"storage_classes": [
"COLD",
"STANDARD"
]
}
},
{
"key": "hot-placement",
"val": {
"name": "hot-placement",
"tags": [],
"storage_classes": [
"HOT"
]
}
},
{
"key": "indexless-placement",
"val": {
"name": "indexless-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
}
]
这里可以看见创建出来的hot-placement只有一种HOT的storage class，没有了STANDARD。
这时候zonegroup的情况是这样的：
store@ceph1-1 ~$ sudo radosgw-admin zonegroup get
{
"id": "cfef7aa0-adce-4d00-b314-20be40c76013",
"name": "default-zonegroup",
"api_name": "default-zonegroup",
"is_master": "true",
"endpoints": [
"http://192.168.5.11:8080"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "70a9dd7a-e126-4ee2-b803-79b433bf0896",
"zones": [
{
"id": "70a9dd7a-e126-4ee2-b803-79b433bf0896",
"name": "default",
"endpoints": [
"http://192.168.5.11:8080",
"http://192.168.5.12:8080"
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": [],
"storage_classes": [
"COLD",
"STANDARD"
]
},
{
"name": "hot-placement",
"tags": [],
"storage_classes": [
"HOT"
]
},
{
"name": "indexless-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
],
"default_placement": "default-placement",
"realm_id": "3b4d8a22-8b32-4e2a-b54d-a9a49a564a24"
}
zone的情况是这样的：
store@ceph1-1 ~$ sudo radosgw-admin zone get
{
"id": "70a9dd7a-e126-4ee2-b803-79b433bf0896",
"name": "default",
"domain_root": "default.rgw.meta:root",
"control_pool": "default.rgw.control",
"gc_pool": "default.rgw.log:gc",
"lc_pool": "default.rgw.log:lc",
"log_pool": "default.rgw.log",
"intent_log_pool": "default.rgw.log:intent",
"usage_log_pool": "default.rgw.log:usage",
"reshard_pool": "default.rgw.log:reshard",
"user_keys_pool": "default.rgw.meta:users.keys",
"user_email_pool": "default.rgw.meta:users.email",
"user_swift_pool": "default.rgw.meta:users.swift",
"user_uid_pool": "default.rgw.meta:users.uid",
"otp_pool": "default.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"COLD": {
"data_pool": "default.rgw.buckets.data3",
"compression_type": "lz4"
},
"STANDARD": {
"data_pool": "default.rgw.buckets.data"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "indexless-placement",
"val": {
"index_pool": "default.rgw.buckets.indexless",
"storage_classes": {
"STANDARD": {
"data_pool": "default.rgw.buckets.data2"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 1
}
}
],
"metadata_heap": "",
"realm_id": "3b4d8a22-8b32-4e2a-b54d-a9a49a564a24"
}
然后为hot-placement这个placement设定data pool：
store@ceph1-1 ~$ sudo radosgw-admin zone placement add --rgw-zonegroup default-zonegroup --placement-id hot-placement --storage-class HOT --index-pool default.rgw.buckets.index --data-extra-pool=default.rgw.buckets.non-ec --data-pool default.rgw.buckets.data3
{
"id": "70a9dd7a-e126-4ee2-b803-79b433bf0896",
"name": "default",
"domain_root": "default.rgw.meta:root",
"control_pool": "default.rgw.control",
"gc_pool": "default.rgw.log:gc",
"lc_pool": "default.rgw.log:lc",
"log_pool": "default.rgw.log",
"intent_log_pool": "default.rgw.log:intent",
"usage_log_pool": "default.rgw.log:usage",
"reshard_pool": "default.rgw.log:reshard",
"user_keys_pool": "default.rgw.meta:users.keys",
"user_email_pool": "default.rgw.meta:users.email",
"user_swift_pool": "default.rgw.meta:users.swift",
"user_uid_pool": "default.rgw.meta:users.uid",
"otp_pool": "default.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"COLD": {
"data_pool": "default.rgw.buckets.data3",
"compression_type": "lz4"
},
"STANDARD": {
"data_pool": "default.rgw.buckets.data"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "hot-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"HOT": {
"data_pool": "default.rgw.buckets.data3"
},
"STANDARD": {}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "indexless-placement",
"val": {
"index_pool": "default.rgw.buckets.indexless",
"storage_classes": {
"STANDARD": {
"data_pool": "default.rgw.buckets.data2"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 1
}
}
],
"metadata_heap": "",
"realm_id": "3b4d8a22-8b32-4e2a-b54d-a9a49a564a24"
}
此时zone的情况：
store@ceph1-1 ~$ sudo radosgw-admin zone get
{
"id": "70a9dd7a-e126-4ee2-b803-79b433bf0896",
"name": "default",
"domain_root": "default.rgw.meta:root",
"control_pool": "default.rgw.control",
"gc_pool": "default.rgw.log:gc",
"lc_pool": "default.rgw.log:lc",
"log_pool": "default.rgw.log",
"intent_log_pool": "default.rgw.log:intent",
"usage_log_pool": "default.rgw.log:usage",
"reshard_pool": "default.rgw.log:reshard",
"user_keys_pool": "default.rgw.meta:users.keys",
"user_email_pool": "default.rgw.meta:users.email",
"user_swift_pool": "default.rgw.meta:users.swift",
"user_uid_pool": "default.rgw.meta:users.uid",
"otp_pool": "default.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"COLD": {
"data_pool": "default.rgw.buckets.data3",
"compression_type": "lz4"
},
"STANDARD": {
"data_pool": "default.rgw.buckets.data"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "hot-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"HOT": {
"data_pool": "default.rgw.buckets.data3"
},
"STANDARD": {}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "indexless-placement",
"val": {
"index_pool": "default.rgw.buckets.indexless",
"storage_classes": {
"STANDARD": {
"data_pool": "default.rgw.buckets.data2"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 1
}
}
],
"metadata_heap": "",
"realm_id": "3b4d8a22-8b32-4e2a-b54d-a9a49a564a24"
}
从zone的情况可以看到虽然没有没有为hot-placement设置STANDARD这个storage class，但它还是出现了，只不过它没有data pool。
保存一下配置变更：
store@ceph1-1 ~$ sudo radosgw-admin period update --commit
接下来就是验证可用性的过程，同样还是使用AWS S3 Go SDK，
先用以下代码创建一个桶test01-bucket2，注意这里设置了LocationConstraint属性，意思就是创建的这个桶使用的placement是hot-placement。
func CreateBucketRequest(svc *s3.S3) {
createBucketInput := &s3.CreateBucketInput{
Bucket: aws.String("test01-bucket2"),
CreateBucketConfiguration: &s3.CreateBucketConfiguration{
LocationConstraint: aws.String(":hot-placement"),
},
}
req, resp := svc.CreateBucketRequest(createBucketInput)
err := req.Send()
if err != nil {
fmt.Printf("fail to create bucket. %v\n", err)
} else {
fmt.Println(resp)
}
}
上传一个文件并指定它的storage class为HOT：
func PutObject(svc *s3.S3) {
bucket := "test01-bucket2"
key := "flower.zip"
filePath := "D:/flower.zip"
file, err := os.Open(filePath)
if err != nil {
fmt.Printf("Unable to open file:%v", err)
os.Exit(1)
}
defer func() {
err := file.Close()
if err != nil {
fmt.Printf("fail to close file. %v\n", err)
}
}()
putObjectInput := &s3.PutObjectInput{
Body: file,
Bucket: aws.String(bucket),
Key: aws.String(key),
StorageClass: aws.String("HOT"),
}
putObjectOutput, err := svc.PutObject(putObjectInput)
if err != nil {
fmt.Printf("Failed to put object %q, %v", key, err)
} else {
fmt.Println(putObjectOutput)
}
}
显示结果为400 Bad Request
-----------------------------------------------------
2021/09/10 17:22:30 DEBUG: Response s3/PutObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Content-Length: 239
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/xml
Date: Fri, 10 Sep 2021 09:22:30 GMT
X-Amz-Request-Id: tx000000000000000000003-00613b23d6-21c33-default
-----------------------------------------------------
Failed to put object "flower.zip", InvalidArgument:
status code: 400, request id: tx000000000000000000003-00613b23d6-21c33-default, host id: --- PASS: TestPutObject (0.63s)
从测试的结果可以知道，如果一条placement没有设置STANDARD的storage class，是不能被正常使用的。
那么这里就问题了，如果一个存储池的存储介质是机械硬盘，它对应的存储类型是低频的，为了正常地使用这个集群，难道还需要为它搭建一个storage class为STANDARD的存储池吗？先说结论，是不用的！具体做法看下文。
1.4 多个storage class使用同一个data pool
为hot-placement的STANDARD storage class修改data pool：
store@ceph1-1 ~$ sudo radosgw-admin zone placement modify --rgw-zonegroup default-zonegroup --placement-id hot-placement --storage-class STANDARD --data-pool default.rgw.buckets.data3
ERROR: storage class 'STANDARD' is not defined in zonegroup 'hot-placement' placement target
虽然在zone get的信息上显示有STANDARD这个storage class，但是直接修改它会报错。
还是要先在hot-placement上创建STANDARD这个storage class
store@ceph1-1 ~$ sudo radosgw-admin zonegroup placement add --rgw-zonegroup default-zonegroup --placement-id hot-placement --storage-class STANDARD
[
{
"key": "default-placement",
"val": {
"name": "default-placement",
"tags": [],
"storage_classes": [
"COLD",
"STANDARD"
]
}
},
{
"key": "hot-placement",
"val": {
"name": "hot-placement",
"tags": [],
"storage_classes": [
"HOT",
"STANDARD"
]
}
},
{
"key": "indexless-placement",
"val": {
"name": "indexless-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
}
]
然后再为hot-placement的STANDARD这个storage class指定data-pool，在这里同样使用default.rgw.buckets.data3这个pool。
store@ceph1-1 ~$ sudo radosgw-admin zone placement modify --rgw-zonegroup default-zonegroup --placement-id hot-placement --storage-class STANDARD --data-pool default.rgw.buckets.data3
{
"id": "70a9dd7a-e126-4ee2-b803-79b433bf0896",
"name": "default",
"domain_root": "default.rgw.meta:root",
"control_pool": "default.rgw.control",
"gc_pool": "default.rgw.log:gc",
"lc_pool": "default.rgw.log:lc",
"log_pool": "default.rgw.log",
"intent_log_pool": "default.rgw.log:intent",
"usage_log_pool": "default.rgw.log:usage",
"reshard_pool": "default.rgw.log:reshard",
"user_keys_pool": "default.rgw.meta:users.keys",
"user_email_pool": "default.rgw.meta:users.email",
"user_swift_pool": "default.rgw.meta:users.swift",
"user_uid_pool": "default.rgw.meta:users.uid",
"otp_pool": "default.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"COLD": {
"data_pool": "default.rgw.buckets.data3",
"compression_type": "lz4"
},
"STANDARD": {
"data_pool": "default.rgw.buckets.data"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "hot-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"HOT": {
"data_pool": "default.rgw.buckets.data3"
},
"STANDARD": {
"data_pool": "default.rgw.buckets.data3"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "indexless-placement",
"val": {
"index_pool": "default.rgw.buckets.indexless",
"storage_classes": {
"STANDARD": {
"data_pool": "default.rgw.buckets.data2"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 1
}
}
],
"metadata_heap": "",
"realm_id": "3b4d8a22-8b32-4e2a-b54d-a9a49a564a24"
}
最后commit上述变更
store@ceph1-1 ~$ sudo radosgw-admin period update --commit
再重新上传一个storage class为HOT的对象就会成功，输出结果为成功：
-----------------------------------------------------
2021/09/10 17:32:21 DEBUG: Response s3/PutObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Connection: Keep-Alive
Date: Fri, 10 Sep 2021 09:32:21 GMT
Etag: "8d4dcacecb59a93dbb3ea9d93713b1ad"
X-Amz-Request-Id: tx000000000000000000001-00613b2623-21c81-default
-----------------------------------------------------
{
ETag: "\"8d4dcacecb59a93dbb3ea9d93713b1ad\""
}
--- PASS: TestPutObject (2.11s)
1.5 结论
- 在Ceph RGW上创建placement的时候，如果不指定storage class，则默认会有一个STANDARD的storage class；
- 当一条placement_target有多个storage class的时候，如果创建的桶使用的是这条placment_target，那么上传到该桶的文件的头对象会存放在STANDARD storage class对应的data pool中，尾对象会存放在上传文件时指定的storage class对应的data pool中；
- 如果一条placement_target只有一种storage class，那么该种storage class一定要是STANDARD类型的；
- 在一条placement_target中，不同的storage class可以对应同一个data pool。通过将STANDARD storage class和其他storage class的data pool设置为同一个，从而使得storage class为非STANDARD的存储池也可以单独对外提供服务。
