1、S3cmd介绍
S3cmd是一个由Amazon提供的客户端命令行工具，用于用户操作支持s3访问协议的存储系统。它支持上传、检索、管理数据等功能，适合批量同步和调度中使用。
2、安装方式
- 方式1：yum install s3cmd
- 方式2：pip install s3cmd
3、使用方法
3.1 首先在存储端创建用户，记录AK,SK：
--uid 用户名自定义id
-- dispaly-name 自定义
[root@ceph_node1 cluster]# radosgw-admin user create --uid cjid --display-name cj
{
"user_id": "cjid",
"display_name": "cj",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "cjid",
"access_key": "9IU0G32UQE8JEN76PLFT",
"secret_key": "2mGNgKTBaTMAo5qkf6WXALMWJzw8V6GWAmyVgJvZ"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}
记住如下三个字段，如果没记住，可以采用如下命令查看
查看信息radosgw-admin user info --uid cjid
"user": "cjid",
"access_key": "9IU0G32UQE8JEN76PLFT",
"secret_key": "2mGNgKTBaTMAo5qkf6WXALMWJzw8V6GWAmyVgJvZ"
3.2 客户端更改配置文件
两种方式：
方式1：s3cmd --configure
执行s3cmd --configure，填写AK、SK、DNS等参数，其他的默认回车
[root@client1 ~]# s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 9IU0G32UQE8JEN76PLFT #需要输入radosgwuser的Access Key
Secret Key: 2mGNgKTBaTMAo5qkf6WXALMWJzw8V6GWAmyVgJvZ #需要输入radosgwuser的Secret Key
Default Region [US]: #保持默认回车，注：国家只支持US
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: 192.168.212.202 #输入rgw的ip
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: 192.168.212.202:80/%(bucket)s#输入rgw的ip和端口，注意格式是固定的
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: #默认回车，不启用加密
Path to GPG program [/usr/bin/gpg]: #默认回车
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: no #是否启用https，这里写 no ，因为没有提供 https
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: #是否设置代理，这里不用写，因为没有代理
New settings:
Access Key: 9IU0G32UQE8JEN76PLFT
Secret Key: 2mGNgKTBaTMAo5qkf6WXALMWJzw8V6GWAmyVgJvZ
Default Region: US
S3 Endpoint: 192.168.212.202
DNS-style bucket+hostname:port template for accessing a bucket: 192.168.212.202:80/%(bucket)s
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] y#测试访问
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y#保持配置
Configuration saved to '/root/.s3cfg'#配置保持路径
2、配置完成后，执行s3cmd ls查看桶是否创建成功
[root@client1 ~]# s3cmd ls
2022-03-18 06:05 s3://my-new-bucket
方式2 任意目录下配置文件
（建议使用，配置更加简洁）
vim s3cfg
[default]
access_key = 9IU0G32UQE8JEN76PLFT【更改ak】
secret_key = 2mGNgKTBaTMAo5qkf6WXALMWJzw8V6GWAmyVgJvZ 【更改sk】
host_base = 192.168.212.202:80 【更改访问地址和端口】
host_bucket = 192.168.212.202:80 /%(bucket)【更改访问地址和端口， /%(bucket)保持不变】
use_https = False
查看方式: s3cmd -c 配置文件名 ls
[root@test-env-nm05-compute-11e5e72e193 date4_8]# s3cmd -c s3cfg ls
2022-04-11 08:18 s3://bucket_28
2022-04-08 07:46 s3://bucket_2999
2022-04-08 07:58 s3://bucket_3001
2022-04-11 03:24 s3://bucket_411
2022-04-12 02:00 s3://bucket_mini
2022-04-12 01:15 s3://bucket_mode4
4、常用命令
查看桶
[root@client1 ~]# s3cmd ls
2022-03-18 06:05 s3://my-new-bucket
创建桶，报错
[root@client1 ~]# s3cmd mb s3://s3cmd-demo
ERROR: S3 error: 403 (SignatureDoesNotMatch)
没有启用signature_v2，需要启用
[root@client1 ~]# cat /root/.s3cfg |grep signature
signature_v2 = False
[root@client1 ~]# sed -i '/signature_v2/s/False/True/g' /root/.s3cfg
[root@client1 ~]# cat /root/.s3cfg |grep signature
signature_v2 = True
再次创建桶成功
[root@client1 ~]# s3cmd mb s3://s3cmd-demo
Bucket 's3://s3cmd-demo/' created
[root@client1 ~]#
[root@client1 ~]# s3cmd ls
2022-03-18 06:05 s3://my-new-bucket
2022-03-18 06:44 s3://s3cmd-demo
删除桶
[root@client1 ~]# s3cmd rb s3://s3cmd-demo
Bucket 's3://s3cmd-demo/' removed
[root@client1 ~]#
[root@client1 ~]# s3cmd ls
2022-03-18 06:05 s3://my-new-bucket
上传文件
[root@client1 ~]# s3cmd put /tmp/* s3://my-new-bucket
upload: '/tmp/ks-script-rL5xKG' -> 's3://my-new-bucket/ks-script-rL5xKG' [1 of 2]
836 of 836 100% in 0s 22.04 KB/s done
upload: '/tmp/yum.log' -> 's3://my-new-bucket/yum.log' [2 of 2]
0 of 0 0% in 0s 0.00 B/s done
查看文件
[root@client1 ~]# s3cmd ls s3://my-new-bucket
2022-03-18 06:59 836 s3://my-new-bucket/ks-script-rL5xKG
2022-03-18 06:58 548 s3://my-new-bucket/s3test.py
2022-03-18 06:58 5 s3://my-new-bucket/test.py
2022-03-18 06:59 0 s3://my-new-bucket/yum.log
[root@client1 ~]#
删除文件
[root@client1 ~]# s3cmd del s3://my-new-bucket/yum.log
delete: 's3://my-new-bucket/yum.log'
删除桶，前提是桶是空的
[root@client1 ~]# s3cmd rb s3://my-new-bucket
ERROR: S3 error: 409 (BucketNotEmpty)
再次删除所有文件
[root@client1 ~]# s3cmd del s3://my-new-bucket/*
delete: 's3://my-new-bucket/ks-script-rL5xKG'
delete: 's3://my-new-bucket/s3test.py'
delete: 's3://my-new-bucket/test.py'
删除桶
[root@client1 ~]# s3cmd rb s3://my-new-bucket
Bucket 's3://my-new-bucket/' removed
[root@client1 ~]#
[root@client1 ~]# s3cmd ls
[root@client1 ~]#
两个桶之间拷贝对象
[root@client1 ~]# s3cmd cp s3://bucket/* s3://bucket1/
[root@client1 ~]#
[root@client1 ~]# s3cmd du -H
0 1 objects s3://bucket/
0 0 objects s3://bucket1/
0 0 objects s3://bucket2/
0 Total
[root@client1 ~]# s3cmd cp s3://bucket/tmp/* s3://bucket1/
remote copy: 's3://bucket/tmp/yum.log' -> 's3://bucket1/yum.log' [1 of 1]
[root@client1 ~]#
[root@client1 ~]#
[root@client1 ~]# s3cmd du -H
0 1 objects s3://bucket/
0 1 objects s3://bucket1/
0 0 objects s3://bucket2/
0 Total
[root@client1 ~]#
查看对象信息
[root@client1 ~]# s3cmd info s3://bucket/tmp/yum.log
s3://bucket/tmp/yum.log (object):
File size: 0
Last mod: Fri, 18 Mar 2022 07:36:20 GMT
MIME type: text/plain
Storage: STANDARD
MD5 sum: d41d8cd98f00b204e9800998ecf8427e
SSE: none
Policy: none
CORS: none
ACL: cj: FULL_CONTROL
x-amz-meta-mtime: 1647575071.187600
获取对应的bucket所占用的空间大小
[root@client1 ~]# s3cmd du -H
0 1 objects s3://bucket/
0 0 objects s3://bucket1/
0 0 objects s3://bucket2/
0 Total
下载对象
[root@client1 ~]# s3cmd get s3://bucket/tmp/*
download: 's3://bucket/tmp/yum.log' -> './yum.log' [1 of 1]
0 of 0 0% in 0s 0.00 B/s done
[root@client1 ~]# ll
total 12
drwxr-xr-x. 2 root root 6 Mar 18 15:35 aaa
-rw-------. 1 root root 1360 Mar 18 11:49 anaconda-ks.cfg
-rw-r--r--. 1 root root 548 Mar 18 14:05 s3test.py
-rw-r--r--. 1 root root 5 Mar 18 14:54 test.py
-rw-r--r--. 1 root root 0 Mar 18 07:36 yum.log
[root@client1 ~]#
两个桶之间mv对象
[root@client1 ~]# s3cmd du -H
0 1 objects s3://bucket/
0 1 objects s3://bucket1/
0 0 objects s3://bucket2/
0 Total
[root@client1 ~]#
[root@client1 ~]#
[root@client1 ~]# s3cmd ls s3://bucket
DIR s3://bucket/tmp/
[root@client1 ~]# s3cmd mv s3://bucket/tmp/ s3://bucket2/
ERROR: Parameter problem: Expecting S3 URI with a filename or --recursive: s3://bucket/tmp/
[root@client1 ~]#
[root@client1 ~]# s3cmd mv s3://bucket/tmp/* s3://bucket2/
move: 's3://bucket/tmp/yum.log' -> 's3://bucket2/yum.log' [1 of 1]
[root@client1 ~]#
[root@client1 ~]# s3cmd ls s3://bucket2
2022-03-18 07:58 0 s3://bucket2/yum.log
[root@client1 ~]# s3cmd du -H
0 0 objects s3://bucket/
0 1 objects s3://bucket1/
0 1 objects s3://bucket2/
