526互联

ceph（六）对象存储网关（RadosGW）总结及s3cmd使用

发布时间 2023-09-26 16:09:54作者: areke

一、radosgw的基本使用总结

https://docs.ceph.com/en/latest/radosgw/

1.1 RadosGW对象存储网关简介

RadosGW是对象存储（OSS,Object Storage Service）的一种访问实现方式，RADOS网关也称为Ceph对象网关、RadosGW、RGW，是一种服务，使客户端能够利用标准对象存储API来访问Ceph集群，它支持AWS S3和Swift API，在 ceph 0.8版本之后使用Civetweb（https:/lgithub.com/civetweb/civetweb）的 web服务器来响应api请求，客户端使用http/https协议通过RESTful API与RGW通信，而RGW则通过librados与ceph集群通信，RGW客户端通过s3或者swift api使用RGW用户进行身份验证，然后RGW网关代表用户利用cephx与ceph存储进行身份验证。
说明：S3由Amazon于2006年推出，全称为Simple Storage Service,S3定义了对象存储，是对象存储事实上的标准，从某种意义上说，S3就是对象存储，对象存储就是S3，它是对象存储市场的霸主，后续的对象存储都是对S3的模仿。

1.2 RadosGW存储特点

  1. 通过对象存储网关将数据存储为对象，每个对象除了包含数据，还包含数据自身的元数据。
  2. 对象通过Object ID来检索，不是通过普通文件系统的挂载方式，而是通过文件路径加文件名称操作来直接访问对象，只能通过API来访问，或者第三方客户端(实际上也是对API的封装)来访问。
  3. 对象的存储不是垂直的目录树结构，而是存储在扁平的命名空间中，Amazon S3将这个扁平命名空间称为bucket，而swift则将其称为容器。无论是bucket还是容器，都不能再嵌套（bucket不能再包含 bucket）。
  4. bucket需要被授权才能访问到，一个帐户可以对多个bucket 授权，而权限可以不同。
  5. 方便横向扩展、快速检索数据。
  6. 不支持客户端挂载,且需要客户端在访问的时候指定文件名称。
  7. 不是很适用于文件过于频繁修改及删除的场景。

ceph 使用bucket作为存储桶(存储空间)，实现对象数据的存储和多用户隔离，数据存储在bucket 中，用户的权限也是针对bucket进行授权，可以设置用户对不同的bucket拥有不同的权限，以实现权限管理。

bucket特性：

  1. 存储空间(bucket)是用于存储对象(Object)的容器，所有的对象都必须隶属于某个存储空间，可以设置和修改存储空间属性用来控制地域、访问权限、生命周期等，这些属性设置直接作用于该存储空间内所有对象，因此可以通过灵活创建不同的存储空间来完成不同的管理功能
  2. 同一个存储空间的内部是扁平的，没有文件系统的目录等概念，所有的对象都直接隶属于其对应的存储空间
  3. 每个用户可以拥有多个存储空间
  4. 存储空间的名称在OSS范围内必须是全局唯一的，一旦创建之后无法修改名称
  5. 存储空间内部的对象数目没有限制

bucket命名规范：

https://docs.amazonaws.cn/AmazonS3/latest/userguide/bucketnamingrules.html

  1. 只能包括小写字母、数字和短横线(-)
  2. 必须以小写字母或者数字开头和结尾
  3. 长度必须在3-63字节之间
  4. 存储桶名称不能使用用IP地址格式
  5. Bucket名称必须全局唯一

1.3 部署RadosGW服务

radosgw架构图

‍

radosgw逻辑图

​​

1.3.1 安装radosgw服务

将ceph-mgr1、ceph-mgr2服务器部署为radosGW高可用服务

# 安装radosgw服务
[root@ceph-mgr1 ~]#apt install radosgw
[root@ceph-mgr2 ~]#apt install radosgw

# 在deploy服务器将rgw服务的ceph-mgr1、ceph-mgr2节点添加至集群
cephadmin@ceph-deploy:/data/ceph-cluster$ ceph-deploy rgw create ceph-mgr1
cephadmin@ceph-deploy:/data/ceph-cluster$ ceph-deploy rgw create ceph-mgr2
# 添加ceph-mgr2的服务过程
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.1.0): /usr/local/bin/ceph-deploy rgw create ceph-mgr2
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf object at 0x7f578fb45d60>
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0x7f578fc36f70>
[ceph_deploy.cli][INFO  ]  rgw                           : [('ceph-mgr2', 'rgw.ceph-mgr2')]
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph-mgr2:rgw.ceph-mgr2
[ceph-mgr2][DEBUG ] connection detected need for sudo
[ceph-mgr2][DEBUG ] connected to host: ceph-mgr2 
[ceph_deploy.rgw][INFO  ] Distro info: ubuntu 20.04 focal
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-mgr2
[ceph-mgr2][WARNIN] rgw keyring does not exist yet, creating one
[ceph-mgr2][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-mgr2 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-mgr2/keyring
[ceph-mgr2][INFO  ] Running command: sudo systemctl enable ceph-radosgw@rgw.ceph-mgr2
[ceph-mgr2][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-mgr2.service → /lib/systemd/system/ceph-radosgw@.service.
[ceph-mgr2][INFO  ] Running command: sudo systemctl start ceph-radosgw@rgw.ceph-mgr2
[ceph-mgr2][INFO  ] Running command: sudo systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-mgr2 and default port 7480

‍

1.3.2 验证radosgw服务状态

cephadmin@ceph-deploy:/data/ceph-cluster$ ceph -s
  cluster:
    id:     28820ae5-8747-4c53-827b-219361781ada
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 29h)
    mgr: ceph-mgr2(active, since 29h), standbys: ceph-mgr1
    mds: 2/2 daemons up, 2 standby
    osd: 20 osds: 20 up (since 28h), 20 in (since 4d)
    rgw: 2 daemons active (2 hosts, 1 zones)				# rgw服务：2个活跃节点
 
  data:
    volumes: 1/1 healthy
    pools:   10 pools, 321 pgs
    objects: 392 objects, 243 MiB
    usage:   6.5 GiB used, 20 TiB / 20 TiB avail
    pgs:     321 active+clean

1.3.3 验证radosgw服务进程

cephadmin@ceph-mgr1:~$ ps -ef|grep radosgw
ceph       49861       1  0 05:42 ?        00:00:00 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr1 --setuser ceph --setgroup ceph
cephadm+   50649   50608  0 05:44 pts/0    00:00:00 grep --color=auto radosgw

cephadmin@ceph-mgr2:~$ ps -ef|grep radosgw
ceph       50222       1  0 05:42 ?        00:00:04 /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-mgr2 --setuser ceph --setgroup ceph
cephadm+   51421   51408  0 06:00 pts/0    00:00:00 grep --color=auto radosgw

1.3.4 radosgw存储池类型

cephadmin@ceph-deploy:/data/ceph-cluster$ ceph osd pool ls
device_health_metrics
mypool
myrbd1
rbd-data1
cephfs-metadata
cephfs-data
.rgw.root
default.rgw.log
default.rgw.control
default.rgw.meta

# 查看默认radosgw存储池信息
cephadmin@ceph-deploy:/data/ceph-cluster$ radosgw-admin zone get --rgw-zone=default --rgw-zonegroup=default
{
    "id": "a202e4c0-376b-4848-956f-5b072662c3a3",	# 区域的唯一标识符
    "name": "default",								# 默认区域的名称
    "domain_root": "default.rgw.meta:root",			# 区域的根域名
    "control_pool": "default.rgw.control",			# 系统控制池，在有数据更新是，通知其他RGW更新缓存
    "gc_pool": "default.rgw.log:gc",				# 用于垃圾回收的存储池
    "lc_pool": "default.rgw.log:lc",				# 用于存储日志的存储池
    "log_pool": "default.rgw.log",					# 存储日志信息，用于记录各种log信息
    "intent_log_pool": "default.rgw.log:intent",
    "usage_log_pool": "default.rgw.log:usage",
    "roles_pool": "default.rgw.meta:roles",			# default.rgw.meta:元数据存储池，通过不同的名称空间分别存储不同的rados对象
    "reshard_pool": "default.rgw.log:reshard",
    "user_keys_pool": "default.rgw.meta:users.keys",	# 用户的密钥名称空间users.keys
    "user_email_pool": "default.rgw.meta:users.email",	# 用户的email名称空间users.email
    "user_swift_pool": "default.rgw.meta:users.swift",	# 用户的subuser的名称空间users.swift
    "user_uid_pool": "default.rgw.meta:users.uid",		# 用户UID
    "otp_pool": "default.rgw.otp",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",		# 存放bucket到object的索引信息
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "default.rgw.buckets.data"	# 存放对象的数据
                    }
                },
                "data_extra_pool": "default.rgw.buckets.non-ec",	# 数据的额外信息存储池
                "index_type": 0
            }
        }
    ],
    "realm_id": "",
    "notif_pool": "default.rgw.log:notif"
}


# 默认crush规则是副本池即一主两备的三副本机制
cephadmin@ceph-deploy:/data/ceph-cluster$ ceph osd pool get default.rgw.meta crush_rule
crush_rule: replicated_rule

# 默认副本数为3
cephadmin@ceph-deploy:/data/ceph-cluster$ ceph osd pool get default.rgw.meta size
size: 3

# 默认pgp数量为32
cephadmin@ceph-deploy:/data/ceph-cluster$ ceph osd pool get default.rgw.meta pgp_num
pgp_num: 32

# 默认pg数量为32
cephadmin@ceph-deploy:/data/ceph-cluster$ ceph osd pool get default.rgw.meta pg_num
pg_num: 32

‍

1.3.5 radosgw存储池功能

cephadmin@ceph-deploy:/data/ceph-cluster$ ceph osd lspools
1 device_health_metrics
2 mypool
3 myrbd1
4 rbd-data1
5 cephfs-metadata
6 cephfs-data
7 .rgw.root
8 default.rgw.log
9 default.rgw.control
10 default.rgw.meta

‍

1.3.6 验证radosgw zone信息

cephadmin@ceph-deploy:/data/ceph-cluster$ radosgw-admin zone get --rgw-zone=default
{
    "id": "a202e4c0-376b-4848-956f-5b072662c3a3",
    "name": "default",
    "domain_root": "default.rgw.meta:root",
    "control_pool": "default.rgw.control",
    "gc_pool": "default.rgw.log:gc",
    "lc_pool": "default.rgw.log:lc",
    "log_pool": "default.rgw.log",
    "intent_log_pool": "default.rgw.log:intent",
    "usage_log_pool": "default.rgw.log:usage",
    "roles_pool": "default.rgw.meta:roles",
    "reshard_pool": "default.rgw.log:reshard",
    "user_keys_pool": "default.rgw.meta:users.keys",
    "user_email_pool": "default.rgw.meta:users.email",
    "user_swift_pool": "default.rgw.meta:users.swift",
    "user_uid_pool": "default.rgw.meta:users.uid",
    "otp_pool": "default.rgw.otp",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "default.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0
            }
        }
    ],
    "realm_id": "",
    "notif_pool": "default.rgw.log:notif"
}

‍

1.3.7 访问radosgw服务

‍

1.4 radosgw高可用

1.4.1 radosgw http高可用

1.4.1.1 自定义http端口

配置文件可以在ceph deploy服务器修改然后统一推送，或者单独修改每个radosgw服务器的配置，然后重启RGW服务

https://docs.ceph.com/en/latest/radosgw/frontends/

cat /etc/ceph/ceph.conf
...
# 添加如下配置
[client.rgw.ceph-mgr1]
rgw_host = ceph-mgr1
rgw_frontends = civetweb port=9900

[client.rgw.ceph-mgr2]
rgw_host = ceph-mgr2
rgw_frontends = civetweb port=9900

# 统一推送配置文件至radosgw服务器
cephadmin@ceph-deploy:/data/ceph-cluster$ sudo scp ceph.conf 10.0.0.54:/etc/ceph/                                                                                                                                                                                                                          100%  763   743.7KB/s   00:00  
cephadmin@ceph-deploy:/data/ceph-cluster$ sudo scp ceph.conf 10.0.0.55:/etc/ceph/


# 重启服务
[root@ceph-mgr1 ~]#systemctl restart ceph-radosgw@rgw.ceph-mgr1.service 
[root@ceph-mgr2 ~]#systemctl restart ceph-radosgw@rgw.ceph-mgr2.service

‍

验证

‍

‍

1.4.1.2 高可用配置

使用haproxy进行反向代理

[root@lb1 ~]#cat /etc/haproxy/haproxy.cfg 

# 添加如下配置
listen ceph-rgw
	bind 10.0.0.63:80
	mode tcp
	server 10.0.0.54 10.0.0.54:9900 check inter 3s fall 3 rise 5
	server 10.0.0.55 10.0.0.55:9900 check inter 3s fall 3 rise 5

1.4.1.3 测试http反向代理

‍

1.4.2 radosgw https

在rgw节点生成签名证书并配置radosgw启用SSL

1.4.2.1 自签名证书

[root@ceph-mgr1 ~]#mkdir -p /etc/ceph/certs
[root@ceph-mgr1 ~]#cd /etc/ceph/certs/
[root@ceph-mgr1 certs]#openssl genrsa -out civetweb.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
............................................................................+++++
.......+++++
e is 65537 (0x010001)
[root@ceph-mgr1 certs]#openssl req -new -x509 -key civetweb.key -out civetweb.crt -subj "/CN=rgw.chu.net"
[root@ceph-mgr1 certs]#cat civetweb.key civetweb.crt > civetweb.pem
[root@ceph-mgr1 certs]#tree
.
├── civetweb.crt
├── civetweb.key
└── civetweb.pem

0 directories, 3 files

# 可直接将证书复制到ceph-mgr2节点上
[root@ceph-mgr1 ceph]#scp -r /etc/ceph/certs 10.0.0.55:/etc/ceph/

‍

1.4.2.2 SSL配置

[root@ceph-mgr1 certs]#vim /etc/ceph/ceph.conf 
....
# ceph-mgr1节点配置
[client.rgw.ceph-mgr1]
rgw_host = ceph-mgr1
rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/civetweb.pem"

# ceph-mgr2节点配置
[client.rgw.ceph-mgr2]
rgw_host = ceph-mgr2
rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/civetweb.pem"

# 重启服务
[root@ceph-mgr1 certs]#systemctl restart ceph-radosgw@rgw.ceph-mgr1.service
[root@ceph-mgr2 certs]#systemctl restart ceph-radosgw@rgw.ceph-mgr2.service

‍

1.4.2.3 验证https端口

[root@ceph-mgr1 certs]#netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name  
tcp        0      0 0.0.0.0:9900            0.0.0.0:*               LISTEN      80864/radosgw       	# 9900端口
tcp        0      0 127.0.0.1:38351         0.0.0.0:*               LISTEN      991/containerd  
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/init          
tcp        0      0 10.0.0.54:6800          0.0.0.0:*               LISTEN      46495/ceph-mds  
tcp        0      0 10.0.0.54:6801          0.0.0.0:*               LISTEN      46495/ceph-mds  
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      919/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1017/sshd: /usr/sbi 
tcp        0      0 0.0.0.0:37015           0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:35481           0.0.0.0:*               LISTEN      942/rpc.mountd  
tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      30872/sshd: root@pt 
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:9443            0.0.0.0:*               LISTEN      80864/radosgw			# 9443端口     
tcp        0      0 0.0.0.0:35875           0.0.0.0:*               LISTEN      942/rpc.mountd  
tcp        0      0 0.0.0.0:37639           0.0.0.0:*               LISTEN      942/rpc.mountd  

[root@ceph-mgr1 certs]#lsof -i:9443
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
radosgw 80864 ceph   80u  IPv4 343956      0t0  TCP *:9443 (LISTEN)

[root@ceph-mgr2 ~]#lsof -i:9443
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
radosgw 81763 ceph   80u  IPv4 371937      0t0  TCP *:9443 (LISTEN)

‍

1.4.2.4 验证访问

在本地host添加域名解析

echo "10.0.0.54 rgw.chu.net" >> /etc/hosts

# 或者 echo "10.0.0.55 rgw.chu.net" >> /etc/hosts
  • http访问

​

‍

  • https访问

​

证书信息

‍

1.4.3 radosgw高可用

通过负载均衡haproxy对radosgw进行反向代理，实现高可用

1.4.3.1 域名解析至负载均衡

将域名解析至负载均衡（VIP）

10.0.0.63 rgw.chu.net

1.4.3.2 负载均衡配置

如下单haproxy配置，也可结合keepalived实现负载均衡集群

# 配置监听
listen ceph-rgw-http
	bind 10.0.0.63:80
	mode tcp
	server 10.0.0.54 10.0.0.54:9900 check inter 3s fall 3 rise 5
	server 10.0.0.55 10.0.0.55:9900 check inter 3s fall 3 rise 5

listen ceph-rgw-https
	bind 10.0.0.63:443
	mode tcp
	server 10.0.0.54 10.0.0.54:9443 check inter 3s fall 3 rise 5
	server 10.0.0.55 10.0.0.55:9443 check inter 3s fall 3 rise 5

重启服务

[root@lb1 ~]#systemctl restart haproxy.service

1.4.3.3 测试访问

‍

‍

1.4.4 添加日志记录功能

增加日志及其他优化配置

  1. 新建日志目录并授权

    [root@ceph-mgr1 ceph]#mkdir -p /var/log/radosgw
[root@ceph-mgr1 ceph]#chown -R ceph.ceph /var/log/radosgw

  2. 修改配置文件

    [root@ceph-mgr1 ceph]#cat /etc/ceph/ceph.conf 
...
# 修改rgw本节点配置
[client.rgw.ceph-mgr1]
rgw_host = ceph-mgr1
rgw_frontends = "civetweb port=9900+9443s ssl_certificate=/etc/ceph/certs/civetweb.pem error_log_file=/var/log/radosgw/civetweb.error.log access_log_file=/var/log/radosgw/civetweb.access.log request_timeout_ms=30000	num_threads=200"

    num_threads默认值等于rgw_thread_pool_size=100

  3. 重启服务

    # systemctl daemon-reload
[root@ceph-mgr1 ceph]#systemctl restart ceph-radosgw@rgw.ceph-mgr1.service

  4. 服务测试

    本地curl命令和客户端浏览器访问

    [root@ceph-mgr1 ceph]#curl -k https://10.0.0.54:9443
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner>
<ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

  5. 验证日志

    [root@ceph-mgr1 ceph]#tail /var/log/radosgw/civetweb.access.log 
10.0.0.54 - - [26/Sep/2023:01:29:11 +0800] "GET / HTTP/1.1" 200 413 - curl/7.68.0
10.0.0.63 - - [26/Sep/2023:01:29:16 +0800] "GET / HTTP/1.1" 200 437 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36
10.0.0.63 - - [26/Sep/2023:01:29:16 +0800] "GET / HTTP/1.1" 200 437 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36
10.0.0.63 - - [26/Sep/2023:01:29:16 +0800] "GET / HTTP/1.1" 200 437 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36
10.0.0.63 - - [26/Sep/2023:01:29:16 +0800] "GET / HTTP/1.1" 200 437 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36

‍

‍

二、s3cmd客户端使用、实现基于nginx实现短视频的业务案例

https://docs.ceph.com/en/latest/radosgw/s3/

2.1 RGW Server配置

通常情况下，RGW1（10.0.0.54）和RGW2（10.0.0.55）参数配置是完全一样的

[root@ceph-mgr1 ceph]#cat /etc/ceph/ceph.conf 
[global]
fsid = 28820ae5-8747-4c53-827b-219361781ada
public_network = 10.0.0.0/24
cluster_network = 192.168.10.0/24
mon_initial_members = ceph-mon1,ceph-mon2,ceph-mon3
mon_host = 10.0.0.51,10.0.0.52,10.0.0.53
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

...
[client.rgw.ceph-mgr1]
rgw_host = ceph-mgr1
rgw_frontends = "civetweb port=9900"

[client.rgw.ceph-mgr2]
rgw_host = ceph-mgr2
rgw_frontends = "civetweb port=9900"

2.2 创建RGW用户

用于s3cmd客户端连接RGW

cephadmin@ceph-deploy:/data/ceph-cluster$ radosgw-admin user create --uid="user1" --display-name="user1"
{
    "user_id": "user1",
    "display_name": "user1",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "user1",
            "access_key": "26PBQL1JUTRAF3JAGZRA",
            "secret_key": "yJkLphACBrlWNu2rCaQbC8MO3uAFnr0NPYmPEUew"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

保存好user1用户的access_key、secret_key信息。

2.3 安装s3cmd客户端

s3cmd是一个通过命令行访问ceph RGW实现创建存储桶、上传、下载以及管理数据到对象存储的命令行工具。

cephadmin@ceph-deploy:/data/ceph-cluster$ sudo apt-cache madison s3cmd
cephadmin@ceph-deploy:/data/ceph-cluster$ sudo apt install -y s3cmd

2.4 配置s3cmd客户端执行环境

  1. s3cmd客户端添加域名解析
cephadmin@ceph-deploy:/data/ceph-cluster$ cat /etc/hosts
....
10.0.0.63 rgw.chu.net		# 负载均衡地址或RGW网关地址
  1. 配置命令执行环境
[root@ceph-deploy ~]#s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 26PBQL1JUTRAF3JAGZRA						# 输入之前创建用于客户端连接RGW网关的user1用户access key
Secret Key: yJkLphACBrlWNu2rCaQbC8MO3uAFnr0NPYmPEUew	# 输入用户secret key
Default Region [US]: 									# 地域，直接回车

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: rgw.chu.net				# RGW域名，若直连RGW设置为rgw.chu.net:9900

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: rgw.chu.net/%(bucket)		# bucket域名格式

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 123456								# 密码
Path to GPG program [/usr/bin/gpg]: 					# gpg命令路径，用于认证管理，直接回车

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: No							# 是否使用HTTPS

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: 								# 代理

New settings:	# 最终配置
  Access Key: 26PBQL1JUTRAF3JAGZRA
  Secret Key: yJkLphACBrlWNu2rCaQbC8MO3uAFnr0NPYmPEUew
  Default Region: US
  S3 Endpoint: rgw.chu.net
  DNS-style bucket+hostname:port template for accessing a bucket: rgw.chu.net/%(bucket)
  Encryption password: 123456
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y		# 是否测试
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] 						# 保存配置
Configuration saved to '/root/.s3cfg'		# 配置文件保存路径
  1. 验证认证文件
[root@ceph-deploy ~]#cat /root/.s3cfg 
[default]
access_key = 26PBQL1JUTRAF3JAGZRA
...
host_base = rgw.chu.net
host_bucket = rgw.chu.net/%(bucket)
...
secret_key = yJkLphACBrlWNu2rCaQbC8MO3uAFnr0NPYmPEUew
send_chunk = 65536
server_side_encryption = False
...

2.5 s3cmd常见使用命令

使用s3cmd --help​查看帮助信息，详细使用方法见：http://s3tools.org

  Make bucket		# 创建bucket
      s3cmd mb s3://BUCKET
  Remove bucket		# 删除bucket，只能删除空的bucket，如果bucket中有内容，需要先删除内容，才能删除bucket。
      s3cmd rb s3://BUCKET
  List objects or buckets	# 列出bucket中的文件
      s3cmd ls [s3://BUCKET[/PREFIX]]
  List all object in all buckets	# 列出所有bucket中的所有文件
      s3cmd la 
  Put file into bucket				# 上传文件到bucket中
      s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]
  Get file from bucket				# 从bucket中下载文件到本地
      s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
  Delete file from bucket			# 删除文件
      s3cmd del s3://BUCKET/OBJECT
  Delete file from bucket (alias for del)	# 删除文件，del命令的别名
      s3cmd rm s3://BUCKET/OBJECT
  Restore file from Glacier storage			# 恢复文件
      s3cmd restore s3://BUCKET/OBJECT
  Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options, see below)
      s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR		# 同步目录树
  Disk usage by buckets						# 空间使用
      s3cmd du [s3://BUCKET[/PREFIX]]
  Get various information about Buckets or Files	# 获取bucket或文件的详细信息
      s3cmd info s3://BUCKET[/OBJECT]
  Copy object								# 复制文件
      s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
  Modify object metadata					# 修改文件属性
      s3cmd modify s3://BUCKET1/OBJECT
  Move object								# 移动文件
      s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
  Modify Access control list for Bucket or Files
      s3cmd setacl s3://BUCKET[/OBJECT]
  Modify Bucket Policy
      s3cmd setpolicy FILE s3://BUCKET
  Delete Bucket Policy
      s3cmd delpolicy s3://BUCKET
  Modify Bucket CORS
      s3cmd setcors FILE s3://BUCKET
  Delete Bucket CORS
      s3cmd delcors s3://BUCKET
  Modify Bucket Requester Pays policy
      s3cmd payer s3://BUCKET
  Show multipart uploads
      s3cmd multipart s3://BUCKET [Id]
  Abort a multipart upload
      s3cmd abortmp s3://BUCKET/OBJECT Id
  List parts of a multipart upload
      s3cmd listmp s3://BUCKET/OBJECT Id
  Enable/disable bucket access logging
      s3cmd accesslog s3://BUCKET
  Sign arbitrary string using the secret key
      s3cmd sign STRING-TO-SIGN
  Sign an S3 URL to provide limited public access with expiry
      s3cmd signurl s3://BUCKET/OBJECT <expiry_epoch|+expiry_offset>
  Fix invalid file names in a bucket
      s3cmd fixbucket s3://BUCKET[/PREFIX]
  Create Website from bucket
      s3cmd ws-create s3://BUCKET
  Delete Website
      s3cmd ws-delete s3://BUCKET
  Info about Website
      s3cmd ws-info s3://BUCKET
  Set or delete expiration rule for the bucket
      s3cmd expire s3://BUCKET
  Upload a lifecycle policy for the bucket
      s3cmd setlifecycle FILE s3://BUCKET
  Get a lifecycle policy for the bucket
      s3cmd getlifecycle s3://BUCKET
  Remove a lifecycle policy for the bucket
      s3cmd dellifecycle s3://BUCKET
  List CloudFront distribution points
      s3cmd cflist 
  Display CloudFront distribution point parameters
      s3cmd cfinfo [cf://DIST_ID]
  Create CloudFront distribution point
      s3cmd cfcreate s3://BUCKET
  Delete CloudFront distribution point
      s3cmd cfdelete cf://DIST_ID
  Change CloudFront distribution point parameters
      s3cmd cfmodify cf://DIST_ID
  Display CloudFront invalidation request(s) status
      s3cmd cfinvalinfo cf://DIST_ID[/INVAL_ID]

2.6 s3cmd测试数据上传、下载

2.6.1 创建bucket

存储空间Bucket是用于存储对象Object的容器，在上传任意类型的Object前，需要先创建Bucket

bucket/object授权参考：https://docs.amazonaws.cn/AmazonS3/latest/userguide/access-policy-language-overview.html

[root@ceph-deploy ~]#s3cmd mb s3://mybucket
Bucket 's3://mybucket/' created

[root@ceph-deploy ~]#s3cmd mb s3://css
Bucket 's3://css/' created

[root@ceph-deploy ~]#s3cmd mb s3://images
Bucket 's3://images/' created

2.6.2 上传文件

# 上传文件至mybucket
[root@ceph-deploy ~]#s3cmd put /etc/passwd s3://mybucket
upload: '/etc/passwd' -> 's3://mybucket/passwd'  [1 of 1]
 2104 of 2104   100% in    1s  1182.09 B/s  done

# /不表示文件目录层级关系，只表示地址信息
[root@ceph-deploy ~]#s3cmd put /var/log/syslog s3://images/log/
upload: '/var/log/syslog' -> 's3://images/log/syslog'  [1 of 1]
 614406 of 614406   100% in    0s    14.23 MB/s  done


# 验证bucket中的文件
[root@ceph-deploy ~]#s3cmd ls s3://mybucket
2023-09-25 18:23      2104   s3://mybucket/passwd

# 可先查看逻辑上bucket根目录s3://images
[root@ceph-deploy ~]#s3cmd ls s3://images
                       DIR   s3://images/log/
[root@ceph-deploy ~]#s3cmd ls s3://images/log/
2023-09-25 18:26    614406   s3://images/log/syslog

2.6.3 下载文件

# 下载文件
[root@ceph-deploy ~]#s3cmd get s3://images/log/syslog /opt/
download: 's3://images/log/syslog' -> '/opt/syslog'  [1 of 1]
 614406 of 614406   100% in    0s    38.28 MB/s  done

# 验证文件
[root@ceph-deploy ~]#ls /opt/syslog 
/opt/syslog

2.6.4 删除文件

# 查看bucket中文件
[root@ceph-deploy ~]#s3cmd ls s3://mybucket/
2023-09-25 18:23      2104   s3://mybucket/passwd

# 删除
[root@ceph-deploy ~]#s3cmd rm s3://mybucket/passwd
delete: 's3://mybucket/passwd'

# 查看bucket文件删除
[root@ceph-deploy ~]#s3cmd ls s3://mybucket/

‍

2.7 结合nginx实现短视频业务

​

​

2.7.1 上传视频文件至bucket

  1. 创建bucket

    [root@ceph-deploy opt]#s3cmd mb s3://video
Bucket 's3://video/' created

  2. 上传文件

    ‍

    [root@ceph-deploy opt]#s3cmd put 1656753768362045.mp4 s3://video
upload: '1656753768362045.mp4' -> 's3://video/1656753768362045.mp4'  [1 of 1]
 15061866 of 15061866   100% in    0s    40.93 MB/s  done

  3. 授权bucket匿名用户只读权限，用于浏览器访问

    https://docs.amazonaws.cn/AmazonS3/latest/userguide/example-bucket-policies.html

    # 编辑授权文件，对video bucket用户授予读权限
cat video-bucket-single-policy.json
{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow" ,
		"Principal": "*",
		"Action": "s3:GetObject",
		"Resource": [
			"arn:aws:s3:::video/*"
		]
	}]
}

# 执行授权
s3cmd setpolicy video-bucket-single-policy.json s3://video

    ‍

  4. 配置haproxy

    [root@lb1 ~]#cat /etc/haproxy/haproxy.cfg
....
listen ceph-rgw-http
        bind 10.0.0.63:9900
        mode tcp
        server 10.0.0.54 10.0.0.54:9900 check inter 3s fall 3 rise 5
        server 10.0.0.55 10.0.0.55:9900 check inter 3s fall 3 rise 5

    重启服务

    [root@lb1 ~]#systemctl restart haproxy.service

  5. 浏览器直接访问测试

    访问负载均衡测试

    ​

    ​​

​​​

2.7.2 配置nginx反向代理

nginx服务器为10.0.0.60，实现访问www.chu.net的视频内容将请求转向访问ceph文件存储中的video视频

[root@nginx ~]#cat /etc/nginx/nginx.conf 
...
http {
	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 65;
	types_hash_max_size 2048;

	# 配置server块内容
	server {
            listen 80;
            server_name chu.net www.chu.net;

            location / {
                    root html;
                    index index.html index.htm;
            }

            location ~* \.(mp4|avi)$ {			# 以mp4或avi结尾的请求转向10.0.0.63（VIP）
                    proxy_pass http://10.0.0.63:9900;
            }
    }
}

重启nginx

nginx -s reload

2.7.3 访问测试

配置本地hosts域名解析

10.0.0.60 chu.net www.chu.net

浏览器访问视频