codis架构
Codis组件
Codis 3.x 由以下组件组成：
-
Codis FE：集群管理界面
- 多个集群实例共享可以共享同一个前端展示页面；
- 通过配置文件管理后端 codis-dashboard 列表，配置文件可自动更新。
-
Codis Dashboard：集群管理工具，支持 codis-proxy、codis-server 的添加、删除，以及据迁移等操作。在集群状态发生改变时，codis-dashboard 维护集群下所有 codis-proxy 的状态的一致性。
- 对于同一个业务集群而言，同一个时刻 codis-dashboard 只能有 0个或者1个；
- 所有对集群的修改都必须通过 codis-dashboard 完成。
redis-sentinel:redis类似的高可用机制支持自动主从切换（HA）
Storage: 外部存储（注册中心）例如FileSystem、ETCD、Zookeeper
-
Codis Proxy：客户端连接的 Redis 代理服务, 实现了 Redis 协议。 除部分命令不支持以外，表现的和原生的 Redis 没有区别（就像 Twemproxy）。
- 对于同一个业务集群而言，可以同时部署多个 codis-proxy 实例；
- 不同 codis-proxy 之间由 codis-dashboard 保证状态同步。
Codis-Group: 没有对应的组件,而是一个虚拟的节点机制通过槽位映射机制实现proxy与server的映射
- Codis Server(Codis Redis)：基于 redis-3.2.8 分支开发。增加了额外的数据结构，以支持 slot 有关的操作以及数据迁移指令。
-
Codis Admin：集群管理的命令行工具。
- 可用于控制 codis-proxy、codis-dashboard 状态以及访问外部存储。
Codis集群启动
Codis集群启动需要zookeeper或者etcd,我们以zookeeper为例，搭建Codis集群。
关于zookeeper的配置，可以参考zookeeper安装部署
我们使用3台主机搭建集群环境
配置如下：
| hostName | ip |
|---|---|
| vc0 | 192.168.4.50 |
| vc1 | 192.168.4.51 |
| vc2 | 192.168.4.52 |
vc0-1分别配置环境变量，在~/.bashrc中增加
export CODIS_HOME=/storage/dep/go/src/github.com/CodisLabs/codis
export PATH=$PATH:$CODIS_HOME/bin
启动 Codis Dashboard
使用vc0作为主机,修改vc0的Codis Dashboard配置文件vim /storage/dep/go/src/github.com/CodisLabs/codis/config/dashboard.toml
##################################################
# #
# Codis-Dashboard #
# #
##################################################
# Set Coordinator, only accept "zookeeper" & "etcd" & "filesystem".
# for zookeeper/etcd, coorinator_auth accept "user:password"
# Quick Start
coordinator_name = "zookeeper"
coordinator_addr = "192.168.4.50:2181,192.168.4.51:2181,192.168.4.52:2181"
#coordinator_auth = ""
# Set Codis Product Name/Auth.
product_name = "codis-nasa"
product_auth = ""
# Set bind address for admin(rpc), tcp only.
admin_addr = "0.0.0.0:18080"
# Set arguments for data migration (only accept 'sync' & 'semi-async').
migration_method = "semi-async"
migration_parallel_slots = 100
migration_async_maxbulks = 200
migration_async_maxbytes = "32mb"
migration_async_numkeys = 500
migration_timeout = "30s"
# Set configs for redis sentinel.
sentinel_client_timeout = "10s"
sentinel_quorum = 2
sentinel_parallel_syncs = 1
sentinel_down_after = "30s"
sentinel_failover_timeout = "5m"
sentinel_notification_script = ""
sentinel_client_reconfig_script = ""
重点修改的参数有
-
coordinator_name
- 外部存储类型，接受 zookeeper/etcd,本例使用zookeeper
-
coordinator_addr
- 外部存储地址，使用zookeeper集群的客户端暴露访问地址，以逗号分开
-
product_name
- 区分不同的集群，注册到zookeeper也是用这个区分 ,根据自己需求起名
-
admin_addr
- 用于dashboard与codas-proxy,codis-server进行tcp通信
使用命令nohup codis-dashboard --ncpu=1 --config=/storage/dep/go/src/github.com/CodisLabs/codis/config/dashboard.toml --log=/storage/dep/go/src/github.com/CodisLabs/codis/log/dashboard.log --log-level=WARN & 启动
其中：
--ncpu=N 最大使用 CPU 个数
-c CONF, --config=CONF 指定启动配置文件
-l FILE, --log=FILE 设置 log 输出文件
--log-level=LEVEL 设置 log 输出等级：INFO,WARN,DEBUG,ERROR；默认INFO，推荐WARN
执行后查看日志 tail -100f /storage/dep/go/src/github.com/CodisLabs/codis/log/dashboard.log.2018-06-25,得到输出
2018/06/25 17:12:36 main.go:78: [WARN] set ncpu = 1
2018/06/25 17:12:36 topom.go:119: [WARN] create new topom:
{
"token": "cf7a714f4756eec383928d11c2899196",
"start_time": "2018-06-25 17:12:36.761816028 +0800 CST",
"admin_addr": "vc0:18080",
"product_name": "codis-nsa",
"pid": 5364,
"pwd": "/storage/dep/go/src/github.com/CodisLabs/codis/log",
"sys": "Linux vc0 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux"
}
2018/06/25 17:12:36 main.go:142: [WARN] create topom with config
coordinator_name = "zookeeper"
coordinator_addr = "192.168.4.50:2181,192.168.4.51:2181,192.168.4.52:2181"
coordinator_auth = ""
admin_addr = "0.0.0.0:18080"
product_name = "codis-nsa"
product_auth = ""
......
......
2018/06/25 17:12:37 topom_sentinel.go:169: [WARN] rewatch sentinels = []
2018/06/25 17:12:37 main.go:179: [WARN] [0xc4202bed80] dashboard is working ...
2018/06/25 17:12:37 topom.go:429: [WARN] admin start service on [::]:18080
观察日志输出发现Codis Dashboard已经按照配置正常启动
启动 Codis PROXY
配置文件如下：
product_name = "codis-nasa" # 集群名称，同codis-dashboard中配置
product_auth = "" #集群密码
session_auth = ""
# Set bind address for admin(rpc), tcp only.
admin_addr = "0.0.0.0:11080" # 用于和codis-dashboard进行tcp通信同步信息
# Set bind address for proxy, proto_type can be "tcp", "tcp4", "tcp6", "unix" or "unixpacket".
proto_type = "tcp4" #端口协议类型
proxy_addr = "0.0.0.0:19000" #用于redis客户端连接,支持redis协议，端口唯一
jodis_name = ""
jodis_addr = ""
jodis_auth = ""
jodis_timeout = "20s"
jodis_compatible = false
# Set datacenter of proxy.
proxy_datacenter = ""
# Set max number of alive sessions.
proxy_max_clients = 1000
# Set max offheap memory size. (0 to disable)
proxy_max_offheap_size = "1024mb"
# Set heap placeholder to reduce GC frequency.
proxy_heap_placeholder = "256mb"
# Proxy will ping backend redis (and clear 'MASTERDOWN' state) in a predefined interval. (0 to disable)
backend_ping_period = "5s"
# Set backend recv buffer size & timeout.
backend_recv_bufsize = "128kb"
backend_recv_timeout = "30s"
# Set backend send buffer & timeout.
backend_send_bufsize = "128kb"
backend_send_timeout = "30s"
# Set backend pipeline buffer size.
backend_max_pipeline = 20480
# Set backend never read replica groups, default is false
backend_primary_only = false
# Set backend parallel connections per server
backend_primary_parallel = 1
backend_replica_parallel = 1
# Set backend tcp keepalive period. (0 to disable)
backend_keepalive_period = "75s"
# Set number of databases of backend.
backend_number_databases = 16
# If there is no request from client for a long time, the connection will be closed. (0 to disable)
# Set session recv buffer size & timeout.
session_recv_bufsize = "128kb"
session_recv_timeout = "30m"
# Set session send buffer size & timeout.
session_send_bufsize = "64kb"
session_send_timeout = "30s"
# Make sure this is higher than the max number of requests for each pipeline request, or your client may be blocked.
# Set session pipeline buffer size.
session_max_pipeline = 10000
# Set session tcp keepalive period. (0 to disable)
session_keepalive_period = "75s"
# Set session to be sensitive to failures. Default is false, instead of closing socket, proxy will send an error response to client.
session_break_on_failure = false
# Set metrics server (such as http://localhost:28000), proxy will report json formatted metrics to specified server in a predefined period.
metrics_report_server = ""
metrics_report_period = "1s"
# Set influxdb server (such as http://localhost:8086), proxy will report metrics to influxdb.
metrics_report_influxdb_server = ""
metrics_report_influxdb_period = "1s"
metrics_report_influxdb_username = ""
metrics_report_influxdb_password = ""
metrics_report_influxdb_database = ""
# Set statsd server (such as localhost:8125), proxy will report metrics to statsd.
metrics_report_statsd_server = ""
metrics_report_statsd_period = "1s"
metrics_report_statsd_prefix = ""
使用命令nohup codis-proxy --ncpu=1 --config=/storage/dep/go/src/github.com/CodisLabs/codis/config/proxy.toml --log=/storage/dep/go/src/github.com/CodisLabs/codis/log/proxy.log --log-level=WARN & 启动
查看日志tail -f /storage/dep/go/src/github.com/CodisLabs/codis/log/proxy.log.2018-06-25
2018/06/25 18:18:40 main.go:229: [WARN] [0xc4200eb8c0] proxy waiting online ...
2018/06/25 18:18:41 main.go:229: [WARN] [0xc4200eb8c0] proxy waiting online ...
2018/06/25 18:18:43 main.go:229: [WARN] [0xc4200eb8c0] proxy waiting online ...
2018/06/25 18:18:44 main.go:229: [WARN] [0xc4200eb8c0] proxy waiting online ...
可以观察到codis-proxy已经启动，但是游离的，没有加入codis-dashboard管理，也没有与任何codis-server建立联系,我们可以稍后通过codis-fe将proxy纳入管理
启动 Codis Server(Redis实例)
我们可以将vc0,vc1,vc2三个主机上分别启动一个redis,vc0使用端口6380,vc1使用端口6381,vc2使用端口6382,
对vc0-2分别添加配置文件
vc0中 /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6380.conf
vc1中 /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6381.conf
vc2中 /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6382.conf
vc0中主要修改port 6380和pidfile /tmp/redis_6380.pid以及logfile "/tmp/redis_6380.log"以及bind 127.0.0.1 192.168.4.50
vc1中主要修改port 6381和pidfile /tmp/redis_6381.pid以及logfile "/tmp/redis_6381.log"以及bind 127.0.0.1 192.168.4.51
vc0中主要修改port 6382和pidfile /tmp/redis_6382.pid以及logfile "/tmp/redis_6382.log"以及bind 127.0.0.1 192.168.4.52
在vc0-2中分别执行以下命令启动redis实例
codis-server /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6380.conf
codis-server /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6381.conf
codis-server /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6382.conf
启动 Codis FE(管理平台)
vc0中启动Codis FE nohup ./bin/codis-fe --ncpu=1 --log=/storage/dep/go/src/github.com/CodisLabs/codis/log/fe.log --log-level=WARN --zookeeper=127.0.0.1:2181 --listen=192.168.4.50:8080 &
注意上面的命令必须要在CODIS_HOME下执行
其中:
zookeeper=127.0.0.1:2181为zk的连接地址
listen=192.168.4.50:8080为管理界面的访问地址和端口
通过fe管理集群
通过fe增加Proxy
打开FE的控制页面在Proxy栏中增加之前配置好的Proxy，输入Proxy的连接地址192.168.4.50:11080
如图：
[图片上传失败...(image-74f13d-1530000375805)]
通过fe增加Group(redis组)
