说明
采用的mongo版版本： v4.4
install: https://www.mongodb.com/docs/v4.4/tutorial/install-mongodb-on-debian-tarball/
sharding
sharding是把数据分布到多个机器上的方式
mongo使用sharding支持大数据集和高吞吐量
sharded cluster
一个mongo shared cluster 包含如下组件:
shard: 每个shard包含一个数据集的子集，每个shard可以部署为一个replica set
mongos: mongos作为一个查询路由，提供客户端和sharded cluster之间的访问接口
config servers: 存储集群的元数据和配置数据
下图说明了cluster中各组件的关系
部署一个sharded cluster
部署config server
启动mongod
mkdir /home/config1 /home/config2 /home/config3
mongod --configsvr --replSet config --dbpath /home/config1 --logpath=/home/config1_log --bind_ip 127.0.0.1 --port 27021 &
mongod --configsvr --replSet config --dbpath /home/config2 --logpath=/home/config2_log --bind_ip 127.0.0.1 --port 27022 &
mongod --configsvr --replSet config --dbpath /home/config3 --logpath=/home/config3_log --bind_ip 127.0.0.1 --port 27023 &
连上任意一个节点, eg:
mongo --host 127.0.0.1 --port 27021
在mongo shell中执行初始化命令
rs.initiate(
{
_id: "config",
configsvr: true,
members: [
{ _id : 0, host : "127.0.0.1:27021" },
{ _id : 1, host : "127.0.0.1:27022" },
{ _id : 2, host : "127.0.0.1:27023" }
]
}
)
创建分片副本集
启动mongod
mkdir /home/shard_a1 /home/shard_a2 /home/shard_a3
mongod --shardsvr --replSet shard_a --dbpath /home/shard_a1 --logpath=/home/shard_a1_log --bind_ip 127.0.0.1 --port 28021 &
mongod --shardsvr --replSet shard_a --dbpath /home/shard_a2 --logpath=/home/shard_a2_log --bind_ip 127.0.0.1 --port 28022 &
mongod --shardsvr --replSet shard_a --dbpath /home/shard_a3 --logpath=/home/shard_a3_log --bind_ip 127.0.0.1 --port 28023 &
连上任意一个节点, eg:
mongo --host 127.0.0.1 --port 28021
在mongo shell中执行初始化命令
rs.initiate(
{
_id : "shard_a",
members: [
{ _id : 0, host : "127.0.0.1:28021" },
{ _id : 1, host : "127.0.0.1:28022" },
{ _id : 2, host : "127.0.0.1:28023" }
]
}
)
使用相同的命令创建另外一个分片副本集shard_b
127.0.0.1:29021
127.0.0.1:29022
127.0.0.1:29023
启动mongos
mongos --configdb config/127.0.0.1:27021,127.0.0.1:27022,127.0.0.1:27023 --logpath=/home/mongos1_log --bind_ip 127.0.0.1 --port 30001 &
Connect to the Sharded Cluster
用mongo shell连上mongos
mongo --host 127.0.0.1 --port 30001
Add Shards to the Cluster
sh.addShard("shard_a/127.0.0.1:28021,127.0.0.1:28022,127.0.0.1:28023")
sh.addShard("shard_b/127.0.0.1:29021,127.0.0.1:29022,127.0.0.1:29023")
Enable Sharding for a Database
Before you can shard a collection, you must enable sharding for the collection's database. Enabling sharding for a database does not redistribute data but make it possible to shard the collections in that database
sh.enableSharding("test")
Once you enable sharding for a database, MongoDB assigns a primary shard for that database where MongoDB stores all data in that database.
Shard a Collection
基于取值范围分片
sh.shardCollection("test.mycol", {_id: 1})
构造数据
use test
var from=0
var to=10000
for (var i=from;i<to;i++){ db.mycol.save({'_id':i}) }
var from=10000
var to=20000
for (var i=from;i<to;i++){ db.mycol.save({'_id':i}) }
查看数据分片
sh.status()
databases:
{ "_id" : "config", "primary" : "config", "partitioned" : true }
config.system.sessions
shard key: { "_id" : 1 }
unique: false
balancing: true
chunks:
shard_a 512
shard_b 512
too many chunks to print, use verbose if you want to force print
{ "_id" : "test", "primary" : "shard_b", "partitioned" : true, "version" : { "uuid" : UUID("71ad614f-7533-4dee-b5d7-1627b4b4599e"), "lastMod" : 1 } }
test.mycol
shard key: { "_id" : 1 }
unique: false
balancing: true
chunks:
shard_b 1
{ "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard_b Timestamp(1, 0)
可以看到test.mycol只在shard_b上有一个chunk，chunk分裂的默认大小是64M，需要构造多些数据才行
这里给通过给数据和shard打tag的方式，让数据分布到不同的shard
给shard打tag
sh.addShardTag("shard_a", "a")
sh.addShardTag("shard_b", "b")
给数据打分片tag
sh.addTagRange("test.mycol", { _id: 0 }, { _id: 10000}, "a")
sh.addTagRange("test.mycol", { _id: 10000 }, { _id: 20000}, "b")
使用sh.status可以看到有多个chunk分布在shard_a和shard_b上
{ "_id" : "test", "primary" : "shard_b", "partitioned" : true, "version" : { "uuid" : UUID("71ad614f-7533-4dee-b5d7-1627b4b4599e"), "lastMod" : 1 } }
test.mycol
shard key: { "_id" : 1 }
unique: false
balancing: true
chunks:
shard_a 1
shard_b 3
{ "_id" : { "$minKey" : 1 } } -->> { "_id" : 0 } on : shard_b Timestamp(2, 1)
{ "_id" : 0 } -->> { "_id" : 10000 } on : shard_a Timestamp(2, 0)
{ "_id" : 10000 } -->> { "_id" : 20000 } on : shard_b Timestamp(2, 2)
{ "_id" : 20000 } -->> { "_id" : { "$maxKey" : 1 } } on : shard_b Timestamp(2, 3)
tag: a { "_id" : 0 } -->> { "_id" : 10000 }
tag: b { "_id" : 10000 } -->> { "_id" : 20000 }
hashed sharding
mongo还可以基于sharding key做hash分片
sh.shardCollection("<database>.<collection>", { <shard key field> : "hashed" } )
