说明

采用的mongo版版本： v4.4

install: https://www.mongodb.com/docs/v4.4/tutorial/install-mongodb-on-debian-tarball/

sharding

sharding是把数据分布到多个机器上的方式

mongo使用sharding支持大数据集和高吞吐量

sharded cluster

一个mongo shared cluster 包含如下组件:

shard: 每个shard包含一个数据集的子集，每个shard可以部署为一个replica set
mongos: mongos作为一个查询路由，提供客户端和sharded cluster之间的访问接口
config servers: 存储集群的元数据和配置数据

下图说明了cluster中各组件的关系

image.png

部署一个sharded cluster

部署config server

启动mongod

mkdir /home/config1 /home/config2 /home/config3
mongod --configsvr --replSet config --dbpath /home/config1 --logpath=/home/config1_log --bind_ip 127.0.0.1 --port 27021 &
mongod --configsvr --replSet config --dbpath /home/config2 --logpath=/home/config2_log --bind_ip 127.0.0.1 --port 27022 &
mongod --configsvr --replSet config --dbpath /home/config3 --logpath=/home/config3_log --bind_ip 127.0.0.1 --port 27023 &

连上任意一个节点, eg:

mongo --host 127.0.0.1 --port 27021

在mongo shell中执行初始化命令

rs.initiate(
  {
    _id: "config",
    configsvr: true,
    members: [
      { _id : 0, host : "127.0.0.1:27021" },
      { _id : 1, host : "127.0.0.1:27022" },
      { _id : 2, host : "127.0.0.1:27023" }
    ]
  }
)

创建分片副本集

启动mongod

mkdir /home/shard_a1 /home/shard_a2 /home/shard_a3
mongod --shardsvr --replSet shard_a  --dbpath /home/shard_a1 --logpath=/home/shard_a1_log --bind_ip 127.0.0.1 --port 28021 &
mongod --shardsvr --replSet shard_a  --dbpath /home/shard_a2 --logpath=/home/shard_a2_log --bind_ip 127.0.0.1 --port 28022 &
mongod --shardsvr --replSet shard_a  --dbpath /home/shard_a3 --logpath=/home/shard_a3_log --bind_ip 127.0.0.1 --port 28023 &

连上任意一个节点, eg:

mongo --host 127.0.0.1 --port 28021

在mongo shell中执行初始化命令

rs.initiate(
  {
    _id : "shard_a",
    members: [
      { _id : 0, host : "127.0.0.1:28021" },
      { _id : 1, host : "127.0.0.1:28022" },
      { _id : 2, host : "127.0.0.1:28023" }
    ]
  }
)

使用相同的命令创建另外一个分片副本集shard_b

127.0.0.1:29021
127.0.0.1:29022
127.0.0.1:29023

启动mongos

mongos --configdb config/127.0.0.1:27021,127.0.0.1:27022,127.0.0.1:27023 --logpath=/home/mongos1_log --bind_ip 127.0.0.1 --port 30001 &

Connect to the Sharded Cluster

用mongo shell连上mongos

mongo --host 127.0.0.1 --port 30001

Add Shards to the Cluster

sh.addShard("shard_a/127.0.0.1:28021,127.0.0.1:28022,127.0.0.1:28023")
sh.addShard("shard_b/127.0.0.1:29021,127.0.0.1:29022,127.0.0.1:29023")

Enable Sharding for a Database

Before you can shard a collection, you must enable sharding for the collection's database. Enabling sharding for a database does not redistribute data but make it possible to shard the collections in that database

sh.enableSharding("test")

Once you enable sharding for a database, MongoDB assigns a primary shard for that database where MongoDB stores all data in that database.

Shard a Collection

基于取值范围分片

sh.shardCollection("test.mycol", {_id: 1})

构造数据

use test
var from=0
var to=10000
for (var i=from;i<to;i++){ db.mycol.save({'_id':i}) }

var from=10000
var to=20000
for (var i=from;i<to;i++){ db.mycol.save({'_id':i}) }

查看数据分片

sh.status()

  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard_a    512
                                shard_b    512
                        too many chunks to print, use verbose if you want to force print
        {  "_id" : "test",  "primary" : "shard_b",  "partitioned" : true,  "version" : {  "uuid" : UUID("71ad614f-7533-4dee-b5d7-1627b4b4599e"),  "lastMod" : 1 } }
                test.mycol
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard_b    1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard_b Timestamp(1, 0)

可以看到test.mycol只在shard_b上有一个chunk，chunk分裂的默认大小是64M，需要构造多些数据才行

这里给通过给数据和shard打tag的方式，让数据分布到不同的shard

给shard打tag

sh.addShardTag("shard_a", "a")
sh.addShardTag("shard_b", "b")

给数据打分片tag

sh.addTagRange("test.mycol", { _id: 0 }, { _id: 10000}, "a")
sh.addTagRange("test.mycol", { _id: 10000 }, { _id: 20000}, "b")

使用sh.status可以看到有多个chunk分布在shard_a和shard_b上

        {  "_id" : "test",  "primary" : "shard_b",  "partitioned" : true,  "version" : {  "uuid" : UUID("71ad614f-7533-4dee-b5d7-1627b4b4599e"),  "lastMod" : 1 } }
                test.mycol
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard_a    1
                                shard_b    3
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : 0 } on : shard_b Timestamp(2, 1)
                        { "_id" : 0 } -->> { "_id" : 10000 } on : shard_a Timestamp(2, 0)
                        { "_id" : 10000 } -->> { "_id" : 20000 } on : shard_b Timestamp(2, 2)
                        { "_id" : 20000 } -->> { "_id" : { "$maxKey" : 1 } } on : shard_b Timestamp(2, 3)
                         tag: a  { "_id" : 0 } -->> { "_id" : 10000 }
                         tag: b  { "_id" : 10000 } -->> { "_id" : 20000 }

hashed sharding

mongo还可以基于sharding key做hash分片

sh.shardCollection("<database>.<collection>", { <shard key field> : "hashed" } )

mongo分片