09-MongoDB集群-服务器操作实践

0、初始化环境

操作系统：windows server 2012 R2、CentOS Linux release 7.6.1810 (Core)

1、测试账号密码

windows，通过远程桌面登录测试账号、密码是否正确

linux，通过xshell远程登录测试账号、密码是否正确

2、获取服务器信息

windows，我的电脑，右键查看

linux，通过命令查看

# 系统
[mongodb@localhost ~]$ cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core) 

# cpu
mongodb@mongodb01:~$ cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c
     32  Intel(R) Core(TM)2 Duo CPU     T7700  @ 2.40GHz
     
# 内存
[mongodb@localhost ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:            31G        941M         29G         12M        625M         29G
Swap:          7.9G          0B        7.9G

# 磁盘
mongodb@mongodb01:~$ df -h
文件系统        容量  已用  可用 已用% 挂载点
udev             24G     0   24G    0% /dev
tmpfs           4.8G  1.4M  4.8G    1% /run
/dev/vda1       787G  7.6G  739G    2% /
tmpfs            24G     0   24G    0% /dev/shm
tmpfs           5.0M     0  5.0M    0% /run/lock
tmpfs            24G     0   24G    0% /sys/fs/cgroup
...
tmpfs           4.8G   16K  4.8G    1% /run/user/123
/dev/loop16      90M   90M     0  100% /snap/core/8268
tmpfs           4.8G     0  4.8G    0% /run/user/1000

3、修改用户密码

windows，编写批处理文件，拷贝至服务器，右键，管理员权限执行

net user Administrator 密码

linux，通过命令修改密码、机器名（部署pyspark建议同时修改机器名）

# 使用 root 登录
[C:\~]$ ssh root@172.16.12.6

# 修改 root 口令
[root@localhost ~]#echo "BigData120" | passwd --stdin root

# 修改 mongodb 口令
[root@localhost ~]#echo "BigData120" | passwd --stdin mongodb

# 编辑sudoers 文件，将用户添加至sudo中，实现免密
[root@localhost ~]#vim /etc/sudoers

mongodb            ALL=(ALL)                NOPASSWD: ALL

# 修改机器名
[root@localhost ~]hostnamectl set-hostname mongodb01

# 登出
[root@localhost ~]# exit

# 使用mongodb登录验证
[mongodb@mongodb01 ~]$ sudo vim /etc/sudoers
[mongodb@mongodb01 ~]$

Linux 磁盘挂载，默认设置磁盘 Ext4 格式，建议修改为 XFS

# 通过界面工具设置磁盘分区
应用程序->工具->磁盘->选中剩余空间-> 点击 + -> 下一步 -> 设置名称 Data -> 点击启动挂载
注：重启后需要重新设置，可通过修改启动文件设置启动自动挂载

# 设置启动挂载
[mongodb@mongodb01 ~]$ sudo vim /etc/fstab

/dev/vda3    /run/media/mongodb/Data    ext4    defaults    0 0

# 重启
[mongodb@mongodb01 ~]$ sudo shutdown -r now

# 查看验证
[mongodb@mongodb01 ~]$ df -h
文件系统                 容量  已用  可用 已用% 挂载点
/dev/mapper/centos-root   50G  4.1G   46G    9% /
...
/dev/vda3                591G   73M  561G    1% /run/media/mongodb/Data
...
tmpfs                    3.2G     0  3.2G    0% /run/user/1000

4、形成记录表

序号	服务器名称	ip地址	用户名	密码	硬盘	CPU	内存	操作系统

5、Linux 安装 MongoDB

[mongodb@mongodb04 ~]$ sudo chmod -R 777 /run/media/mongodb/Data

# 切换至当前用户操作目录

# 新建目录
[mongodb@mongodb01 ~]$ mkdir -p /run/media/mongodb/Data/software/

[mongodb@mongodb01 ~]$ cd /run/media/mongodb/Data/software

# 安装完成后，执行 rz 命令
[mongodb@mongodb01 software]$ rz

# 选择上传 windows 客户端下载的 mongodb-linux-x86_64-rhel70-4.2.1.tgz 文件

# 解压安装包
[mongodb@mongodb01 software]$ tar -zxvf mongodb-linux-x86_64-rhel70-4.2.1.tgz

# 建立数据库文件、日志文件目录
[mongodb@mongodb01 software]$ mkdir -p /run/media/mongodb/Data/db/
[mongodb@mongodb01 software]$ mkdir -p /run/media/mongodb/Data/logs/

# 修改 mongodb 配置文件
[mongodb@mongodb01 software]$ vim /run/media/mongodb/Data/mongo.conf

fork=true
dbpath=/run/media/mongodb/Data/db
port=27017
bind_ip=0.0.0.0
logpath=/run/media/mongodb/Data/logs/mongodb.log
logappend=true
replSet=database_repl

# 可能解压文件后第一次服务自启，先关闭
[mongodb@mongodb05 logs]$ sudo netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:27017           0.0.0.0:*               LISTEN      8102/mongod         

[mongodb@mongodb05 logs]$ sudo kill -9 8102

# 启动 mongodb 服务
[mongodb@mongodb01 software]$ sudo /run/media/mongodb/Data/software/mongodb-linux-x86_64-rhel70-4.2.1/bin/mongod --config /run/media/mongodb/Data/mongo.conf

# 开启防火墙
[mongodb@mongodb05 logs]$ sudo firewall-cmd --zone=public --add-port=27017/tcp --permanent
Warning: ALREADY_ENABLED: 27017:tcp
success

[mongodb@mongodb05 logs]$ sudo firewall-cmd --reload
success
# 其他服务器重复以上步骤

6、配置副本集（一台机器设置即可）

# 进入mongo客户端
[mongodb@mongodb05 software]$ /run/media/mongodb/Data/software/mongodb-linux-x86_64-rhel70-4.2.1/bin/mongo

# 配置复制集
var rsconf = {
    _id:'database_repl', //_id要与配置文件中指定的服务所属的复制集相同
    members:  //复制集成员
    [
        {
            _id:1, //成员的id
            host:'172.16.12.6:27017' //成员所属节点的ip以及该成员服务启动时所占的端口
        },
        {
            _id:2,
            host:'172.16.12.7:27017'
        },
        {
            _id:3,
            host:'172.16.12.8:27017'
        },
        {
            _id:4,
            host:'172.16.12.9:27017'
        },
        {
            _id:5,
            host:'172.16.12.10:27017'
        }
    ]
}

# 初始化配置(加载rsconf配置文件)
rs.initiate(rsconf);

# 成功提示
{
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1571471070, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1571471070, 1)
}

# 状态查看
rs.status();

# 主节点上设置管理员
database_repl:PRIMARY> use admin
switched to db admin
database_repl:PRIMARY> db.createUser({
...     user: "sa",
...     pwd: "sa_coin",
...     roles: [{ role: "root", db: "admin" }]
... });
Successfully added user: {
    "user" : "sa",
    "roles" : [
        {
            "role" : "root",
            "db" : "admin"
        }
    ]
}
database_repl:PRIMARY> 

# 主节点上设置普通用户
database_repl:PRIMARY> use test
switched to db test
database_repl:PRIMARY> db.createUser({user:'dev',pwd:'123456',roles:[{role:'dbOwner',db:'test'}]})  
Successfully added user: {
    "user" : "dev",
    "roles" : [
        {
            "role" : "dbOwner",
            "db" : "test"
        }
    ]
}
database_repl:PRIMARY> 

# 创建副本集认证key文件并拷贝至其他节点 单机器生成，拷贝至其他机器
[mongodb@mongodb05 ~]$ cd /run/media/mongodb/Data/software
[mongodb@mongodb05 software]$ sudo openssl rand -base64 90 -out ./mongodb-keyfile
[mongodb@mongodb05 software]$ sudo chmod 300 mongodb-keyfile
[mongodb@mongodb05 software]$ sudo scp /run/media/mongodb/Data/software/mongodb-keyfile mongodb@172.16.12.9:/run/media/mongodb/Data/software/mongodb-keyfile

/run/media/mongodb/Data/software/mongodb-linux-x86_64-rhel70-4.2.1/bin/mongo -u sa -p sa_coin

8、各服务器更新配置信息

# 所有机器更新mongo.conf
[mongodb@mongodb01 software]$ vim /run/media/mongodb/Data/mongo.conf

fork=true
dbpath=/run/media/mongodb/Data/db
port=27017
bind_ip=0.0.0.0
logpath=/run/media/mongodb/Data/logs/mongodb.log
logappend=true
replSet=database_repl
auth=true 
keyFile=/run/media/mongodb/Data/software/mongodb-keyfile 

# 关闭服务
[mongodb@mongodb05 software]$ sudo netstat -ntlp | grep 'mongod'
[mongodb@mongodb05 software]$ sudo kill -9 9117

# 启动 mongodb 服务
[mongodb@mongodb01 software]$ sudo /run/media/mongodb/Data/software/mongodb-linux-x86_64-rhel70-4.2.1/bin/mongod --config /run/media/mongodb/Data/mongo.conf

7、各服务器其他设置

# 自动同步服务器时间
[mongodb@mongodb01 bin]$  crontab -e     // 每隔一小时同步更新一次时间
0 */1 * * * /usr/sbin/ntpdate ntp.api.bz >/dev/null 2>&1

# 设置开启Mongodb服务自启，方案一
[mongodb@mongodb01 ~]$ sudo vim /lib/systemd/system/mongodb.service

[Unit]
Description=mongodb
After=network.target remote-fs.target nss-lookup.target
[Service]
Type=forking
ExecStart=/run/media/mongodb/Data/software/mongodb-linux-x86_64-rhel70-4.2.1/bin/mongod --config /run/media/mongodb/Data/mongo.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/run/media/mongodb/Data/software/mongodb-linux-x86_64-rhel70-4.2.1/bin/mongod --shutdown  --config /run/media/mongodb/Data/mongo.conf
PrivateTmp=true
[Install] 
WantedBy=multi-user.target

[mongodb@mongodb04 software]$ sudo chmod 754 /lib/systemd/system/mongodb.service

# 设置启动
#[mongodb@mongodb04 software]$ sudo systemctl start mongodb.service
# 设置关闭
#[mongodb@mongodb04 software]$ sudo systemctl stop mongodb.service
# 设置开机自启
[mongodb@mongodb04 software]$ sudo systemctl enable mongodb.service
# 查看状态
#[mongodb@mongodb04 software]$ sudo systemctl status mongodb.service

# 重启验证
[mongodb@mongodb01 ~]$ sudo reboot

# 查看目前systemctl中的服务
#[mongodb@mongodb01 ~]$ sudo systemctl list-unit-files | grep 'mongo'

# 设置删除
#[mongodb@mongodb04 software]$ sudo systemctl disable mongodb.service
# 移除启动文件
#[mongodb@mongodb04 software]$ sudo rm /lib/systemd/system/mongodb.service

# 设置开启Mongodb服务自启，方案二
[mongodb@mongodb04 ~]$ sudo vim /etc/init.d/mongodb  

#!/bin/sh  
#  
#chkconfig: 2345 80 90  
#description: mongodb  
start() {  
 sudo /run/media/mongodb/Data/software/mongodb-linux-x86_64-rhel70-4.2.1/bin/mongod --config /run/media/mongodb/Data/mongo.conf 
}  
  
stop() {  
  sudo /run/media/mongodb/Data/software/mongodb-linux-x86_64-rhel70-4.2.1/bin/mongod --shutdown  --config /run/media/mongodb/Data/mongo.conf
}  
  
case "$1" in  
  start)  
 start  
 ;;  
  stop)  
 stop  
 ;;  
  restart)  
 stop  
 start  
 ;;  
  *)  
 echo $"Usage: $0 {start|stop|restart}"  
 exit 1  
esac  

[mongodb@mongodb04 ~]$ cd /etc/init.d 
[mongodb@mongodb04 init.d]$ sudo chkconfig --add mongodb
[mongodb@mongodb04 init.d]$ sudo chmod +x  mongodb 
[mongodb@mongodb04 init.d]$ sudo chkconfig mongodb on 
# 重启验证
[mongodb@mongodb01 ~]$ sudo reboot
# 移除服务
#[mongodb@mongodb04 ~]$ sudo chkconfig --del mongodb
[mongodb@mongodb04 ~]$ sudo rm /etc/init.d/mongodb

8、异常汇总

缺失 rz

# 上传安装包，提示需安装 lrzsz
mongodb@mongodb01:~/software$ rz

Command 'rz' not found, but can be installed with:

sudo apt install lrzsz

mongodb@mongodb01:~/software$ sudo apt install lrzsz

服务无法启动异常

# 尝试启动 mongodb 服务，提示
mongodb@mongodb01:~/software/mongodb-linux-x86_64-rhel70-4.2.1/bin$ sudo ./mongod --config /opt/mongo/mongo.conf
./mongod: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory

# 通过 ldd 分析原因
mongodb@mongodb01:~/software/mongodb-linux-x86_64-rhel70-4.2.1/bin$ ldd /lib/x86_64-linux-gnu$ cd /home/mongodb/software/mongodb-linux-x86_64-rhel70-4.2.1/bin/mongod
/lib/x86_64-linux-gnu$:
ldd: /lib/x86_64-linux-gnu$: 没有那个文件或目录
cd:
ldd: ./cd: 没有那个文件或目录
/home/mongodb/software/mongodb-linux-x86_64-rhel70-4.2.1/bin/mongod:
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f827b075000)
    libcrypto.so.10 => not found
    libssl.so.10 => not found
    ......
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f827b06f000)

# 修复 libcrypto.so.10 问题，安装对应包
sudo apt-get update
sudo apt-get install libssl1.0.0 libssl-dev

# 查看安装包
mongodb@mongodb01:/lib/x86_64-linux-gnu$ ll | grep -E 'libssl|libcrypto'
-rw-r--r--   1 root root  2925408 4月   3  2019 libcrypto.so.1.1
-rw-r--r--   1 root root   356928 12月  6 00:46 libssl3.so
-rw-r--r--   1 root root   585504 4月   3  2019 libssl.so.1.1

# 设置软链接
sudo ln -s libssl.so.1.1 libssl.so.10
sudo ln -s libcrypto.so.1.1 libcrypto.so.10

# 查看创建是否正确
mongodb@mongodb01:/lib/x86_64-linux-gnu$ ll | grep -E 'libssl|libcrypto'
lrwxrwxrwx   1 root root       16 12月 13 13:45 libcrypto.so.10 -> libcrypto.so.1.1
-rw-r--r--   1 root root  2925408 4月   3  2019 libcrypto.so.1.1
-rw-r--r--   1 root root   356928 12月  6 00:46 libssl3.so
lrwxrwxrwx   1 root root       13 12月 13 13:40 libssl.so.10 -> libssl.so.1.1
-rw-r--r--   1 root root   585504 4月   3  2019 libssl.so.1.1

# 如果创建链接错误，想重新创建，需先删除已存在软链接
# mongodb@mongodb01:/lib/x86_64-linux-gnu$ sudo rm libssl.so.10

服务启动失败 error number 48

ERROR: child process failed, exited with error number 48

STORAGE  [initandlisten] Failed to set up listener: SocketException: Address already in use

sudo netstat -ntlp | grep 'mongod'

[mongodb@mongodb04 ~]$ sudo kill -9 8041

服务启动失败 error number 100

ERROR: child process failed, exited with error number 100

2019-12-17T16:51:54.154+0800 I  STORAGE  [initandlisten] exception in initAndListen: DBPathInUse: Unable to lock the lock file: /run/media/mongodb/Data/db/mongod.lock (Resource temporarily unavailable). Another mongod instance is already running on the /run/media/mongodb/Data/db directory, terminating


sudo rm /run/media/mongodb/Data/db/mongod.lock

服务启动失败 error number 14

ERROR: child process failed, exited with error number 14


2019-12-17T16:56:14.377+0800 I  STORAGE  [initandlisten] Detected data files in /run/media/mongodb/Data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-12-17T16:56:14.377+0800 I  STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=15491M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
2019-12-17T16:56:14.866+0800 E  STORAGE  [initandlisten] WiredTiger error (11) [1576572974:866218][11150:0x7ffb98de5c00], wiredtiger_open: __posix_file_lock, 391: /run/media/mongodb/Data/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable Raw: [1576572974:866218][11150:0x7ffb98de5c00], wiredtiger_open: __posix_file_lock, 391: /run/media/mongodb/Data/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable
2019-12-17T16:56:14.866+0800 E  STORAGE  [initandlisten] WiredTiger error (16) [1576572974:866314][11150:0x7ffb98de5c00], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy Raw: [1576572974:866314][11150:0x7ffb98de5c00], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy
2019-12-17T16:56:14.866+0800 E  STORAGE  [initandlisten] WiredTiger error (11) [1576572974:866445][11150:0x7ffb98de5c00], wiredtiger_open: __posix_file_lock, 391: /run/media/mongodb/Data/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable Raw: [1576572974:866445][11150:0x7ffb98de5c00], wiredtiger_open: __posix_file_lock, 391: /run/media/mongodb/Data/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable
2019-12-17T16:56:14.866+0800 E  STORAGE  [initandlisten] WiredTiger error (16) [1576572974:866462][11150:0x7ffb98de5c00], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy Raw: [1576572974:866462][11150:0x7ffb98de5c00], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy
2019-12-17T16:56:14.866+0800 E  STORAGE  [initandlisten] WiredTiger error (11) [1576572974:866537][11150:0x7ffb98de5c00], wiredtiger_open: __posix_file_lock, 391: /run/media/mongodb/Data/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable Raw: [1576572974:866537][11150:0x7ffb98de5c00], wiredtiger_open: __posix_file_lock, 391: /run/media/mongodb/Data/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable
2019-12-17T16:56:14.866+0800 E  STORAGE  [initandlisten] WiredTiger error (16) [1576572974:866557][11150:0x7ffb98de5c00], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy Raw: [1576572974:866557][11150:0x7ffb98de5c00], wiredtiger_open: __conn_single, 1682: WiredTiger database is already being managed by another process: Device or resource busy
2019-12-17T16:56:14.866+0800 W  STORAGE  [initandlisten] Failed to start up WiredTiger under any compatibility version.
2019-12-17T16:56:14.866+0800 F  STORAGE  [initandlisten] Reason: 16: Device or resource busy
2019-12-17T16:56:14.866+0800 F  -        [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 786
2019-12-17T16:56:14.866+0800 F  -        [initandlisten] 


sudo rm /run/media/mongodb/Data/db/mongod.lock
sudo rm /run/media/mongodb/Data/db/WiredTiger.lock

重启机器后，自启服务异常

exception in initAndListen std::exception: listen: Address already in use, terminating

检查各种自启方式，确认是否重复启动
方式一，rc.local 中添加启动命令：
/etc/rc.local
方式二，将 sh脚本放置 profile.d 目录下：
/etc/profile.d/
方式三，查看 init.d 目录下：
[mongodb@mongodb04 ~]$ cd /etc/init.d/
[mongodb@mongodb04 init.d]$ ll
总用量 44
-rw-r--r--. 1 root root 18281 8月  24 2018 functions
-rwxr-xr-x. 1 root root   797 12月 13 16:34 mongod
-rwxr-xr-x. 1 root root  4569 8月  24 2018 netconsole
-rwxr-xr-x. 1 root root  7923 8月  24 2018 network
-rw-r--r--. 1 root root  1160 10月 31 2018 README
[mongodb@mongodb04 init.d]$ sudo rm mongod
方式四，查看 chkconfig
sudo chkconfig --list

没有权限无法写入

Failed global initialization

[mongodb@mongodb04 logs]$ sudo chmod 777 -R /run/media/mongodb

重启服务自动开机启动，几分钟后停止

2019-12-19T09:46:56.718+0800 I  CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2019-12-19T09:46:56.720+0800 I  NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2019-12-19T09:46:56.720+0800 I  NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock

设置启动不对，修改为如下
[Unit]
Description=mongodb
After=network.target remote-fs.target nss-lookup.target
[Service]
Type=forking
ExecStart=/run/media/mongodb/Data/software/mongodb-linux-x86_64-rhel70-4.2.1/bin/mongod --config /run/media/mongodb/Data/mongo.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/run/media/mongodb/Data/software/mongodb-linux-x86_64-rhel70-4.2.1/bin/mongod --shutdown  --config /run/media/mongodb/Data/mongo.conf
PrivateTmp=true
[Install] 
WantedBy=multi-user.target