（一）双机方案
1. 工作流程图
2. 服务器环境
简述：目前双机环境要求有四台服务器，并预分配两个虚拟ip地址
| 服务器 | ip | 配置 |
|---|---|---|
| NMS VIP | 10.17.0.1 | |
| NMS Node01 | 10.17.0.2 | Cpu:4core Ram:8Gb Storage:256Gb |
| NMS Node02 | 10.17.0.3 | Cpu:4core Ram:8Gb Storage:256Gb |
| Mariadb VIP | 10.17.0.5 | |
| Mariadb Node01 | 10.17.0.6 | Cpu:4core Ram:8Gb Storage:512Gb |
| Mariadb Node02 | 10.17.0.7 | Cpu:4core Ram:8Gb Storage:512Gb |
注：以上ip分配要求网络互通
3. 高可用策略
3.1 NMS Node
3.1.1 简述
在NMS Node01和NMS Node02上分别部署NMS应用，两个节点之间通过keepalived进行管理并向外暴露虚拟ip（10.17.0.1）进行程序访问。
3.1.2 keepalived策略
目前NMS Node提供以下两种故障转移策略，具体可参考客户意见
主主模式：主主模式下不可抢占，假设一开始keepalived工作在10.17.0.2服务器上，当10.17.0.2服务器上的NMS中某个组件挂掉，会触发故障转移即10.17.0.2上的keepalived服务会被停止，资源程序会转移到10.17.0.3服务器，当10.17.0.2服务器恢复正常时，资源程序仍然由10.17.0.3管理，不会发生抢占被转移。
主备模式：主备模式下会发生抢占，假设一开始keepalived工作在10.17.0.2服务器上，当10.17.0.2服务器上的NMS中某个组件挂掉，会触发故障转移即10.17.0.2上的keepalived服务会被停止，资源程序会转移到10.17.0.3服务器，当10.17.0.2服务器恢复正常时，资源程序不再由10.17.0.3管理，而是被10.17.0.2所抢占。
3.2 Mariadb Node
3.2.1 简述
Mariadb Node01和MariadbNode02上分别部署Mariadb数据库服务，两台服务器间采用Mariadb自带主从机制进行双向主从配置实现数据库同步目的，通过keepalived管理虚拟ip（10.17.0.5）并向NMS Node节点提供数据库服务。
3.2.2 keepalived策略
主主模式：主主模式下不可抢占，假设一开始keepalived工作在10.17.0.6服务器上，当10.17.0.6服务器上的Mariadb服务出现异常或服务器宕机，会触发故障转移即10.17.0.6上的keepalived服务会被停止，资源程序会转移到10.17.0.7服务器，当10.17.0.6服务器恢复正常时，资源程序仍然由10.17.0.7管理，不会发生抢占被转移。
4. 文件同步
4.1 简述
NMS程序会定时对数据库数据做备份操作，集群环境下，当一个分布式定时任务触发时，执行这个定时任务的实例是不确定的。假设Node01节点执行了这个备份任务后，文件只会被保存在当前Node01节点上，当发生故障转移，工作节点被Node02接管后，该文件在Node02上是不存在的。为了保证服务间的文件一致性，我们需要对文件进行同步操作。
4.2 解决方案
通过手写python脚本，指定对应目标服务器及文件夹完成文件夹定期同步。
（二）双机部署
前言
从上述双机方案描述，双机环境建立在keepalived组建基础上，因此需要在每台服务器上都安装keepalived服务，具体安装视操作系统而定，以下步骤默认keepalived服务已经安装。
Ubuntu安装keepalived：sudo apt-get install keepalived
SuseLinux安装keepalived：sudo zypper in keepalived
关于keepalived检验机制：keepalived工作节点以默认间隔1s的频率，给局域网中的其他节点以多播的形式发送VRRP通告，告知自己的优先级，其他节点只负责处理工作节点发出的多播包，当发现工作节点优先级没自己高，或者没收到工作节点的VRRP通告时，表示工作节点已经故障，自己将切换到工作节点状态，完成故障转移。所以我们需要做的就是控制工作节点什么时候去停止发送VRRP通告或者降低自身优先级以触发故障转移
1. Mariadb Node
1.1 keepalived配置
路径：/etc/keepalived/
路径下新建文件keepalived.conf，内容如下
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n68" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">! Configuration File for keepalived
bal_defs {
router_id db01
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 45
priority 110
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.17.0.5/24
}
}
virtual_server 10.17.0.5 3306 {
delay_loop 6
persistence_timeout 50
protocol TCP
real_server 10.17.0.6 3306 { #根据当前实际宿主机IP进行配置
notify_down /etc/keepalived/kill_keepalived.sh
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
connect_port 3306
delay_before_retry 3
}
}
}</pre>
1.2 shell脚本
路径：/etc/keepalived/
keepalived服务停止脚本：kill_keepalived.sh
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n72" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">#!/bin/bash
kill -9 $(cat /var/run/keepalived.pid)</pre>
1.3 keepalived启动
kill_keepalived.sh权限赋予：
chmod 777 /etc/keepalived/checkNms.sh
keepalived启动：
systemctl restart keepalived
注：以上步骤需要在两台Mariadb Node上都做配置，对于keepalived.conf配置文件，属性real_server 10.17.0.6 3306需要根据当前实际宿主机IP进行配置，其他配置、脚本文件保存一致。
2. Mariadb主从
2.1 容器创建：
两台Mariadb节点部署Mariadb容器
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n85" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">docker run --name madb01 --restart=always -p 3306:3306 -v $HOME/docker/volumes/madb01:/var/lib/mysql -v /etc/localtime:/etc/localtime -e MYSQL_ROOT_PASSWORD=root_pwd -e MYSQL_USER=nms9000 -v MYSQL_PASSWORD=nms9000_pwd -d mariadb:10.5.8</pre>
2.2 文件配置
两台Mariadb节点都需要配置
1.将容器内的配置文件拷贝出来
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="shell" cid="n89" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">docker cp madb01:/etc/mysql/mariadb.conf.d/50-server.cnf ./</pre>
2.修改拷贝出来的文件：vim 50-server.cnf ，加入以下内容
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n91" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">server-id = 96 #两台服务设置成不同值
log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
binlog-do-db=nms9000
replicate-do-db=nms9000
relay_log=mysql-relay-bin
gtid_mode=ON
enforce-gtid-consistency=true
slave_skip_errors = 1062
auto-increment-offset = 2</pre>
3.将50-server.cnf文件拷贝回容器内
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="shell" cid="n93" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">docker cp 50-server.cnf madb01:/etc/mysql/mariadb.conf.d/</pre>
4.重启容器
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="shell" cid="n95" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">systemctl restart madb01</pre>
2.3 数据初始化
两台Mariadb节点上的madb01容器都需要初始化
将madb01容器进行数据库数据初始化操作，initDB.sql为数据库文件
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n99" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">docker exec -i madb01 /usr/bin/mysql -uroot -p123456 < initDB.sql</pre>
2.4 主从连接
注：以下所有命令操作都是在madb01容器内的mysql服务执行
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n102" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">#1.进入容器
docker exec -it madb01 /bin/bash
2.进入mysql服务
mysql -uroot -proot_pwd</pre>
2.4.1 生成slave连接用户
Mariadb Node01（10.17.0.6）
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n105" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">#生成连接用户并授权
create user 'slave02'@'10.17.0.7' identified by 's123456';
grant replication slave on . to 'slave02'@'10.17.0.7';
flush privileges;
停止数据更新操作
flush tables with read lock;</pre>
Mariadb Node02（10.17.0.7）
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n107" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">#生成连接用户并授权
create user 'slave01'@'10.17.0.6' identified by 's123456';
grant replication slave on . to 'slave01'@'10.17.0.6';
flush privileges;
停止数据更新操作
flush tables with read lock;</pre>
2.4.2 主从连接建立
Node02连接Node01
1.Node01上查看master状态
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n111" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">#1.查看master状态
show master status\G;
2.返回结果展示
MariaDB [(none)]> show master status\G;
*************************** 1. row ***************************
File: mysql-bin.000001
Position: 328
Binlog_Do_DB: nms9000
Binlog_Ignore_DB:
1 row in set (0.000 sec)
ERROR: No query specified</pre>
2.Node02建立连接
注：MASTER_LOG_FILE和MASTER_LOG_POS对应赋值来自上面master状态信息
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n114" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">#1.主从连接建立
CHANGE MASTER TO
MASTER_HOST='10.17.0.6',
MASTER_USER='slave02',
MASTER_PASSWORD='s123456',
MASTER_LOG_FILE='mysql-bin.000001',
MASTER_LOG_POS=328;
2.开启slave服务
start slave；</pre>
Node01连接Node02
1.Node02上查看master状态
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n118" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">#1.查看master状态
show master status\G;
2.返回结果展示
MariaDB [(none)]> show master status\G;
*************************** 1. row ***************************
File: mysql-bin.000001
Position: 328
Binlog_Do_DB: nms9000
Binlog_Ignore_DB:
1 row in set (0.000 sec)
ERROR: No query specified</pre>
2.Node01建立连接
注：MASTER_LOG_FILE和MASTER_LOG_POS对应赋值来自上面master状态信息
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n121" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">#1.主从连接建立
CHANGE MASTER TO
MASTER_HOST='10.17.0.7',
MASTER_USER='slave01',
MASTER_PASSWORD='s123456',
MASTER_LOG_FILE='mysql-bin.000001',
MASTER_LOG_POS=328;
2.开启slave服务
start slave；</pre>
2.4.3 连接验证
释放上述锁资源
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n124" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">unlock tables;</pre>
两台节点都需要进行验证，如下执行命令show slave status\G;返回结果中Slave_IO_Running和Slave_SQL_Running都为Yes即代表已经互联成功。
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n126" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">MariaDB [(none)]> show slave status\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.27.96
Master_User: slave01
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000007
Read_Master_Log_Pos: 342
Relay_Log_File: mysql-relay-bin.000005
Relay_Log_Pos: 641
Relay_Master_Log_File: mysql-bin.000007
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB: nms9000</pre>
3. NMS Node
触发故障转移点：通过shell脚本定期（每60s）去校验NMS相关组件服务，如果有一个组件挂了，就会主动将keepalived进程关闭，无法再发送VRRP通告以触发故障转移。
3.1 keepalived配置
路径：/etc/keepalived/
路径下新建文件keepalived.conf，内容如下
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="c" cid="n132" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">! Configuration File for keepalived
global_defs {
router_id nms01
添加以下参数即可
script_user root
enable_script_security
}
vrrp_script check_run {
script "/etc/keepalived/checkNms.sh"
interval 60
weight 10
user root
pass nms1234
}
vrrp_instance VI_1 {
state BACKUP
interface ens33 #根据自己的工作网卡决定
virtual_router_id 41
priority 110
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
check_run
}
virtual_ipaddress {
10.17.0.1/24 #虚拟IP地址
}
}</pre>
3.2 shell脚本
路径：/etc/keepalived/
服务校验脚本checkNms.sh
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="shell" cid="n136" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">#!/bin/bash
curPath=(dirname "$0")")
echo $curPath
filePath=filePath
cat $filePath | while read line
do
pIDa=lsof -i :$line | grep -v "PID" | awk '{print $1}'
cID=docker ps | grep $line |awk -F' ' '{printf $1}'
if [ "line ] : false"
kill -9 $(cat /var/run/keepalived.pid)
exit 1
fi
done
exit 0</pre>
应用组件列表文件：nmssvc.txt
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="shell" cid="n138" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">server
gateway
system
monitor
otnservice
adaptor
redis-server</pre>
3.3 keepalived启动
checkNms.sh权限赋予：
chmod 744 /etc/keepalived/checkNms.sh
注：不能赋予777全权限，keepalived会认为这是不安全的行为
keepalived启动：
systemctl restart keepalived
注：因为keepalived服务依赖于NMS程序，所以需要确保NMS程序已经成功启动，由于数据库单独维护，在上述Mariadb连接配置好之后，需要修改
docker-compose.yml中数据库连接地址为Mariadb VIP（10.17.0.5）
注：以上步骤需要在两台NMS Node上都做配置，对于keepalived.conf配置文件，属性priority 110需要根据ip地址来决定，其他脚本文件、服务列表文件保存一致。
3.4 配置文件同步
3.4.1 syncdir.py脚本
syncdir.py通过配置本机源目录和目标服务器文件目录，定期执行将本机源目录同步至目标服务器文件目录。
因此，两台服务器如果要保持文件夹文件一致，需要做双向同步。
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="python" cid="n154" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">#!/usr/bin/python3
from datetime import datetime
from apscheduler.schedulers.blocking import BlockingScheduler
import time
import paramiko
import sys,os
from scp import SCPClient
server = ""
port = 22
user=""
password=""
sourceDir = "/home/nms/sunam/docker-volumes/malaysia/backup/database"
absSourceFiles = []
destDir = "/home/nms/sunam/docker-volumes/malaysia/backup/database"
def createSSHClient(server, port, user, password):
client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(server, port, user, password)
return client
def getLocalFiles():
sourceFiles=[]
for root,dirs,files in os.walk(sourceDir):
for file in files:
sourceFiles.append(file)
return sourceFiles
def getUpdateFiles():
pass
def syncDir():
ssh = createSSHClient(server, port, user, password)
absSourceFiles = []
scp = SCPClient(ssh.get_transport())
localFiles = getLocalFiles()
ftp = ssh.open_sftp()
try:
destFiles = ftp.listdir(destDir)
for file in destFiles:
if file in localFiles:
localFiles.remove(file)
if len(localFiles)>0:
for file in localFiles:
absSourceFiles.append(os.path.join(sourceDir,file))
for file in absSourceFiles:
scp.put(file,destDir)
except:
print("except...")
scp.close()
ssh.close()
def dojob():
ssh = createSSHClient(server, port, user, password)
创建调度器：BlockingScheduler
scheduler = BlockingScheduler()
添加任务,时间间隔10S
scheduler.add_job(syncDir, 'interval', seconds=30, id='test_job1',args=[ssh])
scheduler.add_job(syncDir, 'interval', seconds=60, id='test_job1')
scheduler.start()
if name=='main':
len_argv = len(sys.argv)
if len_argv == 1:
server = "192.168.27.51"
user = "nms"
password="nms1234"
elif len_argv == 4:
server = sys.argv[1]
user = sys.argv[2]
password=sys.argv[3]
else:
exit
dojob()
syncDir()</pre>
运行syncdir.py脚本
sudo python3 syncdir.py
注：执行syncdir.py 脚本需要python相关依赖环境，首次执行可根据报错提示进行依赖加载。
3.4.2 自启配置
将syncdir.py 脚本的执行作为linux服务加入到服务中，并配置为自启服务。
/etc/systemd/system/路径下新建syncDir.service，配置以下内容
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="" cid="n162" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">[Unit]Description=<short description>[Service]Type=simpleExecStart=/home/nms/sunam/deploy/malaysia/output/syncdir/syncdir.py #脚本文件所在目录[Install]WantedBy=multi-user.target </pre>
设置自启
<pre class="md-fences md-end-block ty-contain-cm" spellcheck="false" lang="" cid="n164" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">#1.设置自启systemctl enable syncDir.service#2.以服务形式启动同步文件脚本systemctl start syncDir.service</pre>
