HA Cluster的实现方案
vrrp协议的实现：keepalived
VRRP协议的目的就是为了解决静态路由单点故障问题；VRRP通过竞选(election)协议来动态的将路由任务交给LAN中虚拟路由器中的某台VRRP路由器。
vrrp协议
（Virtual Redundant Routing Protocol）：虚拟冗余路由协议
| vrrp中的术语 | |
|---|---|
| VR | 虚拟路由器 |
| VRID | 虚拟路由器的标识号（0-255） |
| Master（Active） | 主路由器（活动节点） |
| Backup（Passive） | 备份路由器（被动节点） |
| VIP | 虚拟IP地址；流动（浮动）IP |
| VMAC | VIP对应的虚拟MAC地址（48位）；00-00-5E-00-01-{VRID} |
| 优先级 | 初始化过程中就决定了优先级 |
| gracious arp | 免费arp广播 |
vrrp工作方式
- 抢占式：如果 Backup 路由器工作在抢占方式下，当它收到 VRRP 报文后会将自己的优先级与通告报文中的优先级进行比较。如果自己的优先级比当前的 Master 路由器的优先级高就会主动抢占成为 Master 路由器否则将保持 Backup 状态。
- 非抢占式：如果 Backup 路由器工作在非抢占方式下则只要 Master 路由器没有出现故障Backup 路由器即使随后被配置了更高的优先级也不会成为Master 路由器。
vrrp认证方式与工作模式
认证方式
- 无认证
- 简单字符串认证：预共享密钥
- md5认证
工作模式
- master-backup模式
- master-master模式
vrrp工作原理
一个VRRP路由器有唯一的标识：VRID，范围为0-255，该路由器对外表现为唯一的虚拟MAC地址，地址的格式为00-00-5E- 00-01-[VRID]，主控路由器负责对ARP请求用该MAC地址应答，保证给终端设备的是唯一一致的IP和MAC地址，减少了切换对终端设备的影响。
VRRP控制报文只有一种：
VRRP通告(advertisement)使用IP多播数据包进行封装，组地址为224.0.0.18，发布范围只限于同一局域网内。这保证了VRID在不同网络中可以重复使用。在VRRP路由器组中按优先级选举主控路由器
为了保证VRRP协议的安全性，提供了两种安全认证措施：明文认证和IP头认证
vrrp优势
- 负载共享：允许来自LAN客户端的流量由多个路由器设备所共享；
- 多VRRP组：在一个路由器物理接口上可配置多达255个VRRP组；
- 抢占：在master故障时允许优先级更高的backup成为master；
- 通告协议：使用IANA所指定的组播地址224.0.0.18进行VRRP通告；
- VRRP追踪：基于接口状态来改变其VRRP优先级来确定最佳的VRRP路由器成为master；
- 冗余：可以使用多个路由器设备作为LAN客户端的默认网关，大大降低了默认网关成为单点故障的可能性；
- 多IP地址：基于接口别名在同一个物理接口上配置多个IP地址，从而支持在同一个物理接口上接入多个子网；
keepalived
Keepalived是基于VRRP协议实现的保证集群高可用的一个服务软件，运行在LVS之上，它的主要功能是实现真机的故障隔离及负载均衡器间的失败切换FailOver，可以防止单点故障。LVS结合keepalived，就实现了3层、4层、5/7层交换的功能
ka能够根据配置文件中定义自动生成ipvs规则（增、删），并能够对各RS的健康状态进行检测；
支持vrrp_script接口，vrrp_track；
通过编写vrrp脚本，通过vrrp_track跟踪各种服务；
keepalived组件
keepalived架构
控制面板（组件）：配置文件分析器，内存管理，IO复用
核心组件：vrrp stack，checker，ipvs wrapper，watch dog，smtp接口
只支持简单字符串认证
| 组件 | |
|---|---|
| vrrp stack | 实现vrrp协议 |
| smtp | 发送通知邮件；可在地址流动时发邮件，还可根据checkers检查后把宕机从从ipvs规则中移除，发出此邮件 |
| checkers | 检测后端RS健康状态，可基于tcp、htp、ssl、misc检测；发现宕机就从ipvs规则中移除 |
| watchdog | 实现监控vrrp stack和checkers的健康 |
| ipvs wrapper | checkers通过调用该包装器，来实现在ipvs中添加、删除或修改规则 |
HA Cluster配置的前提：（各调度器）
（1）各节点时间同步
基于ntp协议同步，chrony（centos 7）
（2）确保iptables及selinux不会阻碍
心跳信息传递：3种方式
- 单播
- 广播
- 组播：最佳使用方式；组播地址：224.0.0.0-239.0.0.0
（3）各节点之间可通过主机名相互通信（对ka并非必须，ka是无所谓的）
名称解析服务的解析结果必须与“uname -n”命令的结果一致；
如果是主机名通信，dns解析名必须与真实主机名一致；
（4）各节点之间的root用户可以基于密钥认证的sh通信，（对ka并非必须，ka无所谓）
配置示例
准备两台centos7主机，先做时间同步：
[root@node1 ~]# ntpdate 172.18.0.1
创建自动计划任务，向时间服务器每5分钟同步一次时间
[root@node1 ~]# crontab -e
*/5 * * * * /usr/sbin/ntpdate 172.18.0.1 &> /dev/null
安装keepalived并配置
[root@node1 ~]# yum -y install keepalived
...
Complete!
[root@node1 ~]# rpm -ql keepalived
/etc/keepalived
/etc/keepalived/keepalived.conf
/etc/sysconfig/keepalived
/usr/bin/genhash
/usr/lib/systemd/system/keepalived.service
/usr/libexec/keepalived
/usr/sbin/keepalived
...
[root@node1 ~]# cd /etc/keepalived/
[root@node1 keepalived]# cp keepalived.conf{,.bak}
[root@node1 keepalived]# vim /etc/keepalived/keepalived.conf
:set nohlsearch
:.,$s/^/#/g
#先只配置全局段和虚拟路由实例段，其它都注释；
global_defs { #全局配置段
notification_email { #接收通告邮件的地址，收件人
root@localhost
}
notification_email_from kaadmin@magedu.com #发件人
vrrp_iptables #不自动生产iptables规则
smtp_server 127.0.0.1 #配置邮件服务器，要求能在互联网上正反解析；
smtp_connect_timeout 30 #邮件服务器的超时时长
router_id node1 #定义当前路由器设备的ID号，一般为主机名；
vrrp_mcast_group4 224.0.11.18 #组播地址，默认为224.0.0.18，但是集群里的所有节点都有使用此地址；表示同一个组播域；
}
vrrp_instance VI_1 { #虚拟路由器配置段，实例名称为VI_1，也可自定义；
state MASTER #当前节点在此虚拟路由器中的初始状态；表示为主节点；取值为：MASTER|BACKUP；
interface eno16777736 #表示vrrp发送通告的接口；
virtual_router_id 11 #虚拟路由器ID，范围0-255；
priority 100 #当前物理节点在此虚拟路由器中的优先级；
advert_int 1 #表示vrrp的通告的时间间隔；为1秒；
authentication { #认证方式
auth_type PASS #使用简单密码认证
auth_pass vHFHLlTA #随机随机字符串，可使用openssl rand -base64 10生成，前8位字符有效；；
}
virtual_ipaddress { #定义虚拟IP，不指明则为前面的interface指明的接口地址；
172.18.11.66 #网卡别名地址，可写为172.18.11.66 dev eno16777736 label eno16777736:0
}
}
配置文件复制到另一调度器172.18.11.121上
[root@node1 ~]# scp keepalived.conf root@172.18.11.121:/etc/keepalived
在172.18.11.121编辑该配置文件
[root@node2 ~]# vim /etc/keepalived/keepalived.conf
修改：
state BACKUP #修改为备用
priority 50 #修改小优先级；
#密码、router_id、vrrp_mcast_group4要保存一致；
在两个调度器上开启keepalived服务
[root@node1 ~]# systemctl start keepalived.service
[root@node2 ~]# systemctl start keepalived.service
分别查看两个调度器，VIP地址是否自动在主服务器的接口别名上：
[root@node1 ~]# ip addr l
[root@node2 ~]# ip addr l
[root@node1 ~]# tail /var/log/messages
[root@node2 ~]# tail /var/log/messages
手动使172.18.11.111主用服务器停止keepalived服务，再查看
[root@node1 ~]# ip addr l]# systemctl stop keepalived.service
用另一终端查看日志信息不退出
[root@centos7 ~]# tail -f /var/log/messages
显示部分内容：
May 10 05:25:01 localhost systemd: Stopping LVS and VRRP High Availability Monitor...
May 10 05:25:01 localhost Keepalived_vrrp[4258]: VRRP_Instance(VI_1) sending 0 priority 发送优先级为0的报文；
May 10 05:25:01 localhost Keepalived_vrrp[4258]: VRRP_Instance(VI_1) removing protocol VIPs. 移除VIP的协议
May 10 05:25:01 localhost Keepalived_healthcheckers[4257]: Netlink reflector reports IP 172.18.11.66 removed 移除VIP
May 10 05:25:01 localhost systemd: Stopped LVS and VRRP High Availability Monitor
在另一调度器172.18.11.121上，查看日志
[root@node2 ~]# tail /var/log/messages
May 9 08:09:21 localhost Keepalived_vrrp[3881]: VRRP_Instance(VI_1) Transition to MASTER STATE 称为主用服务器
May 9 08:09:22 localhost Keepalived_vrrp[3881]: VRRP_Instance(VI_1) Entering MASTER STATE 确认主用状态
May 9 08:09:22 localhost Keepalived_vrrp[3881]: VRRP_Instance(VI_1) setting protocol VIPs. 设置
May 9 08:09:22 localhost Keepalived_vrrp[3881]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eno16777736 for 172.18.11.66 发送免费ARP报文
May 9 08:09:22 localhost Keepalived_healthcheckers[3880]: Netlink reflector reports IP 172.18.11.66 added 获得VIP地址
May 9 08:09:27 localhost Keepalived_vrrp[3881]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eno16777736 for 172.18.11.66
May 9 08:10:01 localhost systemd: Started Session 58 of user root.
May 9 08:10:01 localhost systemd: Starting Session 58 of user root.
May 9 08:10:01 localhost systemd: Started Session 59 of user root.
May 9 08:10:01 localhost systemd: Starting Session 59 of user root.
再手动使172.18.11.111主用服务器上线：
[root@node1 ~]# systemctl start keepalived.service
#默认工作在抢占模式；优先级高的会自动成为主用服务器；
这就是keepalived自动完成地址浮动；
修改两个调度器的VIP别名：
[root@node1 ~]# vim /etc/keepalived/keepalived.conf
virtual_ipaddress{
172.18.11.66 dev eno16777736 label eno16777736:0
}
把两个调度器停止keepalived服务：
先启动优先级低的服务器：172.18.11.121：
[root@node2 ~]# systemctl start keepalived.service
此时使用ifconfig命令，engine查看网卡别名，已经自动加上了；
再启动优先级高的服务器：172.18.11.111：
[root@node1 ~]# systemctl start keepalived.service
vrrp有两种工作模型：主/备，主/主
为了提高利用率，就使用主/主工作模型；
即A主B从，B主A从；这样AB都能发挥作用；在外部都称为主时，可在前端dns配置A记录把两个ip都配置在一个主机名上即可；
但在有些场景中，AB都称为主时不是配置在同一网卡上的两个不同的别名，而是分别配置在两块网卡上；
AB每个主机都有两块网卡，一块网卡面向外网，一块网卡面向内网通信；期望通信时，这两个地址都是VIP，只不过，前面外网是虚拟路由器1，内网是虚拟路由器2；
里面的别名配置在内网网卡上，外面的别名配置在外网网卡上；当外网其中一个线路掉线时，可自动切换到另一个外网网卡的服务器上，要求而也要内网同样要切换；要求内外通信使用VIP，所以没有必要切换的内网也要切换；
这种把两个虚拟路由器，配置为只能对一个设备是主服务器，同进同退，这种就叫做一个同步组；
安装keepalived
Centos6.4+，程序包已经在base源中提供；
centos 7
主配置文件：/etc/keepalived/keepalived.conf
Unit file：/usr/lib/systemd/system/keepalived.service
配置文件：/etc/sysconfig/keepalived
配置文件内容块：
GLOBAL CONFIGURATION
#全局配置段，对所有vvrp实例虚拟服务器都是生效；
global_defs {
...
}
VRRPD CONFIGURATION #虚拟路由器同步组配置段
vrrp_sync_group VG_1 {
...
}
VRRP instance(s) #虚拟路由器实例配置段
vrrp_instance INST_NAME {
...
}
LVS CONFIGURATION #集群配置段
virtual_server_group <STRING> { #虚拟服务器组配置段
...
}
Virtual server(s) #虚拟服务器配置段
virtual_server IP port |
virtual_server fwmark int { #十进制的整数，（实际最终用的是十六进制数）
protocol TCP #注意：keepalived只支持tcp；
...
real_server <IPADDR> <PORT>{ #真实后端服务器配置段
...
}
}
全局配置段示例
global_defs {
notification_email { #用于指定报警邮件发往的邮箱地址
root@localhost
}
notification_email_from keepalived@localhost #用于指定报警邮件的发件人
smtp_server 127.0.0.1 #用于指定邮件服务器的地址
smtp_connect_timeout 30 #用于指定邮件服务器的连接超时时长
router_id node1 #路由器的标识
vrrp_mcast_group4 224.0.100.19 #用于设置vrrp的广播地址，在同一个HA cluster中的机器，要确保其广播地址一致才能接受到相应的vrrp报文
vrrp_strict #执行严格的vrrp协议检查，某些情况下会禁止到vip的访问。
}
Static routes/address/rules：用于配置keepalived中不会被VRRP移除的静态地址、路由或者规则，一般不会使用。
VRRP相关配置段示例
vrrp_instance <STRING> {
state MASTER|BACKUP #用于指定此虚拟路由器在vrrp组的角色状态
interface eno16777736 #用于绑定当前虚拟路由器所使用的物理接口
virtual_router_id 14 #用于指定当前虚拟路由器在vrrp组的唯一标识id，范围为0-255
priority 100 #用于设置当前虚拟路由器在vrrp组里面的优先级，范围为1-254；
advert_int 1 #用于设置虚拟路由器发送vrrp通告的时间间隔
nopreempt|preempt #定义工作模式为非抢占或抢占模式；
preempt_delay 300 #定义在抢占式模式下，节点上线后触发新选举操作的延迟时长；
authentication { #用于设置vrrp组协商的方式及密码
auth_type PASS #定义认证类型为简单密码认证
auth_pass 571f97b2 #定义密码串，最长不超过8个字符
}
virtual_ipaddress { #用于指定需要在绑定的物理接口上添加的虚拟ip地址
#<IPADDR>/<MASK> brd <IPADDR> dev <STRING> scope <SCOPE> label <LABEL>
172.18.11.66/16 dev eno16777736
}
track_interface { #配置需要监控的网络接口，一旦接口出现故障，此虚拟路由器转为FAULT状态
eth0
eth1
...
}
track_script { #用于调用在vrrp_script中定义的脚本，然后根据脚本的来进行监控调整
<SCRIPT_NAME>
<SCRIPT_NAME> weight <-254..254>
}
notify_master <STRING>|<QUOTED-STRING> #当前节点成为主节点时触发的通知脚本；
notify_backup <STRING>|<QUOTED-STRING> #当前节点转为备节点时触发的通知脚本；
notify_fault <STRING>|<QUOTED-STRING> #当前节点转为“失败”状态时触发的通知脚本；
notify_stop <STRING>|<QUOTED-STRING> #当前节点停止时所触发的通知脚本
}
VRRP script(s)：用于定义周期性执行的脚本，可定义调用用于检查相应的服务或Ip的状态的脚本。
vrrp_script <SCRIPT_NAME> { #定义周期执行的脚本，此脚本的退出码决定了当前监控的vrrp组的运行状态
script <STRING>|<QUOTED-STRING> #定义执行脚本的存放路径
interval INT #定义调用执行脚本的周期，默认为1s。
timeout <INTEGER> # 脚本执行超时时间，脚本执行超时后，则被认为失败
rise <INTEGER> # 定义脚本检查成功多少次，才认可当前的状态为正常
fall <INTEGER> #定义检查失败多少次，才认为当前状态为失败
}
LVS配置段示例
Virtual server(s)：用于定义虚拟服务器的设置，虚拟服务器可以用Ip端口、fwmark或virtual server group来定义。
virtual_server IP port | virtual_server fwmark <int> |virtual_server group string
{
delay_loop <INT> #=服务轮询的时间间隔；
lb_algo rr|wrr|lc|wlc|lblc|sh|dh #定义调度方法；
lb_kind NAT|DR|TUN #集群的类型；
persistence_timeout <INT> #持久连接时长；
protocol TCP|UDP|SCTP #服务协议；
sorry_server <IPADDR> <PORT> #备用服务器地址；
real_server <IPADDR> <PORT>{
weight <INT> #设置real server的权重
notify_up <STRING>|<QUOTED-STRING> #当出现匹配字符串时，就认为服务是up的
notify_down <STRING>|<QUOTED-STRING> #当出现匹配字符串时，就认为服务是down的
HTTP_GET|SSL_GET { #对real server作应用层检测
url {
path <URL_PATH> #定义要监控的URL；
status_code <INT> #判断上述检测机制为健康状态的响应码；
digest <STRING> #判断上述检测机制为健康状态的响应的内容的校验码；
}
nb_get_retry <INT> #重试次数；
delay_before_retry <INT> #重试之前的延迟时长；
connect_ip <IP ADDRESS> #向当前RS的哪个IP地址发起健康状态检测请求
connect_port <PORT> #向当前RS的哪个PORT发起健康状态检测请求
bindto <IP ADDRESS> #发出健康状态检测请求时使用的源地址；
bind_port <PORT> #发出健康状态检测请求时使用的源端口；
connect_timeout <INTEGER> #连接请求的超时时长；
}
TCP_CHECK {
connect_ip <IP ADDRESS> #向当前RS的哪个IP地址发起健康状态检测请求
connect_port <PORT> #向当前RS的哪个PORT发起健康状态检测请求
bindto <IP ADDRESS> #发出健康状态检测请求时使用的源地址；
bind_port <PORT> #发出健康状态检测请求时使用的源端口；
connect_timeout <INTEGER> #连接请求的超时时长；
}
}
}
高可用的ipvs集群
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node1
vrrp_mcast_group4 224.0.100.19
}
vrrp_instance VI_1 {
state MASTER
interface eno16777736
virtual_router_id 14
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 571f97b2
}
virtual_ipaddress {
10.1.0.93/16 dev eno16777736
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
virtual_server 10.1.0.93 80 {
delay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 10.1.0.69 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.1.0.71 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
TCP_CHECK {
nb_get_retry 3
delay_before_retry 2
connect_timeout 3
}
}
}
双主模型
主机配置：
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node1
vrrp_mcast_group4 224.0.100.19
}
vrrp_instance VI_1 {
state MASTER
interface eno16777736
virtual_router_id 14
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 571f97b2
}
virtual_ipaddress {
172.18.11.66/16 dev eno16777736
}
}
vrrp_instance VI_2 {
state BACKUP
interface eno16777736
virtual_router_id 15
priority 98
advert_int 1
authentication {
auth_type PASS
auth_pass 578f07b2
}
virtual_ipaddress {
172.18.11.66/16 dev eno16777736
}
}
备机配置：
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node2
vrrp_mcast_group4 224.0.100.19
}
vrrp_instance VI_1 {
state BACKUP
interface eno16777736
virtual_router_id 16
priority 98
advert_int 1
authentication {
auth_type PASS
auth_pass 571f97b2
}
virtual_ipaddress {
172.18.11.66/16 dev eno16777736
}
}
vrrp_instance VI_2 {
state MASTER
interface eno16777736
virtual_router_id 17
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 578f07b2
}
virtual_ipaddress {
172.18.11.66/16 dev eno16777736
}
}
通告脚本
可定义短信网关、微信网关、邮件服务器等方式发送；
此处，仅以本地邮件服务编写脚本；
[root@node1 ~]# vim notify.sh
#!/bin/bash
#
contact='root@localhost'
notify(){
mailsuject="$(hostname) to be $1:vip floating"
mailbody="$(date +'%F %T'):vrrp transition,$(hostname) change to be $1"
echo $mailbody |mail -s "$mailsubject" $contact
}
case $1 in
master)
notify master
;;
backup)
notify bakcup
;;
fault)
notify fault
;;
*)
echo "Usage :$(basename $0) {master|backup|fault}"
;;
esac
[root@node1 ~]# chmod +x notify.sh
[root@node1 ~]# ./notify.sh master
[root@node1 ~]# mail
显示：
Heirloom Mail version 12.5 7/5/10. Type ? for help.
"/var/spool/mail/root": 1 message 1 new
>N 1 root Tue May 10 05:55 18/682 ""
& 1 输入1查看邮件1
部分内容：
2016-05-10 05:55:33:vrrp transition,localhost.localdomain change to be master
exit 退出邮件
把通告脚本复制给另一调度器上
[root@node1 ~]# scp notify.sh root@172.18.11.121:/etc/keepalived
在两个调度器上配置通告脚本
[root@node2 ~]# vim /etc/keepalived/keepalived.conf
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
把两个调度器停止keepalived服务器
[root@node1 ~]# systemctl stop keepalived.service
[root@node2 ~]# systemctl stop keepalived.service
[root@node1 ~]# mail
[root@node2 ~]# mail
删除已有邮件，为后续清空环境；
d 1
d 2
再先后启动两个调度器keepalived服务器
先启动优先级低的备用调度器：
[root@node2 ~]# systemctl start keepalived.service
[root@node2 ~]# mail
显示：刚启动发一个邮件为backup的，因为只有一个调度器，过一会又收到一个邮件成为master；
再启动优先级高的调度器：
[root@node1 ~]# mail
则查看，显示也收到2条邮件，均为master信息；
通告脚本示例：
#!/bin/bash
#
contact='root@localhost'
notify(){
mailsuject="$(hostname) to be $1:vip floating"
mailbody="$(date +'%F %T'):vrrp transition,$(hostname) change to be $1"
echo $mailbody |mail -s "$mailsubject" $contact
}
case $1 in
master)
notify master
;;
backup)
notify bakcup
;;
fault)
notify fault
;;
*)
echo "Usage :$(basename $0) {master|backup|fault}"
;;
esac
调用方法：
vrrp_instanace <STRING>{
...
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
注意：脚本路径要使用双引号；
示例：keepalived的主从架构
搭建RS1和RS2
首先安装nginx程序：
[root@RS1 ~]# yum install -y epel-release
[root@RS1 ~]# yum install -y nginx
然后编辑/etc/hosts文件：
[root@RS1 ~]# vim /etc/hosts
192.168.0.81 DR1.ilinux.io DR1
192.168.0.87 DR2.ilinux.io DR2
192.168.0.83 RS1.ilinux.io RS1
192.168.0.84 RS2.ilinux.io RS2
随后修改/usr/share/nginx/html/index.html 内容为如下：
[root@RS1 ~]# vim /usr/share/nginx/html/index.html
<h1>This is RS1 192.168.0.83</h1>
启动nginx：
[root@RS1 ~]# systemctl start nginx
关闭firewalld和修改selinux的状态为permissive：
[root@RS1 ~]# systemctl stop firewalld
[root@RS1 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@RS1 ~]# setenforce 0
接着在RS1上配置lvs-dr的配置，首先创建rs脚本：
[root@RS1 ~]# vim RS.sh
#/bin/bash
vip=192.168.0.99
mask='255.255.255.255'
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig lo:0 $vip netmask $mask broadcast $vip up
route add -host $vip dev lo:0
;;
stop)
ifconfig lo:0 down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
;;
*)
echo "Usage $(basename $0) start|stop "
exit 1
;;
esac
随后执行RS脚本：
[root@RS1 ~]# bash -x RS.sh start
+ vip=192.168.0.99
+ mask=255.255.255.255
+ case $1 in
+ echo 1
+ echo 1
+ echo 2
+ echo 2
+ ifconfig lo:0 192.168.0.99 netmask 255.255.255.255 broadcast 192.168.0.99 up
+ route add -host 192.168.0.99 dev lo:0
重复以上步骤并结合相应的信息搭建RS2。
搭建DR1
首先yum安装keepalived和ipvsadm程序包：
[root@DR1 ~]# yum install -y ipvsadm keepalived
然后编辑/etc/hosts文件：
[root@DR1 ~]# vim /etc/hosts
192.168.0.81 DR1.ilinux.io DR1
192.168.0.87 DR2.ilinux.io DR2
192.168.0.83 RS1.ilinux.io RS1
192.168.0.84 RS2.ilinux.io RS2
随后编辑/etc/keepalived/keepalived.conf文件：
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id DR1
vrrp_mcast_group4 224.0.0.0.20
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state MASTER
interface eno16777736
virtual_router_id 1
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass ^&IUYH*&
}
virtual_ipaddress {
192.168.0.99/24 dev eno16777736 label eno16777736:0
}
}
virtual_server 192.168.0.99 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.0.83 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.0.84 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
启动keepalived，查看ipvsadm和接口的状态：
[root@DR1 ~]# systemctl start keepalived
[root@DR1 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.99:http rr
-> 192.168.0.83:http Route 1 1 4
-> 192.168.0.84:http Route 1 1 80
[root@DR1 ~]# ifconfig
eno16777736: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.81 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::20c:29ff:fe21:59b9 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:21:59:b9 txqueuelen 1000 (Ethernet)
RX packets 62277 bytes 72099132 (68.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 21742 bytes 2744670 (2.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eno16777736:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.99 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:21:59:b9 txqueuelen 1000 (Ethernet)
关闭firewalld和修改selinux的状态为permissive：
[root@DR1 ~]# systemctl stop firewalld
[root@DR1 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@DR1 ~]# setenforce 0
搭建DR2
首先yum安装keepalived和ipvsadm程序包：
[root@DR2 ~]# yum install -y ipvsadm keepalived
然后编辑/etc/hosts文件：
[root@DR2 ~]# vim /etc/hosts
192.168.0.81 DR1.ilinux.io DR1
192.168.0.87 DR2.ilinux.io DR2
192.168.0.83 RS1.ilinux.io RS1
192.168.0.84 RS2.ilinux.io RS2
随后编辑/etc/keepalived/keepalived.conf文件：
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id DR2
vrrp_mcast_group4 224.0.0.0.20
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 1
priority 98
advert_int 1
authentication {
auth_type PASS
auth_pass ^&IUYH*&
}
virtual_ipaddress {
192.168.0.99/24 dev ens33 label ens33:0
}
}
virtual_server 192.168.0.99 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.0.83 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.0.84 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
最后启动keepalived并查看ipvsadm的状态：
[root@DR2 ~]# systemctl start keepalived
[root@DR2 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.99:80 rr
-> 192.168.0.83:80 Route 1 0 0
-> 192.168.0.84:80 Route 1 0 0
[root@DR2 ~]# ifconfig #因为DR2的角色为BACKUP，因此不会创建Ip为192.168.0.99的子接口
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.87 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::5e4b:719d:3781:43a0 prefixlen 64 scopeid 0x20<link>
inet6 fe80::4c6e:8b7e:2dcd:665d prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:26:a3:20 txqueuelen 1000 (Ethernet)
RX packets 34041 bytes 27376635 (26.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 14131 bytes 2169836 (2.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 101 bytes 8902 (8.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 101 bytes 8902 (8.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
关闭firewalld和修改selinux的状态为permissive：
[root@DR2 ~]# systemctl stop firewalld
[root@DR2 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@DR2 ~]# setenforce 0
测试访问
在客户端上测试访问vip所提供的服务：
[root@client ~]# for i in {1..10} ; do curl http://192.168.0.99; done
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
此时DR1为主机，DR2为备机，两者都工作正常，因此请求连接是由DR1来处理。
那么我们模拟DR1宕机，把DR1的服务停用后，再来观察DR2的状态和客户端的访问情况。
DR2的keepalived状态：
[root@DR2 ~]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2018-05-29 14:34:08 CST; 1h 29min ago
Process: 11808 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 11809 (keepalived)
CGroup: /system.slice/keepalived.service
├─11809 /usr/sbin/keepalived -D
├─11810 /usr/sbin/keepalived -D
└─11811 /usr/sbin/keepalived -D
May 29 16:03:06 DR2 Keepalived_vrrp[11811]: VRRP_Instance(VI_1) removing protocol VIPs.
May 29 16:03:47 DR2 Keepalived_vrrp[11811]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: VRRP_Instance(VI_1) Entering MASTER STATE
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: VRRP_Instance(VI_1) setting protocol VIPs.
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: Sending gratuitous ARP on ens33 for 192.168.0.99
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.0.99
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: Sending gratuitous ARP on ens33 for 192.168.0.99
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: Sending gratuitous ARP on ens33 for 192.168.0.99
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: Sending gratuitous ARP on ens33 for 192.168.0.99
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: Sending gratuitous ARP on ens33 for 192.168.0.99
DR2检测到DR1的宕机，主动变成了MASTER状态。
在客户端的访问情况：
[root@client ~]# for i in {1..10} ; do curl http://192.168.0.99; done
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
此时DR1模拟为宕机状态，DR2为MASTER，客户端的访问不受影响。
示例：Keepalived的主主架构
此处以上面主从架构的拓扑为例，将主从架构更改为主主架构，首先我们需更改DR1和DR2的keepalived的配置，然后要在RS1和RS2上添加lvs-dr中与192.168.0.98虚拟IP相关的配置。
DR模式开启ip转发
[root@DR1 ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
修改DR1
编辑/etc/keepalived/keepalived.conf文件
[root@DR1 ~]# vim /etc/keepalived/keepalived.conf
#添加如下配置
vrrp_instance VI_2 {
state BACKUP
interface eno16777736
virtual_router_id 2
priority 98
advert_int 1
authentication {
auth_type PASS
auth_pass POM123(*
}
virtual_ipaddress {
192.168.0.98/24 dev eno16777736 label eno16777736:1
}
}
#添加虚拟服务器组backend
virtual_server_group backend {
192.168.0.99 80
192.168.0.98 80
}
#修改虚拟服务器调用虚拟服务器组
virtual_server group backend {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.0.83 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.0.84 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
停用再启动keepalived：
[root@DR1 ~]# systemctl stop keepalived
[root@DR1 ~]# systemctl start keepalived
此时ipvsadm和接口的状态为：
[root@DR2 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.98:80 rr
-> 192.168.0.83:80 Route 1 0 0
-> 192.168.0.84:80 Route 1 0 0
TCP 192.168.0.99:80 rr
-> 192.168.0.83:80 Route 1 0 0
-> 192.168.0.84:80 Route 1 0 0
[root@DR1 ~]# ifconfig
eno16777736: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.81 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::20c:29ff:fe21:59b9 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:21:59:b9 txqueuelen 1000 (Ethernet)
RX packets 63844 bytes 72282542 (68.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 23654 bytes 2934901 (2.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eno16777736:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.99 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:21:59:b9 txqueuelen 1000 (Ethernet)
修改DR2
编辑/etc/keepalived/keepalived.conf文件：
[root@DR2 ~]# vim /etc/keepalived/keepalived.conf
#添加如下配置
vrrp_instance VI_2 {
state MASTER
interface ens33
virtual_router_id 2
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass POM123(*
}
virtual_ipaddress {
192.168.0.98/24 dev ens33 label ens33:1
}
}
#添加虚拟服务器组backend
virtual_server_group backend {
192.168.0.99 80
192.168.0.98 80
}
#修改虚拟服务器调用虚拟服务器组
virtual_server group backend {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.0.83 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.0.84 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
停用再启动keepalived：
[root@DR2 ~]# systemctl stop keepalived
[root@DR2 ~]# systemctl start keepalived
此时ipvsadm和接口的状态为：
[root@DR2 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.98:80 rr
-> 192.168.0.83:80 Route 1 0 0
-> 192.168.0.84:80 Route 1 0 0
TCP 192.168.0.99:80 rr
-> 192.168.0.83:80 Route 1 0 0
-> 192.168.0.84:80 Route 1 0 0
[root@DR2 ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.87 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::5e4b:719d:3781:43a0 prefixlen 64 scopeid 0x20<link>
inet6 fe80::4c6e:8b7e:2dcd:665d prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:26:a3:20 txqueuelen 1000 (Ethernet)
RX packets 39989 bytes 28047325 (26.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 20816 bytes 2894556 (2.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.98 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:26:a3:20 txqueuelen 1000 (Ethernet)
配置RS1和RS2
复制编辑RS脚本：
[root@RS1 ~]# cp RS.sh RS_new.sh
#/bin/bash
#修改vip为192.168.0.98
vip=192.168.0.98
mask='255.255.255.255'
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig lo:1 $vip netmask $mask broadcast $vip up #修改接口为lo:1
route add -host $vip dev lo:1 #修改接口为lo:1
;;
stop)
ifconfig lo:1 down #修改接口为lo:1
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
;;
*)
echo "Usage $(basename $0) start|stop "
exit 1
;;
esac
运行脚本：
[root@RS1 ~]# bash -x RS_new.sh start
+ vip=192.168.0.98
+ mask=255.255.255.255
+ case $1 in
+ echo 1
+ echo 1
+ echo 2
+ echo 2
+ ifconfig lo:1 192.168.0.98 netmask 255.255.255.255 broadcast 192.168.0.98 up
+ route add -host 192.168.0.98 dev lo:1
在RS2 上也按照如上步骤执行操作即可
测试访问
此时在客户端通过192.168.0.99和192.168.0.98均能访问到后端RS所提供的web服务：
[root@client ~]# for i in {1..10} ; do curl http://192.168.0.98; done
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
[root@client ~]# for i in {1..10} ; do curl http://192.168.0.99; done
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
在客户端上编辑/etc/hosts，添加域名解析到99和98：
[root@client ~]# vim /etc/hosts
192.168.0.99 www.ilinux.io
192.168.0.98 www.ilinux.io
此时通过域名解析能使得只要有99和98有一个正常工作，客户端均能正常访问服务。
[root@client ~]# for i in {1..10} ; do curl http://www.ilinux.io; done
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
<h1>This is RS2 192.168.0.84</h1>
<h1>This is RS1 192.168.0.83</h1>
