备份容灾

一、备份

1.手动备份redis数据库

#!/bin/bash
# 方式1.通过redis-cli内置命令将内存中的数据存储到rdb文件中
echo "auth 123456\nping\nsave\n" | redis-cli -h 127.0.0.1 -p 6379
echo "auth 123456\nping\nbgsave\n" | redis-cli -h 127.0.0.1 -p 6379

# 方式2.将远程主机Redis-Server中存储的数据保存到本地/tmp/backup/目录中。
redis-cli -h 127.0.0.1 -p 6379 --rdb /tmp/backup/app-6379.rdb

# 方式3.定时执行拷贝rdb与aof文件进行备份。
# 例如: 每天0点执行一次 0 0 * * * sh /tmp/redisBackup.sh
current_date=$(date +%Y%m%d-%H%M%S)
BACKUPDIR="/backup/redis"
RDBFILE="/data/dump.rdb"
AOFFILE="/data/appendonly.aof"
# del_date=$(date -d -1day +%Y%m%d)
if [ ! -d ${BACKUPDIR} ];then
  mkdir -vp ${BACKUPDIR}
fi

if [ ! -f ${RDBFILE} ];then
  cp -a ${RDBFILE} ${BACKUPDIR}/${current_date}-dump.rdb
fi

if [ ! -f ${AOFFILE} ];then
  cp -a ${AOFFILE} ${BACKUPDIR}/${current_date}-appendonly.aof
fi

# 删除七天前的备份
find ${BACKUPDIR} -type f -mtime +7 >> delete.log
find ${BACKUPDIR} -type f -mtime +7 -exec rm -rf {} \; 

2.迁移Redis指定db-数据库

方式1.同主机db迁移到另外一个dbn中

$ redis-cli -h localhost -a weiyigeek.top -n 0 keys "*" | while read key
do
  redis-cli -h localhost -a weiyigeek.top -n 0 --raw dump $key | perl -pe 'chomp if eof' | redis-cli -h localhost -a weiyigeek.top -n 12 -x restore $key 0
done 

方式2.跨主机迁移db

# redis 把db2 的数据迁移到 db14 里 (注意:某些格式的数据不能完全以此种方式进行迁移)
# 需求分析:
'''1、建立两个redis连接
   2、获取所有的keys（）
   3、获取keys的类型:string hash'''

import redis
src_redis = redis.Redis(host='211.149.218.16',
                password='123456',
                port=6379,
                db=2)
target_redis = redis.Redis(host='211.149.218.16',
                password='123456',
                port=6379,
                db=14)

for key in src_redis.keys(): # redis获取的数据都是bytes类型的，所以key的类型是 bytes 类型
    if src_redis.type(key) == b'string':# 也可以用decode() 把key转换成string，这样等号右边就不需要加b
        v = src_redis.get(key)  #先获取原来的数据
        target_redis.set(key,v) #set到新的数据库里
    else:
        all_hash_data = src_redis.hgetall(key) # 获取hash类型里面所有的数据，获取出来的数据是字典格式的  但是有b，需要转换
        for k,v in all_hash_data.items():# 因为获取到字典格式的hash类型的原数据有b，所以需要用for循环来进行转换后重新赋值给新的数据库
            target_redis.hset(key,k,v) # key是外面的，k是里面的key，v是k对应的value 

3.Redis集群数据备份与迁移

描述: 当我们需要备份或迁移Redis集群时可以采用以下方案。

# (1) 备份集群数据到本地目录中(已rdb格式文件存储)。
redis-cli -a weiyigeek --cluster backup 172.16.243.97:6379 .
  # >>> Node 172.16.243.97:6379 -> Saving RDB...
  # SYNC sent to master, writing 178 bytes to './redis-node-172.16.243.97-6379-d97cb5b15b7130ca0bd5322758e0c2dce061fd7b.rdb'
  # Transfer finished with success.
  # >>> Node 172.16.183.95:6379 -> Saving RDB...
  # SYNC sent to master, writing 178 bytes to './redis-node-172.16.183.95-6379-94b8d3748dc47053454e657da8d6bb90e0081f2c.rdb'
  # Transfer finished with success.
  # >>> Node 172.16.24.214:6379 -> Saving RDB...
  # SYNC sent to master, writing 178 bytes to './redis-node-172.16.24.214-6379-2674f21a88a9573f51ec46f9dc248ad4a5c5974d.rdb'
  # Transfer finished with success.

# (2) 把 192.168.1.187:6379 上的数据导入到 192.168.75.187:6379 这个节点所在的集群，如有密码将询问
redis-cli -a weiyigeek --cluster import 192.168.75.187:6379 --cluster-from 192.168.1.187:6379 --cluster-from-askpass --cluster-copy

# (3) 迁移后利用 dbsize 命令查看数据是否正确 

第三方redis集群数据迁移工具项目参考(https://github.com/alibaba/RedisShake)

二、恢复

1.系统Redis用户被删除后配置数据恢复流程

描述:在系统删除了配置文件后以及用户账号后恢复方法流程，实际环境中建议利用rdb文件进行重新部署。

• Step1.Redis账户数据恢复首先确定系统中是否还有redis用户。（如果拷贝过来的系统也安装了redis，那么肯定是会有redis账户）

• Step2.如果发现有redis用户以下步骤可以跳过，否则进行手动添加。

echo "redis:x:996:994:Redis Database Server:/var/lib/redis:/sbin/nologin" >> /etc/passwd
echo "redis:!!:17416::::::" >> /etc/shadow
echo "redis:x:994:" >> /etc/group
echo "redis:!::" >> /etc/gshadow 

• Step3.Redis配置文件恢复, Redis的配置文件恢复相对简单一些，官方提供了命令重写redis.conf配置文件。

redis-cli
> CONFIG REWRITE
OK 

• Step4.修改配置文件权限

touch /etc/redis.conf
chown redis:redis /etc/redis.conf 

2.Kubernetes中单实例异常数据迁移恢复实践
方案1.利用其他kubernetes集群进行恢复原k8s集群的redis数据。

#!/bin/bash
# author: WeiyiGeek
# usage: ./K8SRedisRecovery.sh [aof|rdb] redis原持久化目录
BACKUP_TYPE=$1
BACKUP_DIR=$2
DATA_DIR="$(pwd)/data"
echo "开始时间: $(date +%s)"

# 1.判断备份文件以及持久化文件是否存在
if [ ! -d ${BACKUP_DIR} ];then echo -e "[Error] - Not Found ${BACKUP_DIR} Dirctory!"; return -1; fi
if [ ! -d ${DATA_DIR} ];then mkdir -vp ${DATA_DIR};else rm -rf "${DATA_DIR}/*"; fi

# 2.redis配置与k8s部署恢复redis清单
# tee redis.conf <<'EOF'
# bind 0.0.0.0
# port 6379
# daemonize no
# supervised no
# protected-mode no
# requirepass "weiyigeek"
# dir "/data"
# pidfile "/var/run/redis.pid"
# logfile "/var/log/redis.log"
# loglevel verbose
# maxclients 10000
# timeout 300
# tcp-keepalive 60
# maxmemory-policy volatile-lru
# slowlog-max-len 128
# lua-time-limit 5000
# save 300 10
# save 60 10000
# dbfilename "dump.rdb"
# rdbcompression yes
# rdb-save-incremental-fsync yes
# # appendonly yes
# appendfilename "appendonly.aof"
# appendfsync everysec
# # rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
# rename-command FLUSHDB WeiyiGeekFLUSHDB
# rename-command FLUSHALL WeiyiGeekFLUSHALL
# rename-command EVAL WeiyiGeekEVAL
# rename-command DEBUG WeiyiGeekDEBUG
# rename-command SHUTDOWN WeiyiGeekSHUTDOWN
# EOF

tee redisrecovery.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-recovery
  namespace: database
data:
  redis.conf: |+
    bind 0.0.0.0
    port 6379
    daemonize no
    supervised no
    protected-mode no
    requirepass "weiyigeek"
    dir "/data"
    pidfile "/var/run/redis.pid"
    logfile "/var/log/redis.log"
    loglevel verbose
    maxclients 10000
    timeout 300
    tcp-keepalive 60
    maxmemory-policy volatile-lru
    slowlog-max-len 128
    lua-time-limit 5000
    save 300 10
    save 60 10000
    dbfilename "dump.rdb"
    rdbcompression yes
    rdb-save-incremental-fsync yes
    # appendonly yes
    appendfilename "appendonly.aof"
    appendfsync everysec
    # rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
    rename-command FLUSHDB WeiyiGeekFLUSHDB
    rename-command FLUSHALL WeiyiGeekFLUSHALL
    rename-command EVAL WeiyiGeekEVAL
    rename-command DEBUG WeiyiGeekDEBUG
    rename-command SHUTDOWN WeiyiGeekSHUTDOWN
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-recovery
  namespace: database
spec:
  serviceName: redisrecovery
  replicas: 1
  selector:
    matchLabels:
      app: redis-recovery
  template:
    metadata:
      labels:
        app: redis-recovery
    spec:
      containers:
      - name: redis
        image: redis:6.2.5-alpine
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 6379
          name: server
        command: [ "redis-server", "/conf/redis.conf"]
        volumeMounts:
        # 从configmap获取的配置文件，挂载到指定文件中
        - name: conf
          mountPath: /conf/redis.conf
          subPath: redis.conf
        - name: data
          mountPath: /data
        # 时区设置
        - name: timezone
          mountPath: /etc/localtime
      volumes:
      - name: conf
        # 配置文件采用configMap
        configMap:
          name: redis-recovery
          defaultMode: 0755
        # redisc持久化目录采用hostPath卷
      - name: data
        hostPath:
          type: DirectoryOrCreate
          path: {PersistentDir}
        # 时区定义
      - name: timezone
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
---
apiVersion: v1
kind: Service
metadata:
  name: redisrecovery
  namespace: database
spec:
  type: ClusterIP
  ports:
  - port: 6379
    targetPort: 6379
    name: server
  selector:
    app: redis-recovery
EOF

# 3.删除原有的恢复Pod
kubectl delete -f redisrecovery.yaml
# kubectl delete configmap -n database redis-recovery

# 4.判断redis备份格式
if [ "${BACKUP_TYPE}" == "aof" ];then
  sed -i "s|# appendonly yes|appendonly yes|g" redisrecovery.yaml
  cp -a ${BACKUP_DIR}/appendonly.aof ${DATA_DIR}
else
  cp -a ${BACKUP_DIR}/dump.rdb ${DATA_DIR}
fi

# 5.更新redis备份恢复k8s资源清单中的持久化目录
sed -i "s#{PersistentDir}#${DATA_DIR}#g" redisrecovery.yaml

# 6.创建configmap和部署redis备份恢复应用
# kubectl create configmap -n database redis-recovery --from-file=$(pwd)/redis.conf
kubectl create --save-config -f redisrecovery.yaml

# 7.验证Pod状态是否正常
flag=$(kubectl get pod -n database -o wide -l app=redis-recovery | grep -c "Running")
echo -e "\e[31m[Error]: Pod Status is not Running!\e[0m"
while [ ${flag} -ne 1 ];do
sleep 0.5
flag=$(kubectl get pod -n database -o wide -l app=redis-recovery | grep -c "Running")
done

# 8.验证数据是否恢复
# apt install -y redis-tools
echo "[OK] redis-recovery Status is Running"
echo -e "AUTH weiyigeek\nping\ninfo" | redis-cli -h redisrecovery.database.svc.cluster.local | grep -A 16 "# Keyspace"
while [ $? -ne 0 ];do
echo -e "AUTH weiyigeek\nping\ninfo" | redis-cli -h redisrecovery.database.svc.cluster.local | grep -A 16 "# Keyspace"
done
echo "数据恢复完成......"
echo "完成时间: $(date +%s)" 

命令执行示例:

# (1) 利用 AOF 文件进行恢复百万数据
/nfsdisk-31/datastore/redis/demo2# time ./K8sredisRecovery.sh aof /nfsdisk-31/data/database-data-redis-cm-0-pvc-00ee48e8-b1ca-4640-96c5-16f7265a2c61
  # # Keyspace
  # db0:keys=1000000,expires=0,avg_ttl=0
  # 数据恢复完成......
  # 完成时间: 1631067931

  # real    0m12.039s
  # user    0m0.647s
  # sys     0m0.199s

/nfsdisk-31/datastore/redis/demo2/data# ls -alh
  # -rw-r--r-- 1 root root  41M Sep  7 22:14 appendonly.aof
  # -rw-r--r-- 1 root root  41M Sep  8  2021 dump.rdb

# (2) 利用 rdb 文件进行恢复 百万数据
root@WeiyiGeek-107:/nfsdisk-31/datastore/redis/demo2# time ./K8sredisRecovery.sh rdb /nfsdisk-31/data/database-data-redis-cm-0-pvc-00ee48e8-b1ca-4640-96c5-16f7265a2c61
  # # Keyspace
  # db0:keys=993345,expires=0,avg_ttl=0
  # 数据恢复完成......
  # 完成时间: 1631069375

  # real    0m11.106s
  # user    0m0.606s
  # sys     0m0.192s

/nfsdisk-31/datastore/redis/demo2# ls -lah data/
  # -rw-r--r-- 1 root root  41M Sep  8  2021 dump.rdb
  # root@WeiyiGeek-107:/nfsdisk-31/datastore/redis/demo2# 

image.png

image.png

Tips : 从上述恢复结果可以看出以aof方式恢复的数据比rdb恢复的数据完整，但所加载的时间会随着数据增大会使得AOF方式耗时比rdb耗时更多。

方案2.利用宿主机安装编译redis源码，进行恢复原k8s集群的redis数据

#!/bin/bash
# author: WeiyiGeek
# usage: ./RedisRecovery.sh [aof|rdb] redis原持久化目录
# 验证环境: Ubuntu 20.04.1 LTS
# 脚本说明: 将Redis数据恢复到物理中

if [ $# -eq 0 ];then
  echo -e "\e[31m[*] $0 [aof|rdb] redis原持久化目录 \e[0m"
  exit
fi

BACKUP_TYPE=$1
BACKUP_DIR=$2
DATA_DIR="$(pwd)/data"
REDIS_DIR="/usr/local/redis"

echo "开始时间: $(date +%s)"

if [ ! -d ${BACKUP_DIR} ];then echo -e "[Error] - Not Found ${BACKUP_DIR} Dirctory!"; return -1; fi
if [ ! -d ${DATA_DIR} ];then mkdir -vp ${DATA_DIR};else rm -rf "${DATA_DIR}/*"; fi

## 1.基础环境准备
# - 设置内存分配策略
sudo sysctl -w vm.overcommit_memory=1
# - 尽量使用物理内存(速度快)针对内核版本大于>=3.x
sudo sysctl -w vm.swapniess=1
# - SYN队列长度设置，此参数可以容纳更多等待连接的网络。
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=4096
# - 禁用 THP 特性减少内存消耗
echo never > /sys/kernel/mm/transparent_hugepage/enabled
# redis 客户端: apt remove redis-tools


## 2.REDIS 源码包
if [ ! -f /usr/local/redis/redis.conf ];then
REDIS_VERSION="redis-6.2.5"
REDIS_URL_TAR="https://download.redis.io/releases/${REDIS_VERSION}.tar.gz"
REDIS_TAR="${REDIS_VERSION}.tar.gz"

# redis 编译环境以及编译redis安装在指定目录
apt install -y gcc make gcc+ pkg-config
wget ${REDIS_URL_TAR} -O /tmp/${REDIS_TAR}
tar -zxf /tmp/${REDIS_TAR} -C /usr/local/
mv /usr/local/${REDIS_VERSION} ${REDIS_DIR}
cd ${REDIS_DIR} && make distclean
make PREFIX=${REDIS_DIR} install
cp -a ${REDIS_DIR}/redis.conf /etc/redis.conf
for i in $(ls -F ${REDIS_DIR}/bin | grep "*"| sed 's#*##g');do
  sudo chmod +700 ${REDIS_DIR}/bin/${i}
  sudo ln -s ${REDIS_DIR}/bin/${i} /usr/local/bin/${i}
done
fi

# 物理机运行可以将daemonize设置为后台运行。
tee ${REDIS_DIR}/redis.conf <<'EOF'
bind 0.0.0.0
port 6379
daemonize yes
supervised no
protected-mode no
requirepass "weiyigeek"
dir "/data"
pidfile "/var/run/redis.pid"
logfile "/var/log/redis.log"
# loglevel verbose
maxclients 10000
timeout 300
tcp-keepalive 60
maxmemory-policy volatile-lru
slowlog-max-len 128
lua-time-limit 5000
save 900 1
save 300 100
save 60 10000
dbfilename "dump.rdb"
rdbcompression yes
rdb-save-incremental-fsync yes
# appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
# rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
rename-command FLUSHDB WeiyiGeekFLUSHDB
rename-command FLUSHALL WeiyiGeekFLUSHALL
rename-command EVAL WeiyiGeekEVAL
rename-command DEBUG WeiyiGeekDEBUG
rename-command SHUTDOWN WeiyiGeekSHUTDOWN
EOF

# 3.判断要使用的备份格式
if [ "${BACKUP_TYPE}" == "aof" ];then
  sed -i "s|# appendonly yes|appendonly yes|g" ${REDIS_DIR}/redis.conf
  cp -a ${BACKUP_DIR}/appendonly.aof ${DATA_DIR}
else
  cp -a ${BACKUP_DIR}/dump.rdb ${DATA_DIR}
fi

sed -i "s#/data#${DATA_DIR}#g" ${REDIS_DIR}/redis.conf

# 4.启动redis服务恢复数据
${REDIS_DIR}/bin/redis-server ${REDIS_DIR}/redis.conf
ps -aux | grep redis-server
echo -e "AUTH weiyigeek\nping\ninfo" | redis-cli -h 127.0.0.1 | grep -A 16 "# Keyspace"
while [ $? -ne 0 ];do
echo -e "AUTH weiyigeek\nping\ninfo" | redis-cli -h 127.0.0.1 | grep -A 16 "# Keyspace"
done
echo "数据恢复完成......"
echo "完成时间: $(date +%s)" 

方案3.利用Kubernetes部署的Redis集群，进行恢复原k8s集群的redis数据

#!/bin/bash
# author: WeiyiGeek
# K8s 中 redis 集群数据恢复
# useage: K8SRedisClusterRecovery.sh [single|cluster] PODMATCH K8SVOLUMNDIR

if [ $# -eq 0 ];then
  echo -e "\e[31m[*] $0 [single|cluster] PODMATCH K8SVOLUMNDIR \e[0m"
  echo -e "\e[31m[*] PODMATCH : redis-cluster {异常集群 statefulset 资源对象名称} \e[0m"
  echo -e "\e[31m[*] K8SVOLUMNDIR : /nfsdisk-31/data  {K8S 持久化跟目录} \e[0m"
  exit
fi


RECTARGET="${1}"
PODMATCH="${2}"
K8SVOLUMNDIR="${3}"
AUTH="weiyigeek"
PWD=$(pwd)
AOFNAME="appendonly.aof"
DATADIR="${PWD}/database"
K8SSVCNAME="database.svc.cluster.local"


# 1.原`Redis`集群nodes信息一览,获取aof文件路径。
grep "myself,master" ${K8SVOLUMNDIR}/*${PODMATCH}-[0-9]-*/nodes.conf | head -n 3 > /tmp/nodes.log
cat /tmp/nodes.log
node1=$(grep "0-5460" /tmp/nodes.log | cut -d ":" -f 1)
aofpath1=${node1%/*}
node2=$(grep "5461-10922" /tmp/nodes.log | cut -d ":" -f 1)
aofpath2=${node2%/*}
node3=$(grep "10923-16383" /tmp/nodes.log | cut -d ":" -f 1)
aofpath3=${node3%/*}


# 2.验证aof文件是否存在并拷贝到当前路径下,合并AOF文件到当前目录的data下。
if [ ! -f ${aofpath1}/${AOFNAME} -o ! -f ${aofpath2}/${AOFNAME} -o ! -f ${aofpath3}/${AOFNAME} ];then echo -e "\e[31m[-] ${AOFNAME} file not found \e[0m";exit;fi
cp ${aofpath1}/${AOFNAME} ./1.${AOFNAME}
cp ${aofpath2}/${AOFNAME} ./2.${AOFNAME}
cp ${aofpath3}/${AOFNAME} ./3.${AOFNAME}
if [ ! -d ${DATADIR} ];then mkdir -v ${DATADIR};fi
cat *.aof > ${DATADIR}/${AOFNAME}
# 校验合并的原集群aof文件并尝试进行修复
echo "y" | /usr/local/redis/bin/redis-check-aof --fix ${DATADIR}/${AOFNAME}
ls -alh ${DATADIR}

# 3.验证原集群相关文件是否有误(如有误需要人工进行相应处理)
echo -e "\e[32m[*] 请验证原Redis集群Nodes信息是否无误? 请输入[Y|N] \e[0m"
read flag
if [ "${flag}" == "N" -o "${flag}" == "n" ];then echo -e "\e[31m[-] FAILED: 需要人工进行干预处理. \e[0m";exit;fi


# 4.判断恢复到单实例还是cluster集群中。
if [ "${RECTARGET}" == "single" ];then
  echo -e "\e[32m[*] 正在进行异常的K8S集群 -> 单实例数据恢复! \e[0m"
  ./K8SRedisRecovery.sh aof ${DATADIR}
  return 0
else
  echo -e "\e[32m[*] 正在进行异常的K8S集群 -> 集群数据恢复! \e[0m"
fi

# 5.redis集群资源清单(注意:此处默认是采用nfs类型的动态卷）
tee Redis-cluster-6.2.5.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster-recovery
  namespace: database
data:
  update-node.sh: |
    #!/bin/sh
    if [ ! -f /data/nodes.conf ];then touch /data/nodes.conf;fi
    REDIS_NODES="/data/nodes.conf"
    sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${REDIS_NODES}
    exec "$@"
  redis.conf: |+
    port 6379
    protected-mode no
    masterauth {RedisAuthPass}
    requirepass {RedisAuthPass}
    dir /data
    dbfilename dump.rdb
    rdbcompression yes
    no-appendfsync-on-rewrite no
    appendonly yes
    appendfilename appendonly.aof
    appendfsync everysec
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 128mb
    # 集群模式打开
    cluster-enabled yes
    cluster-config-file /data/nodes.conf
    cluster-node-timeout 5000
    slave-read-only yes
    # 当负责一个插槽的主库下线且没有相应的从库进行故障恢复时集群仍然可用
    cluster-require-full-coverage no
    # 只有当一个主节点至少拥有其他给定数量个处于正常工作中的从节点的时候，才会分配从节点给集群中孤立的主节点
    cluster-migration-barrier 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster-recovery
  namespace: database
spec:
  serviceName: redisclusterrecovery
  replicas: 6
  selector:
    matchLabels:
      app: redis-cluster-recovery
  template:
    metadata:
      labels:
        app: redis-cluster-recovery
    spec:
      containers:
      - name: redis
        image: redis:6.2.5-alpine
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        command: ["/conf/update-node.sh", "redis-server", "/conf/redis.conf"]
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        volumeMounts:
        - name: conf
          mountPath: /conf
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
        - name: timezone
          mountPath: /etc/localtime
      volumes:
      - name: conf
        configMap:
          name: redis-cluster-recovery
          defaultMode: 0755
      - name: timezone
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "managed-nfs-storage"
      resources:
        requests:
          storage: 1Gi
---
# headless Service
apiVersion: v1
kind: Service
metadata:
  name: redisclusterrecovery
  namespace: database
spec:
  clusterIP: "None"
  ports:
  - port: 6379
    targetPort: 6379
    name: client
  - port: 16379
    targetPort: 16379
    name: gossip
  selector:
    app: redis-cluster-recovery
EOF
sed -i "s#{RedisAuthPass}#${AUTH}#g" Redis-cluster-6.2.5.yaml
sed "s#replicas: 6#replicas: 0#g" Redis-cluster-6.2.5.yaml > Redis-cluster-6.2.5-empty.yaml


# 6.部署恢复数据的redis集群：判断是否存在旧的资源对象，是则清理相关文件，否则创建集群。
sts=$(kubectl get sts -n database  | grep -c "redis-cluster-recovery")
if [ ${sts} -eq 1 ];then
  # 将副本数量至为0
  kubectl apply -f Redis-cluster-6.2.5-empty.yaml;
  podrun=$(kubectl get pod -n database  | grep -c "redis-cluster-recovery")
  while [ ${run} -ne 0 ];do podrun=$(kubectl get pod -n database  | grep -c "redis-cluster-recovery");sleep 5;echo -n .; done
  find ${K8SVOLUMNDIR}/database-data-redis-cluster-recovery-* -type f -delete;
  # 清理后将副本数量至为6
  kubectl apply -f Redis-cluster-6.2.5.yaml
  run=$(kubectl get pod -n database -l app=redis-cluster-recovery | grep -c "Running")
  while [ ${run} -ne 6 ];do run=$(kubectl get pod -n database -l app=redis-cluster-recovery | grep -c "Running");sleep 2;echo -n .; done
else
  # 新建部署集群资源清单
  kubectl create --save-config -f Redis-cluster-6.2.5.yaml
  run=$(kubectl get pod -n database -l app=redis-cluster-recovery | grep -c "Running")
  while [ ${run} -ne 6 ];do run=$(kubectl get pod -n database -l app=redis-cluster-recovery | grep -c "Running");sleep 2;echo -n .; done
fi

echo -e "\e[32m[*] Happy,redis-cluster-recovery Pod is Running....\e[0m"
kubectl get pod -n database -l app=redis-cluster-recovery


# 7.redis集群快速创建配置
echo -e "yes" | redis-cli -h redisclusterrecovery.${database.svc.cluster.local} -a weiyigeek --cluster create --cluster-replicas 1 $(kubectl get pods -n database -l app=redis-cluster-recovery -o jsonpath='{range.items[*]}{.status.podIP}:6379 '| sed "s# :6379 ##g")
# 将其他两个Master卡槽归于一个Master
redis-cli -h redis-cluster-recovery-0.redisclusterrecovery.${database.svc.cluster.local} --no-auth-warning -a weiyigeek cluster nodes | grep "master" > /tmp/rnodes.log
cat /tmp/rnodes.log
m1=$(grep "0-5460" /tmp/rnodes.log | cut -d " " -f 1)
m2=$(grep "5461-10922" /tmp/rnodes.log | cut -d " " -f 1)
m3=$(grep "10923-16383" /tmp/rnodes.log | cut -d " " -f 1)
redis-cli --no-auth-warning -a weiyigeek --cluster reshard --cluster-from ${m2} --cluster-to ${m1} --cluster-slots 5462 --cluster-yes redis-cluster-recovery-0.redisclusterrecovery.${database.svc.cluster.local}:6379 > /dev/null
redis-cli --no-auth-warning -a weiyigeek --cluster reshard --cluster-from ${m3} --cluster-to ${m1} --cluster-slots 5461 --cluster-yes redis-cluster-recovery-0.redisclusterrecovery.${database.svc.cluster.local}:6379 > /dev/null
redis-cli -h redis-cluster-recovery-0.redisclusterrecovery.database.svc.cluster.local --no-auth-warning -a weiyigeek cluster nodes
echo -e "\e[32m[*] 请验证Nodes信息是否无误? 请输入[Y|N] \e[0m"
read flag
if [ "${flag}" == "N" -o "${flag}" == "n" ];then echo -e "\e[31m[-] FAILED: 需要人工进行干预处理. \e[0m";exit;fi


# 8.将所有slots都归于一个master节点后我们需要,将redisclusterrecovery 资源对象所管理的Pod先关闭。
# 将副本数量至为0
kubectl apply -f Redis-cluster-6.2.5-empty.yaml;
podrun=$(kubectl get pod -n database  | grep -c "redis-cluster-recovery")
while [ ${run} -ne 0 ];do podrun=$(kubectl get pod -n database  | grep -c "redis-cluster-recovery");sleep 5;echo -n .; done
# find ${K8SVOLUMNDIR}/database-data-redis-cluster-recovery-* -type f -delete;

# 此处只清空了redis持久化数据文件。
master=$(find /nfsdisk-31/data/*redis-cluster-recovery-0* -type d)
rm -rf ${master}/*.aof ${master}*/.rdb
cp ${DATADIR}/appendonly.aof {${master}}

# 清理后将副本数量至为6
kubectl apply -f Redis-cluster-6.2.5.yaml
run=$(kubectl get pod -n database -l app=redis-cluster-recovery | grep -c "Running")
while [ ${run} -ne 6 ];do run=$(kubectl get pod -n database -l app=redis-cluster-recovery | grep -c "Running");sleep 2;echo -n .; done

echo -e "\e[32m[*] Happy,redis-cluster-recovery Pod is Running....\e[0m"
kubectl get pod -n database -l app=redis-cluster-recovery


# 9.处理K8s重启redis集群出现的fail问题，我们可以将错误节点剔出集群并重新指定节点信息加入到集群之中
redis-cli -h redis-cluster-recovery-0.redisclusterrecovery.database.svc.cluster.local --no-auth-warning -a weiyigeek cluster nodes | grep "fail" > /tmp/errnodes.log
# 将该从节点剔出集群
for err in $(cat /tmp/errnodes.log | cut -d " " -f 1);do
  redis-cli -h redis-cluster-recovery-0.redisclusterrecovery.database.svc.cluster.local -a weiyigeek cluster forget ${errid}
done
# 重新将该节点加入集群
for ip in $(kubectl get pods -n database -l app=redis-cluster-recovery -o jsonpath='{range.items[*]}{.status.podIP} '| sed "s# :6379 ##g");do
  redis-cli -h  -a weiyigeek cluster meet ${ip} 6379
done

# 10.集群状态检测以及重新分配slots卡槽到Master节点
redis-cli -h redisclusterrecovery.database.svc.cluster.local -a weiyigeek --cluster check redis-cluster-recovery-0.redisclusterrecovery.database.svc.cluster.local:6379
redis-cli -h redis-cluster-recovery-0.redisclusterrecovery.database.svc.cluster.local -a weiyigeek --cluster rebalance --cluster-threshold 1 --cluster-use-empty-masters redis-cluster-recovery-1.redisclusterrecovery.database.svc.cluster.local:6379

# 11.验证分配的slots卡槽到各个Master节点节点信息。
redis-cli -h redisclusterrecovery.database.svc.cluster.local -a weiyigeek --cluster check redis-cluster-recovery-0.redisclusterrecovery.database.svc.cluster.local:6379


# 12.主master节点验证keyspace数据
redis-cli -h redis-cluster-recovery-0.redisclusterrecovery.database.svc.cluster.local -c -a weiyigeek info keyspace

# 13.任何节点访问恢复的数据
redis-cli -h redisclusterrecovery.database.svc.cluster.local -c -a weiyigeek 

3.当Redis集群中出现从节点slave,fail,noaddr问题进行处理恢复流程：

• Step 1.利用cluster nodes查看你集群状态，发现其中一个从节点异常(是Fail状态)。

f86464011d9f8ec605857255c0b67cff1e794c19 :0@0 slave,fail,noaddr 2cb35944b4492748a8c739fab63a0e90a56e414a 

• Step 2.在问题节点上查看节点状态，发现它已脱离集群，且其ID都已发生了变化.

127.0.0.1:6379> cluster nodes
0cbf44ef3f9c3a8a473bcd303644388782e5ee78 192.168.109.132:6379@16379 myself,master - 0 0 0 connected 0-5461 

Tips : 若id没发生变化，直接重启下该从节点就能解决。

• Step 3.但如果ID与Node IP都发生变化时，此时我们需要将该从节点剔出集群

# 在集群每个正常节点上执行cluster forget 故障从节点id
echo 'cluster forget f86464011d9f8ec605857255c0b67cff1e794c19' | /usr/local/bin/redis-cli -a "weiyigeek" 

• Step 4.我们重新将该节点加入集群，此时我们只需要在集群内任意节点上执行命令加入新节点，握手状态会通过信息在集群内传播，这样其他节点会自动发现新节点并发起握手流程。

# 1.使用集群强制联系指定节点(握手)
echo 'cluster meet 192.168.109.132 6379' | /usr/local/bin/redis-cli -p 6379 -a "密码"

# 2.从节上执行cluster replicate 主节点id( 配置主从关系 )
echo 'cluster replicate 2cb35944b4492748a8c739fab63a0e90a56e414a' | /usr/local/bin/redis-cli -p 6383 -a "密码" 

• Step 5.最后检测集群是否恢复正常,执行如下命令即可。

echo 'cluster nodes' | /usr/local/bin/redis-cli -p 6384 -a "密码"
38287a7e715c358b5537a369646e9698a7583459 192.168.109.132:6383@16383 slave 2cb35944b4492748a8c739fab63a0e90a56e414a 0 1615233239757 8 connected
2cb35944b4492748a8c739fab63a0e90a56e414a 192.168.109.133:6383@16383 master - 0 1615233239000 8 connected 0-5461
....... 

Redis数据的导出和导入：dump和load方式
https://www.jianshu.com/p/03da3b9774d8

转载自https://www.bilibili.com/read/cv13235983