docker swarm部署minio集群

安装系统

略

更换软件源

首先备份原来的软件源并另存：

sudo cp -v /etc/apt/sources.list /etc/apt/sources.list.backup

编辑软件源文件

sudo vim /etc/apt/sources.list

注释所有软件源

阿里云源：

deb http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse

清华源

deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ eoan main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ eoan main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ eoan-updates main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ eoan-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ eoan-backports main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ eoan-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ eoan-security main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ eoan-security main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ eoan-proposed main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ eoan-proposed main restricted universe multiverse

修改完软件源后，更新软件列表和软件：

sudo apt update
sudo apt upgrade

安装docker

sudo apt install docker.io

设置docker镜像加速

编辑daemon.json文件：

sudo vim /etc/docker/daemon.json

添加阿里云加速镜像地址内容：

{
"registry-mirrors": ["https://******.mirror.aliyuncs.com"]
}

重启服务

sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl status docker

创建Swarm

在管理节点上创建一个swarm,请运行下面的命令

docker swarm init --advertise-addr <MANAGER-IP>

一旦swarm初使化了，你可以看到下面的响应信息

docker swarm join \
  --token  SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
  192.168.99.100:2377

你现在可以运行上述命令添加worker节点到swarm。

添加`label`标签

# 查看节点信息
sudo docker node ls
# 为节点添加标签
sudo docker node update --label-add minio1=true (node_name)

minio1=true标签是minio在stack文件中定义的，用来确定节点部署的，所以必须添加，不然节点将不会部署。

为MinIO创建Docker secret

sudo echo "AKIAIOSFODNN7EXAMPLE" | sudo docker secret create access_key -
sudo echo "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" | sudo docker secret create secret_key -

access_key和secret_key可以自定义，尽量不要相同。

部署分布式minio服务

在你的Swarm master上下载Docker Compose file ，然后运行下面的命令

sudo docker stack deploy --compose-file=docker-compose-secrets.yaml minio_stack

查看节点部署信息

sudo docker service ls

服务列表

确认每个节点都起来了。

查看节点日志

sudo docker service logs (节点ID)

确认每个节点都运行正常

配置nginx负载均衡

创建nginx配置文件minio.conf

内容如下

upstream  minio-server {
       server    10.168.1.136:9003;
       server    10.168.1.136:9004;
       server    10.168.1.131:9001;
       server    10.168.1.131:9002;
}


server {
    listen              80;
    server_name 10.168.1.136;
    location / {
        proxy_buffering off;
        proxy_set_header Host $http_host;
        proxy_pass http://minio-server;
        proxy_redirect off;
    }
}

创建nginx相关文件夹

sudo mkdir -p /home/nginx/logs /home/nginx/conf

部署nginx

sudo docker run -d -p 80:80 --name nginx \
  -v /home/nginx/conf/:/etc/nginx/conf.d/ \
  -v /home/nginx/logs:/var/log/nginx \
  nginx

在浏览器中输入服务器ip进行访问

登录界面

如果现在为初始化，请重试，请检查每个节点是否正常运行，以及每个节点是否能连通。
检查每个节点的磁盘空间是否大于1G。

附：docker-compose-secrets.yaml文件内容

version: '3.7'

services:
  minio1:
    image: minio/minio:RELEASE.2020-07-02T00-15-09Z
    hostname: minio1
    volumes:
      - minio1-data:/export
    ports:
      - "9001:9000"
    networks:
      - minio_distributed
    deploy:
      restart_policy:
        delay: 10s
        max_attempts: 10
        window: 60s
      placement:
        constraints:
          - node.labels.minio1==true #和节点的label对应
    command: server http://minio{1...4}/export
    secrets:
      - secret_key
      - access_key
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio2:
    image: minio/minio:RELEASE.2020-07-02T00-15-09Z
    hostname: minio2
    volumes:
      - minio2-data:/export
    ports:
      - "9002:9000"
    networks:
      - minio_distributed
    deploy:
      restart_policy:
        delay: 10s
        max_attempts: 10
        window: 60s
      placement:
        constraints:
          - node.labels.minio2==true #和节点的label对应
    command: server http://minio{1...4}/export
    secrets:
      - secret_key
      - access_key
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio3:
    image: minio/minio:RELEASE.2020-07-02T00-15-09Z
    hostname: minio3
    volumes:
      - minio3-data:/export
    ports:
      - "9003:9000"
    networks:
      - minio_distributed
    deploy:
      restart_policy:
        delay: 10s
        max_attempts: 10
        window: 60s
      placement:
        constraints:
          - node.labels.minio3==true #和节点的label对应
    command: server http://minio{1...4}/export
    secrets:
      - secret_key
      - access_key
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio4:
    image: minio/minio:RELEASE.2020-07-02T00-15-09Z
    hostname: minio4
    volumes:
      - minio4-data:/export
    ports:
      - "9004:9000"
    networks:
      - minio_distributed
    deploy:
      restart_policy:
        delay: 10s
        max_attempts: 10
        window: 60s
      placement:
        constraints:
          - node.labels.minio4==true #和节点的label对应
    command: server http://minio{1...4}/export
    secrets:
      - secret_key
      - access_key
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

volumes:
  minio1-data:
  minio2-data:
  minio3-data:
  minio4-data:

networks:
  minio_distributed:
    driver: overlay

secrets:
  secret_key:
    external: true
  access_key:
    external: true