k8s集群安装一etcd

本文参考了网易云课堂阿良老师的系列视频《1天入门Kubernetes/K8S》，学习之后所做的笔记。本文不使用kubeadm等自动化工具，而是从官网下载二进制包手动部署。

集群环境规划

阿里云非大陆节点，避免各种墙的问题。3台机器，2CPU 2G最低配。系统镜像：ubuntu_16_04_64_XXXX

角色 IP 组件
master 172.31.173.35 kube-apiserver
kube-controller-manager
kube-scheduler
etcd
node1 172.31.173.36 kubelet
kube-proxy
docker
flannel
etcd
node2 172.31.173.37 kubelet
kube-proxy
docker
flannel
etcd

安装 docker

$ apt-get update 
$ apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
$ apt-get update
$ apt-get install docker-ce 
$ systemctl start docker
$ systemctl enable docker
# 测试docker
$ docker --version

TLS 证书

请先了解CA，公钥私钥概念，加解密过程。
证书有机构签发证书，收费，被浏览器信任。
自签证书不受信任，但是功能都是一样的。

组件 证书
etcd ca.pem，server.pem，server-key.pem
kube-apiserver ca.pem，server.pem，server-key.pem
kubelet ca.pem，ca-key.pem
kube-proxy ca.pem，kube-proxy.pem，kube-proxy-key.pem
kubectl ca.pem，admin.pem，admin-key.pem

安装cfssl

当然openssl也可以生成数字证书，这里用cfssl。

# 参考
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/
https://pkg.cfssl.org/

# 下载
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

# 放到/usr/local/bin目录下，方便使用
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

# 验证
cfssl --help

# 生成证书模板，然后，在模板文件上修改，在模板上修改，在模板上修改，说三遍
cfssl print-defaults config > config.json
cfssl print-defaults csr > csr.json

生成CA证书，需要ca-config.json和ca-csr.json两个配置文件

# ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    }
  },
  "profiles": {
    "kubernetes": {
      "expiry": "87600h",
      "usages": [
        "signing",
        "key encipherment",
        "server auth",
        "client auth"
      ]
    }
  }
}
# ca-csr.json
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

# 生成CA证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
# 执行命令，会生成ca.pem和ca-key.pem两个文件

生成server证书

# server-csr.json
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "172.31.173.35",
    "172.31.173.36",
    "172.31.173.37",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

# 生成server证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

# 生成server-key和server.pem

生成admin证书

admin-csr.json

{
  "CN": "admin",
  "hosts": [
    
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

生成admin.pem和admin-key.pem

生成kube-proxy证书

kube-proxy-csr.json

{
  "CN": "System:kube-proxy",
  "hosts": [
    
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

保留*.pem证书，其他文件删除
ls | grep -v pem | xargs -i rm {}

/root/ssl目录下有以下文件：

admin-key.pem
admin.pem
ca-key.pem
ca.pem
kube-proxy-key.pem
kube-proxy.pem
server-key.pem
server.pem

部署etcd

首先我们确定下安装配置文件的目录

# bin目录下放脚本
/opt/kubernetes/bin
# cfg目录下放配置文件
/opt/kubernetes/cfg
# ssl目录下放证书
/opt/kubernetes/ssl

下载地址：
https://github.com/etcd-io/etcd/releases/download/v3.3.12/etcd-v3.3.12-linux-amd64.tar.gz
将etct-XX解压后的文件etcd/etcdctl两个脚本放到/opt/kubernetes/bin目录下

etcd配置文件

放到/opt/kubernetes/cfg目录下

#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.31.173.35:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.31.173.35:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.31.173.35:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.31.173.35:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://172.31.173.35:2380,etcd02=https://172.31.173.36:2380,etcd03=https://172.31.173.37:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

用systemd管理etcd进程，vi /usr/lib/systemd/system/etcd.service，文件写入以下内容

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/kubernetes/cfg/etcd
ExecStart=/opt/kubernetes/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/opt/kubernetes/ssl/server.pem \
--key-file=/opt/kubernetes/ssl/server-key.pem \
--peer-cert-file=/opt/kubernetes/ssl/server.pem \
--peer-key-file=/opt/kubernetes/ssl/server-key.pem \
--trusted-ca-file=/opt/kubernetes/ssl/ca.pem \
--peer-trusted-ca-file=/opt/kubernetes/ssl/ca.pem \


[Install]
WantedBy=multi-user.target

启动etcd

# 启动etcd
systemctl start etcd
systemctl enable etcd

#可以看到etcd进程已经启动，并在尝试连接其他节点。
#如果启动失败，可用以下命令调试
ps -ef | grep etcd
systemctl status etcd.service
journalctl -u etcd
journalctl -xe
etcdctl cluster-health

systemctl daemon-reload
systemctl start etcd
systemctl enable etcd
systemctl restart etcd

# 如果启动失败，仔细检查参数配置。

设置ssh免密码登录

ssh-copy-id root@172.31.173.36
ssh-copy-id root@172.31.173.37
输入密码回车。。。

从master节点拷贝文件到node节点

ssh root@172.31.173.36
ssh root@172.31.173.37
#ssh登录到node节点创建目录
mkdir -p /opt/kubernetes/{bin,cfg,ssl}

# 拷贝文件到node节点
scp -r /opt/kubernetes/{bin,cfg,ssl} root@172.31.173.36:/opt/kubernetes
scp -r /opt/kubernetes/{bin,cfg,ssl} root@172.31.173.37:/opt/kubernetes

scp /usr/lib/systemd/system/etcd.service root@172.31.173.36:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/etcd.service root@172.31.173.37:/usr/lib/systemd/system/

# 修改node节点下的etcd.conf文件
ETCD_NAME=xxx02 在node2节点
ETCD_NAME=xxx03 在node3节点
对应的ip也修改下

测试etcd集群状态

/opt/kubernetes/bin/etcdctl \
--ca-file=/opt/kubernetes/ssl/ca.pem \
--cert-file=/opt/kubernetes/ssl/server.pem \
--key-file=/opt/kubernetes/ssl/server-key.pem \
--endpoints="https://172.31.173.35:2379,https://172.31.173.36:2379,https://172.31.173.37:2379" \
cluster-health

见证奇迹的时刻截图：


image.png

etcd snapshot


# 快照
export ETCDCTL_API=3
etcdctl  --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=127.0.0.1:2379 snapshot save /tmp/snapshot.db

# 从快照恢复
etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=127.0.0.1:2379 snapshot restore /tmp/snapshot.db --data-dir=/var/lib/etcd/

#  启动新etcd节点，指定--data-dir=/var/lib/etcd/

待续。。。

最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
【社区内容提示】社区部分内容疑似由AI辅助生成，浏览时请结合常识与多方信息审慎甄别。
平台声明：文章内容（如有图片或视频亦包括在内）由作者上传并发布，文章内容仅代表作者本人观点，简书系信息发布平台，仅提供信息存储服务。

相关阅读更多精彩内容

友情链接更多精彩内容