部署方式：
方式1. minikube
Minikube是一个工具，可以在本地快速运行一个单点的Kubernetes，尝试Kubernetes或日常开发的用户使用。不能用于生产环境。
官方地址：https://kubernetes.io/docs/setup/minikube/

方式2. kubeadm
Kubeadm也是一个工具，提供kubeadm init和kubeadm join，用于快速部署Kubernetes集群。
官方地址：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

方式3. 直接使用epel-release yum源，缺点就是版本较低

方式4. 二进制包
从官方下载发行版的二进制包，手动部署每个组件，组成Kubernetes集群。

官方也提供了一个互动测试环境供大家测试：https://kubernetes.io/zh/docs/tutorials/kubernetes-basics/cluster-interactive/
https://kubernetes.io/docs/tutorials/kubernetes-basics/

二进制方式部署 kubernetes 集群

目标任务

1 Kubernetes集群部署架构规划
2 部署Etcd集群
3 在Node节点安装Docker
4 部署Flannel网络
5 在Master节点部署组件
6 在Node节点部署组件
7 查看集群状态
8 运行一个测试示例
9 部署Dashboard（Web UI）

1. 准备

三台虚拟机
固定IP、改名字、相互解析、关闭防火墙
# systemctl stop firewalld;setenforce 0
# vim /etc/hosts
192.168.1.18 k8s-master
192.168.1.22 k8s-node1
192.168.1.25 k8s-node2

软件版本：
    Docker  18.09.0-ce
    Kubernetes  1.11

服务器角色、IP、组件：
    k8s-master       
        192.168.1.18    kube-apiserver，kube-controller-manager，kube-scheduler，etcd

    k8s-node1           
        192.168.1.22    kubelet，kube-proxy，docker，flannel，etcd

    k8s-node2           
        192.168.1.25    kubelet，kube-proxy，docker，flannel，etcd
因我电脑配置较低 etcd 配置在了三台服务器上，实际生产环境中应该部署在单独的三台服务器

2. 部署Etcd集群

使用cfssl来生成自签证书，任何机器都行，证书这块儿知道怎么生成、怎么用即可，暂且不用过多研究。

#下载cfssl工具
[root@k8s-master k8s]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
[root@k8s-master k8s]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
[root@k8s-master k8s]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
[root@k8s-master k8s]# ls
cfssl-certinfo_linux-amd64  cfssljson_linux-amd64  cfssl_linux-amd64
[root@k8s-master k8s]# chmod +x cfssl*
[root@k8s-master k8s]# mv cfssl_linux-amd64 /usr/local/bin/cfssl
[root@k8s-master k8s]# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
[root@k8s-master k8s]# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

#生成Etcd证书
[root@k8s-master k8s]# vim ca-config.json
[root@k8s-master k8s]# cat ca-config.json 
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}

[root@k8s-master k8s]# vim ca-csr.json
[root@k8s-master k8s]# cat ca-csr.json
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}

[root@k8s-master k8s]# vim server-csr.json
[root@k8s-master k8s]# cat server-csr.json     #IP填写etcd集群的IP
{
    "CN": "etcd",
    "hosts": [
    "192.168.1.18",
    "192.168.1.22",
    "192.168.1.25"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}

#生成证书
[root@k8s-master k8s]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
[root@k8s-master k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
[root@k8s-master k8s]# ls *pem
ca-key.pem  ca.pem  server-key.pem  server.pem

#安装Etcd，三台都配置
# wget https://github.com/etcd-io/etcd/releases/download/v3.2.12/etcd-v3.2.12-linux-amd64.tar.gz
# mkdir /opt/etcd/{bin,cfg,ssl} -p
# tar zxvf etcd-v3.2.12-linux-amd64.tar.gz
# mv etcd-v3.2.12-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

#创建etcd配置文件，三台都配置：IP要写当前的
[root@k8s-master ~]# vim /opt/etcd/cfg/etcd
[root@k8s-master ~]# cat /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd01"     #节点名称   修改
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"     #数据目录
ETCD_LISTEN_PEER_URLS="https://192.168.1.18:2380"     #集群通信监听地址   修改
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.18:2379"     #客户端访问监听地址   修改

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.18:2380"     #集群通告地址   修改
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.18:2379"     #客户端通告地址   修改
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.1.18:2380,etcd02=https://192.168.1.22:2380,etcd03=https://192.168.1.25:2380"     #集群节点地址   修改名字和IP对应
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"     #集群Token
ETCD_INITIAL_CLUSTER_STATE="new"     #加入集群的当前状态，new是新集群，existing表示加入已有集群

#systemd管理etcd，三台都配置
[root@k8s-master ~]# vim /usr/lib/systemd/system/etcd.service
[root@k8s-master ~]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd
ExecStart=/opt/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

#把刚才生成的证书拷贝到配置文件中的位置
[root@k8s-master ~]# cp /k8s/*pem /opt/etcd/ssl

#启动并设置开启启动，三台都配置
# systemctl start etcd
# systemctl enable etcd

#检查etcd集群状态
[root@k8s-master ~]# /opt/etcd/bin/etcdctl \
--ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem \
--endpoints="https://192.168.1.18:2379,https://192.168.1.22:2379,https://192.168.1.25:2379" \
cluster-health

cluster is healthy 输出此行，集群部署成功

3. node节点安装docker

# yum install -y yum-utils device-mapper-persistent-data lvm2

# yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

# yum makecache fast
# yum -y install docker-ce

# systemctl start docker
# systemctl enable docker

# docker -v
Docker version 19.03.8, build afacb8b

4. 部署Flannel网络

工作原理

image.png

#Falnnel要用etcd存储自身一个子网信息，所以要保证能成功连接Etcd，写入预定义子网段，注意修改IP
[root@k8s-master ~]# cd /opt/etcd/ssl/
[root@k8s-master ssl]# ls
ca-key.pem  ca.pem  server-key.pem  server.pem

[root@k8s-master ssl]# /opt/etcd/bin/etcdctl \
> --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem \
> --endpoints="https://192.168.1.18:2379,https://192.168.1.22:2379,https://192.168.1.25:2379" \
> set /coreos.com/network/config  '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'

#安装Flannel，node节点操作
# wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
# tar xzvf flannel-v0.10.0-linux-amd64.tar.gz
# mkdir -p /opt/kubernetes/bin
# mv flanneld mk-docker-opts.sh /opt/kubernetes/bin

#配置Flannel，node节点操作
# mkdir -p /opt/kubernetes/cfg/
# vim /opt/kubernetes/cfg/flanneld
# vim /opt/kubernetes/cfg/flanneld
# cat /opt/kubernetes/cfg/flanneld     #注意修改IP
FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.1.18:2379,https://192.168.1.22:2379,https://192.168.1.25:2379 -etcd-cafile=/opt/etcd/ssl/ca.pem -etcd-certfile=/opt/etcd/ssl/server.pem -etcd-keyfile=/opt/etcd/ssl/server-key.pem"

#systemd管理Flannel，node节点操作
# vim /usr/lib/systemd/system/flanneld.service
# cat /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target

#配置Docker启动指定子网段，node节点操作
# vim /usr/lib/systemd/system/docker.service
# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

#将证书拷贝到node节点
[root@k8s-master ~]# scp /k8s/*pem k8s-node1:/opt/etcd/ssl
[root@k8s-master ~]# scp /k8s/*pem k8s-node2:/opt/etcd/ssl

#重启flannel和docker，node节点操作
# systemctl daemon-reload
# systemctl start flanneld
# systemctl enable flanneld
# systemctl restart docker

#检查是否生效，node节点操作
[root@k8s-node1 ~]# ps -ef |grep docker
root      12103      1  2 18:07 ?        00:00:00 /usr/bin/dockerd --bip=172.17.90.1/24 --ip-masq=false --mtu=1450
root      12240  10550  0 18:07 pts/0    00:00:00 grep --color=auto docker

[root@k8s-node2 ~]# ps -ef |grep docker
root      12112      1  1 18:07 ?        00:00:00 /usr/bin/dockerd --bip=172.17.96.1/24 --ip-masq=false --mtu=1450
root      12267   1314  0 18:08 pts/0    00:00:00 grep --color=auto docker

#确保node节点的docker0与flannel.1在同一网段，node节点网段互相不一样
[root@k8s-node1 ~]# ifconfig docker0 | grep inet | head -n 1 | awk '{print $2}' 
172.17.90.1
[root@k8s-node1 ~]# ifconfig flannel.1 | grep inet | head -n 1 | awk '{print $2}' 
172.17.90.0

[root@k8s-node2 ~]# ifconfig docker0 | grep inet | head -n 1 | awk '{print $2}' 
172.17.96.1
[root@k8s-node2 ~]# ifconfig flannel.1 | grep inet | head -n 1 | awk '{print $2}' 
172.17.96.0

#node节点互相ping测试
[root@k8s-node1 ~]# ping 172.17.96.1
[root@k8s-node2 ~]# ping 172.17.90.1

#ping通Flannel部署成功

5. 在Master节点部署组件

在部署Kubernetes之前一定要确保etcd、flannel、docker是正常工作的。

5.1 生成证书

[root@k8s-master ~]# cd /k8s/
[root@k8s-master k8s]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  server.csr  server-csr.json  server-key.pem  server.pem
[root@k8s-master k8s]# mkdir etcdsslbk
[root@k8s-master k8s]# mv * etcdsslbk/
mv: 无法将目录"etcdsslbk" 移动至自身的子目录"etcdsslbk/etcdsslbk" 下
[root@k8s-master k8s]# ls
etcdsslbk
[root@k8s-master k8s]# ls etcdsslbk/
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  server.csr  server-csr.json  server-key.pem  server.pem

[root@k8s-master k8s]# vim ca-config.json
[root@k8s-master k8s]# cat ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}

[root@k8s-master k8s]# vim ca-csr.json
[root@k8s-master k8s]# cat ca-csr.json
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

[root@k8s-master k8s]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#生成apiserver证书
[root@k8s-master k8s]# vim server-csr.json
[root@k8s-master k8s]# cat server-csr.json
{
    "CN": "kubernetes",
    "hosts": [
      "192.168.1.1",     #这个是后边dns要用的虚拟网络的网关
      "127.0.0.1",
      "192.168.1.18",     #k8s集群IP
      "192.168.1.22",     #k8s集群IP
      "192.168.1.25",     #k8s集群IP
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

[root@k8s-master k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

#生成kube-proxy证书
[root@k8s-master k8s]# vim kube-proxy-csr.json
[root@k8s-master k8s]# cat kube-proxy-csr.json
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

[root@k8s-master k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

[root@k8s-master k8s]# ls *.pem
ca-key.pem  ca.pem  kube-proxy-key.pem  kube-proxy.pem  server-key.pem  server.pem

5.2 部署apiserver组件

浏览器访问此链接下载包，传到服务器（https://dl.k8s.io/v1.11.0/kubernetes-server-linux-amd64.tar.gz）

[root@k8s-master ~]# mkdir /opt/kubernetes/{bin,cfg,ssl} -p
[root@k8s-master ~]# tar zxvf kubernetes-server-linux-amd64.tar.gz
[root@k8s-master ~]# cd kubernetes/server/bin
[root@k8s-master bin]# cp kube-apiserver kube-scheduler kube-controller-manager kubectl /opt/kubernetes/bin

#将生成的证书拷贝到master对应目录
[root@k8s-master ~]# cd /k8s/
[root@k8s-master k8s]# cp server.pem  server-key.pem ca.pem ca-key.pem /opt/kubernetes/ssl/

#创建token文件
[root@k8s-master k8s]# vim /opt/kubernetes/cfg/token.csv
[root@k8s-master k8s]# cat /opt/kubernetes/cfg/token.csv
674c457d4dcf2eefe4920d7dbb6b0ddc,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
#第一列：随机字符串，自己可生成，第二列：用户名，第三列：UID，第四列：用户组。

#创建apiserver配置文件
[root@k8s-master k8s]# vim /opt/kubernetes/cfg/kube-apiserver
[root@k8s-master k8s]# cat /opt/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.1.18:2379,https://192.168.1.22:2379,https://192.168.1.25:2379 \
--bind-address=192.168.1.18 \
--secure-port=6443 \
--advertise-address=192.168.1.18 \
--allow-privileged=true \
--service-cluster-ip-range=192.168.1.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"

#参数说明：
--logtostderr 启用日志
--v 日志等级
--etcd-servers etcd集群地址
--bind-address 监听地址
--secure-port https安全端口
--advertise-address 集群通告地址
--allow-privileged 启用授权
--service-cluster-ip-range Service虚拟IP地址段
--enable-admission-plugins 准入控制模块
--authorization-mode 认证授权，启用RBAC授权和节点自管理
--enable-bootstrap-token-auth 启用TLS bootstrap功能，后面会讲到
--token-auth-file token文件
--service-node-port-range Service Node类型默认分配端口范围

#systemd管理apiserver
[root@k8s-master k8s]# vim /usr/lib/systemd/system/kube-apiserver.service
[root@k8s-master k8s]# cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

#启动apiserver
[root@k8s-master k8s]# systemctl daemon-reload
[root@k8s-master k8s]# systemctl start kube-apiserver
[root@k8s-master k8s]# systemctl enable kube-apiserver

5.3 部署schduler组件

[root@k8s-master k8s]# vim /opt/kubernetes/cfg/kube-scheduler
[root@k8s-master k8s]# cat /opt/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect"

#参数说明：
--master 连接本地apiserver
--leader-elect 当该组件启动多个时，自动选举（HA）

#systemd管理schduler组件
[root@k8s-master k8s]# vim /usr/lib/systemd/system/kube-scheduler.service
[root@k8s-master k8s]# cat /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

#启动
[root@k8s-master k8s]# systemctl daemon-reload
[root@k8s-master k8s]# systemctl start kube-scheduler
[root@k8s-master k8s]# systemctl enable kube-scheduler

5.4 部署controller-manager组件

[root@k8s-master k8s]# vim /opt/kubernetes/cfg/kube-controller-manager
[root@k8s-master k8s]# cat /opt/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=192.168.1.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem"

#systemd管理controller-manager组件
[root@k8s-master k8s]# vim /usr/lib/systemd/system/kube-controller-manager.service
[root@k8s-master k8s]# cat /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

#启动
[root@k8s-master k8s]# systemctl daemon-reload
[root@k8s-master k8s]# systemctl start kube-controller-manager
[root@k8s-master k8s]# systemctl enable kube-controller-manager

5.5 通过kubectl工具查看当前集群组件状态

[root@k8s-master k8s]# /opt/kubernetes/bin/kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-2               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}   
etcd-0               Healthy   {"health": "true"}   

如上输出说明组件都正常

6. 在Node节点部署组件

Master apiserver启用TLS认证后，Node节点kubelet组件想要加入集群，必须使用CA签发的有效证书才能与apiserver通信，当Node节点很多时，签署证书是一件很繁琐的事情，因此有了TLS Bootstrapping机制，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。

6.1 master节点操作

#将kubelet-bootstrap用户绑定到系统集群角色
[root@k8s-master k8s]# /opt/kubernetes/bin/kubectl create clusterrolebinding kubelet-bootstrap \
   --clusterrole=system:node-bootstrapper \
   --user=kubelet-bootstrap

#创建kubeconfig文件，在生成kubernetes证书的目录下执行以下命令生成kubeconfig文件：
#指定apiserver 内网负载均衡地址
[root@k8s-master k8s]# KUBE_APISERVER="https://192.168.1.18:6443"
[root@k8s-master k8s]# BOOTSTRAP_TOKEN=674c457d4dcf2eefe4920d7dbb6b0ddc

#设置集群参数
[root@k8s-master k8s]# /opt/kubernetes/bin/kubectl config set-cluster kubernetes \
  --certificate-authority=./ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

#设置客户端认证参数
[root@k8s-master k8s]# /opt/kubernetes/bin/kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig

#设置上下文参数
[root@k8s-master k8s]# /opt/kubernetes/bin/kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

#设置默认上下文
[root@k8s-master k8s]# /opt/kubernetes/bin/kubectl config use-context default --kubeconfig=bootstrap.kubeconfig


#创建kube-proxy kubeconfig文件
[root@k8s-master k8s]# /opt/kubernetes/bin/kubectl config set-cluster kubernetes \
  --certificate-authority=./ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

[root@k8s-master k8s]# /opt/kubernetes/bin/kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

[root@k8s-master k8s]# /opt/kubernetes/bin/kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

[root@k8s-master k8s]# /opt/kubernetes/bin/kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig


#将这两个文件拷贝到Node节点/opt/kubernetes/cfg/目录下。
[root@k8s-master k8s]# scp bootstrap.kubeconfig kube-proxy.kubeconfig k8s-node1:/opt/kubernetes/cfg/
[root@k8s-master k8s]# scp bootstrap.kubeconfig kube-proxy.kubeconfig k8s-node2:/opt/kubernetes/cfg/


#部署kubelet组件，将前面下载的二进制包中的kubelet和kube-proxy拷贝到Node节点/opt/kubernetes/bin目录下。
[root@k8s-master k8s]# cd /root/kubernetes/server/bin/
[root@k8s-master bin]# scp kubelet kube-proxy k8s-node1:/opt/kubernetes/bin/
[root@k8s-master bin]# scp kubelet kube-proxy k8s-node2:/opt/kubernetes/bin/

6.2 部署kubelet组件

所有node节点操作

#创建kubelet配置文件
[root@k8s-node1 ~]# vim /opt/kubernetes/cfg/kubelet
[root@k8s-node1 ~]# cat /opt/kubernetes/cfg/kubelet
KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.1.22 \     #修改为本机为IP
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet.config \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

#参数说明：
--hostname-override 在集群中显示的主机名
--kubeconfig 指定kubeconfig文件位置，会自动生成
--bootstrap-kubeconfig 指定刚才生成的bootstrap.kubeconfig文件
--cert-dir 颁发证书存放位置
--pod-infra-container-image 管理Pod网络的镜像

#其中/opt/kubernetes/cfg/kubelet.config配置文件如下
[root@k8s-node1 ~]# vim /opt/kubernetes/cfg/kubelet.config
[root@k8s-node1 ~]# cat /opt/kubernetes/cfg/kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 192.168.1.22     #修改为本机为IP
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["192.168.1.2"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
    enabled: true 
  webhook:
    enabled: false

#systemd管理kubelet组件
[root@k8s-node1 ~]# vim /usr/lib/systemd/system/kubelet.service
[root@k8s-node1 ~]# cat /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target

#启动
[root@k8s-node1 ~]# systemctl daemon-reload
[root@k8s-node1 ~]# systemctl start kubelet
[root@k8s-node1 ~]# systemctl enable kubelet

#在Master审批Node加入集群
[root@k8s-master bin]# /opt/kubernetes/bin/kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-Oecfo2IVLnnCGoYqhQ3f6q7FbN0C2Dr7c0o8j-6v2rI   2m        kubelet-bootstrap   Pending
node-csr-tRpwDHVhxW-9DxLkW_hu2XErEs1LBveGaah9_cREJrE   2m        kubelet-bootstrap   Pending

[root@k8s-master bin]# /opt/kubernetes/bin/kubectl certificate approve node-csr-Oecfo2IVLnnCGoYqhQ3f6q7FbN0C2Dr7c0o8j-6v2rI

[root@k8s-master bin]# /opt/kubernetes/bin/kubectl certificate approve node-csr-tRpwDHVhxW-9DxLkW_hu2XErEs1LBveGaah9_cREJrE

[root@k8s-master bin]# /opt/kubernetes/bin/kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-Oecfo2IVLnnCGoYqhQ3f6q7FbN0C2Dr7c0o8j-6v2rI   3m        kubelet-bootstrap   Approved,Issued
node-csr-tRpwDHVhxW-9DxLkW_hu2XErEs1LBveGaah9_cREJrE   2m        kubelet-bootstrap   Approved,Issued

[root@k8s-master bin]# /opt/kubernetes/bin/kubectl get node
NAME           STATUS    ROLES     AGE       VERSION
192.168.1.22   Ready     <none>    6s        v1.11.0
192.168.1.25   Ready     <none>    13s       v1.11.0

6.3 部署kube-proxy组件

所有node节点操作

#创建kube-proxy配置文件
[root@k8s-node1 ~]# vim /opt/kubernetes/cfg/kube-proxy
[root@k8s-node1 ~]# cat /opt/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.1.22 \     #修改为本机IP
--cluster-cidr=192.168.1.0/24 \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"

#systemd管理kube-proxy组件
[root@k8s-node1 ~]# vim /usr/lib/systemd/system/kube-proxy.service
[root@k8s-node1 ~]# cat /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

#启动
[root@k8s-node1 ~]# systemctl daemon-reload
[root@k8s-node1 ~]# systemctl start kube-proxy
[root@k8s-node1 ~]# systemctl enable kube-proxy

7. 查看集群状态

master查看

[root@k8s-master bin]# /opt/kubernetes/bin/kubectl get node
NAME           STATUS    ROLES     AGE       VERSION
192.168.1.22   Ready     <none>    8m        v1.11.0
192.168.1.25   Ready     <none>    8m        v1.11.0

[root@k8s-master bin]# /opt/kubernetes/bin/kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-1               Healthy   {"health": "true"}   
etcd-2               Healthy   {"health": "true"}   
etcd-0               Healthy   {"health": "true"}

8. 运行一个测试示例

创建一个Nginx Web，判断集群是否正常工作

[root@k8s-master bin]# /opt/kubernetes/bin/kubectl run nginx --image=nginx --replicas=3

[root@k8s-master bin]# /opt/kubernetes/bin/kubectl expose deployment nginx --port=88 --target-port=80 --type=NodePort

查看Pod，Service：
[root@k8s-master bin]# /opt/kubernetes/bin/kubectl get pods
NAME                     READY     STATUS             RESTARTS   AGE
nginx-64f497f8fd-fjgt2   1/1       Running            0          3m
nginx-64f497f8fd-gmstq   1/1       Running            0          3m
nginx-64f497f8fd-q6wk9   1/1       Running            0          3m

查看pod详细信息：
[root@k8s-master bin]# /opt/kubernetes/bin/kubectl describe pod nginx-64f497f8fd-fjgt2

[root@k8s-master bin]# /opt/kubernetes/bin/kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   192.168.1.1     <none>        443/TCP        3m
nginx        NodePort    192.168.1.175   <none>        88:38696/TCP   3m


[root@k8s-master bin]# curl -I 192.168.1.22:38696
或者
浏览器访问：http://192.168.1.22:38696 测试

image.png

9. 部署Dashboard（Web UI）

部署UI有三个文件：
* dashboard-deployment.yaml           // 部署Pod，提供Web服务
* dashboard-rbac.yaml                 // 授权访问apiserver获取信息
* dashboard-service.yaml              // 发布服务，提供对外访问

[root@k8s-master ~]# mkdir /k8s/dashboard
[root@k8s-master ~]# cd /k8s/dashboard
[root@k8s-master dashboard]# vim dashboard-deployment.yaml
[root@k8s-master dashboard]# cat dashboard-deployment.yaml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: kubernetes-dashboard
      containers:
      - name: kubernetes-dashboard
        image: registry.cn-hangzhou.aliyuncs.com/kube_containers/kubernetes-dashboard-amd64:v1.8.1 
        resources:
          limits:
            cpu: 100m
            memory: 300Mi
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 9090
          protocol: TCP
        livenessProbe:
          httpGet:
            scheme: HTTP
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"

[root@k8s-master dashboard]# vim dashboard-rbac.yaml
[root@k8s-master dashboard]# cat dashboard-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
    addonmanager.kubernetes.io/mode: Reconcile
  name: kubernetes-dashboard
  namespace: kube-system
---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: kubernetes-dashboard-minimal
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kube-system

[root@k8s-master dashboard]# vim dashboard-service.yaml
[root@k8s-master dashboard]# cat dashboard-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard
  ports:
  - port: 80
    targetPort: 9090

#创建
[root@k8s-master dashboard]# /opt/kubernetes/bin/kubectl create -f dashboard-rbac.yaml
[root@k8s-master dashboard]# /opt/kubernetes/bin/kubectl create -f dashboard-deployment.yaml
[root@k8s-master dashboard]# /opt/kubernetes/bin/kubectl create -f dashboard-service.yaml

#等待数分钟，查看资源状态
[root@k8s-master dashboard]# /opt/kubernetes/bin/kubectl get all -n kube-system

#查看访问端口
[root@k8s-master dashboard]# /opt/kubernetes/bin/kubectl get svc -n kube-system
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort    192.168.1.100   <none>        443:31209/TCP   1m

浏览器访192.168.1.22:31209测试

kubernetes 部署：二进制方式部署 kubernetes 集群

kubernetes 部署：二进制方式部署 kubernetes 集群

二进制方式部署 kubernetes 集群

目标任务

1. 准备

2. 部署Etcd集群

3. node节点安装docker

4. 部署Flannel网络

工作原理

5. 在Master节点部署组件

5.1 生成证书

5.2 部署apiserver组件

5.3 部署schduler组件

5.4 部署controller-manager组件

5.5 通过kubectl工具查看当前集群组件状态

6. 在Node节点部署组件

6.1 master节点操作

6.2 部署kubelet组件

6.3 部署kube-proxy组件

7. 查看集群状态

8. 运行一个测试示例

9. 部署Dashboard（Web UI）

kubernetes 部署：二进制方式 部署 kubernetes 集群

二进制方式部署 kubernetes 集群

目标任务

1. 准备

2. 部署Etcd集群

3. node节点安装docker

4. 部署Flannel网络

工作原理

5. 在Master节点部署组件

5.1 生成证书

5.2 部署apiserver组件

5.3 部署schduler组件

5.4 部署controller-manager组件

5.5 通过kubectl工具查看当前集群组件状态

6. 在Node节点部署组件

6.1 master节点操作

6.2 部署kubelet组件

6.3 部署kube-proxy组件

7. 查看集群状态

8. 运行一个测试示例

9. 部署Dashboard（Web UI）

kubernetes 部署：二进制方式部署 kubernetes 集群