使用kubeadm安装kubernetes1.10.0

kubeadm原理简介

kubeadm reference ：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

主要命令：

kubeadm init  #初始化主节点
kubeadm join  #初始化工作节点并加入集群
kubeadm upgrade  #to upgrade a Kubernetes cluster to a newer version
kubeadm config  #升级已使用低版本kubeadm初始化的集群
kubeadm token  #管理集群计入工作节点使用的token
kubeadm reset  #充值kubeadm设置

kubeadm init

kubeadm init流程

kubeadm init主要工作：

创建集群安全相关的的key、certs和conf文件。
kube-apiserver、kube-controller-manager、kube-scheduler、etcd(如果没有配置external etcd)这些static pod的json格式的manifest文件，kubelet负责启动这些master组件。
addons方式启动kube-discovery deployment、kube-proxy daemonSet、kube-dns deployment。

kubeadm join

kubeadm join流程

kubeadm join主要负责创建kubelet.conf，使kubelet能与API Server建立连接：

访问kube-discovery服务获取cluster info（包含cluster ca证书、API Server endpoint列表和token。
利用定的token，检验cluster info的签名
检验成功后，再与API Server建立连接，请求API Server为该node创建证书。
根据获取到的证书创建kubelet.conf。

kubernetes1.10安装

环境准备

3台centos7.2虚拟机

172.20.95.112 k8s-master

172.20.95.113 k8s-node1

172.20.95.114 k8s-node2

关闭SELinux和防火墙（所有节点）

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config # 修改配置永久生效，需重启
setenforce 0
#关闭防火墙 
systemctl stop firewalld && systemctl disable firewalld

关闭swap（所有节点）

swapoff -a #保证 kubelet 正确运行

配置各节点系统内核参数使流过网桥的流量也进入iptables/netfilter框架中（所有节点）

cat <<EOF >  /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p

配置阿里K8S YUM源（所有节点）

cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF

组件安装

docker安装（所有节点）

yum -y install docker

systemctl enable docker && systemctl start docker

配置阿里镜像加速器

sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://xxxxx.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

kubeadm安装（所有节点）

如果需要制定版本，可以先查看具体版本

yum list kubeadm --showduplicates

安装默认最新版本

yum install -y kubelet kubeadm kubectl

此处使用1.10版本

yum install -y kubelet-1.10.0-0  kubeadm-1.10.0-0  kubectl-1.10.0-0

kubelet设置开机自动运行

systemctl enable kubelet

kubelet启动参数增加

--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice

防止kubelet报错

vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

将 KUBELET_CGROUP_ARGS 一行改为：

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice"

然后reload并启动kubelet

systemctl daemon-reload
systemctl start kubelet

下载镜像

主节点镜像列表

镜像下载脚本 images.sh (注意版本与kubeadm对应)

#!/bin/bash
images=(kube-proxy-amd64:v1.10.0 kube-scheduler-amd64:v1.10.0 kube-controller-manager-amd64:v1.10.0 kube-apiserver-amd64:v1.10.0
etcd-amd64:3.1.12 pause-amd64:3.1 kubernetes-dashboard-amd64:v1.8.3 k8s-dns-sidecar-amd64:1.14.8 k8s-dns-kube-dns-amd64:1.14.8
k8s-dns-dnsmasq-nanny-amd64:1.14.8)
for imageName in ${images[@]} ; do
  docker pull keveon/$imageName
  docker tag keveon/$imageName k8s.gcr.io/$imageName
  docker rmi keveon/$imageName
done

从节点镜像列表

镜像下载脚本 images.sh (注意版本与kubeadm对应)

#!/bin/bash
docker pull quay.io/coreos/flannel:v0.9.1-adm64
images=(kube-proxy-amd64:v1.10.0   
pause-amd64:3.1 kubernetes-dashboard-amd64:v1.8.3)
for imageName in ${images[@]} ; do
  docker pull keveon/$imageName
  docker tag keveon/$imageName k8s.gcr.io/$imageName
  docker rmi keveon/$imageName
done

主节点初始化

[root@k8smaster ~]# kubeadm init --kubernetes-version=v1.10.0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=0.0.0.0
[init] Using Kubernetes version: v1.10.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
    [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
    [WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[preflight] Starting the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8smaster kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.20.95.112]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8smaster] and IPs [172.20.95.112]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 21.001790 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node k8smaster as master by adding a label and a taint
[markmaster] Master k8smaster tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: thczis.64adx0imeuhu23xv
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy
 
Your Kubernetes master has initialized successfully!
 
To start using your cluster, you need to run the following as a regular user:
 
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
 
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
 
You can now join any number of machines by running the following on each node
as root:
 
  kubeadm join 172.20.95.112:6443 --token 18qmjp.535rx7ueuf5ig28x --discovery-token-ca-cert-hash sha256:626d17c8d80dc693bab771420224666214a2be89da1e98c4e739d4e496eda8f0

--kubernetes-version=v1.10.0是必须的，下载的容器镜像版本必须与K8S版本一致否则会出现time out
--pod-network-cidr 指定子网
--apiserver-advertise-address指定apiserver访问方式，默认localhost

记录kubeadm join那一串指令，节点加入集群时使用该命令

配置kubectl认证信息（主节点）

对于非root用户

mkdir -p $HOME/.kube
 
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 
sudo chown $(id -u):$(id -g) $HOME/.kube/config

对于root用户

export KUBECONFIG=/etc/kubernetes/admin.conf

安装flannel网络（主节点）

mkdir -p /etc/cni/net.d/
 
cat <<EOF> /etc/cni/net.d/10-flannel.conf
{
“name”: “cbr0”,
“type”: “flannel”,
“delegate”: {
“isDefaultGateway”: true
}
}
 
EOF
 
mkdir /usr/share/oci-umount/oci-umount.d -p
 
mkdir /run/flannel/
 
cat <<EOF> /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.1.0/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
 
EOF
 
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

node加入集群

[root@k8snode1 ~]# kubeadm join 172.20.95.112:6443 --token thczis.64adx0imeuhu23xv --discovery-token-ca-cert-hash sha256:fa7b11bb569493fd44554aab0afe55a4c051cccc492dbdfafae6efeb6ffa80e6
[preflight] Running pre-flight checks.
    [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
    [WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[discovery] Trying to connect to API Server "172.20.95.112:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.20.95.112:6443"
[discovery] Requesting info from "https://172.20.95.112:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.20.95.112:6443"
[discovery] Successfully established connection with API Server "172.20.95.112:6443"
 
This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.
 
Run 'kubectl get nodes' on the master to see this node join the cluster.

[root@k8s-master ~]# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
[root@k8s-master ~]# kubectl get node
NAME         STATUS    ROLES     AGE       VERSION
k8s-master   Ready     master    2d        v1.10.0
k8s-node-1   Ready     <none>    2d        v1.10.0
k8s-node-2   Ready     <none>    2d        v1.10.0

至此，安装完成。

常见问题（持续更新）

使用kubectl命令是出现如下错误

Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

解决办法：

将$KUBECONFIG指向/etc/kubernetes/kubelet.conf

export KUBECONFIG=/etc/kubernetes/kubelet.conf

Node状态显示为“NotReady”

通过kubectl get node会发现所有的node都还是not ready状态，这是因为还没有配置好flannel网络.
确认节点中是否有flannel镜像，并确认版本是否一致。