环境依赖：
4台机器，安装ubuntu16.04；1台master；2台node；1台harbor；
安装过程：
整体流程梳理为先安装master节点，然后再安装node节点，最后安装harbor（以裸机只安装了ubuntu16开始）：
0，配置hosts网络发现：
sudo vim /etc/hosts
10.10.30.76 k8smaster
10.10.30.75 k8snode1
10.10.30.74 k8snode2
10.10.30.73 harbor
1，配置apt国内源：（本文使用的是阿里源，这样会提高很多速度）
//打开源的配置文件
sudo vim /etc/apt/sources.list//删除原有的内容，如果担心的话可以先备份之前的内容，然后拷贝内容进入文件
deb http://mirrors.aliyun.com/ubuntu/ xenial main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial main
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates main
deb http://mirrors.aliyun.com/ubuntu/ xenial universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial universe
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
deb http://mirrors.aliyun.com/ubuntu/ xenial-security main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security main
deb http://mirrors.aliyun.com/ubuntu/ xenial-security universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security universe//保存后运行
sudo apt-get update
2，安装Docker：（如果在后面kubernetes配置后不想看见warning，那么就需要选择版本低于等于19）
//安装基础包
sudo apt-get update
sudo apt-get -y install apt-transport-https ca-certificates curl software-properties-common//安装GPG证书
curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
//使用国内docker源提高速度
sudo add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"//更新并安装docker-ce，如果需要安装指定版本则进行如下编辑，安装最新版本就不需要指定版本号
sudo apt-get -y update
sudo apt-get -y install docker-ce=19.03.0~3-0~ubuntu-xenial
3，安装kubelet，kubeadm，kubectl：
//安装依赖
sudo apt-get update && apt-get install -y apt-transport-https//安装GPG证书
sudo curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -//使用国内kubernetes的源（阿里源）
sudo vim /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main//更新源后进行安装
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl//关闭swap和防火墙，否则kubernetes会启动失败
sudo swapoff -a
sudo vim /etc/fstab #注释有swap的行// 修改cgroupfs，否则会报错
sudo vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf #增加 --cgroup-driver=cgroupfs 在环境里//重启kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet//初始化k8s集群，192的地址就是master的地址根据自己情况进行修改，cidr和pod必须配置否则没有网络插件后面的node不能加入
kubeadm init --apiserver-advertise-address=192.168.20.147 --image-repository registry.aliyuncs.com/google_containers --service-cidr=10.1.0.0/16 --pod-network-cidr=10.244.0.0/16[init] Using Kubernetes version: v1.20.1 [preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.1. Latest validated version: 19.03 [preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names
[kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local pcno6] and IPs [10.1.0.1 10.10.30.76]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost pcno6] and IPs [10.10.30.76 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost pcno6] and IPs [10.10.30.76 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 21.501296 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node pcno6 as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node pcno6 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: 4xic6x.jhcftora8nvnlbt9
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.10.30.76:6443 --token 4xic6x.jhcftora8nvnlbt9 \ --discovery-token-ca-cert-hash sha256:d23cc3c4aba172df962ecb8735292793cb7a101965ffa71631bf50e593c31904//安装init的输出提示start你的cluster
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config// 安装flannel，这是k8s必须的一个插件，国外源无法直接下载，所以要先加dns
sudo vim /etc/hosts #在最后增加：199.232.28.133 raw.githubusercontent.com
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sed 's/quay.io\/coreos/registry.cn-beijing.aliyuncs.com\/imcto/g'
sudo kubectl apply -f kube-flannel.yml// 只有装了网络插件后k8s集群才能正常启动（安装后需要等几分钟，才能看出来）
// 刚init完kubernetes是这样的
kubectl get nodes
NAME STATUS ROLES AGE VERSION
pcno6 NotReady control-plane,master 6m3s v1.20.1// 过了几分钟是这样的
kubectl get nodes
NAME STATUS ROLES AGE VERSION
pcno6 Ready control-plane,master 7m3s v1.20.1//然后查看，里面的pcno6就是master
kubectl get pods -n kube-system -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-7f89b7bc75-78zxc 1/1 Running 0 16h 10.244.0.3 pcno6 <none> <none>
coredns-7f89b7bc75-xf7sh 1/1 Running 0 16h 10.244.0.2 pcno6 <none> <none>
etcd-pcno6 1/1 Running 0 16h 10.10.30.76 pcno6 <none> <none>
kube-apiserver-pcno6 1/1 Running 0 16h 10.10.30.76 pcno6 <none> <none>
kube-controller-manager-pcno6 1/1 Running 0 16h 10.10.30.76 pcno6 <none> <none>
kube-flannel-ds-hw78n 1/1 Running 0 16h 10.10.30.76 pcno6 <none> <none>
kube-proxy-ntkr4 1/1 Running 0 16h 10.10.30.76 pcno6 <none> <none>
kube-scheduler-pcno6 1/1 Running 0 16h 10.10.30.76 pcno6 <none> <none>
4，安装node机器：
照上面的步骤全部做一遍，只是把kubeadm init ，换成 kubeadm join（用init最后输出的kubeadm join就好）
sudo kubeadm join 10.10.30.76:6443 --token 4xic6x.jhcftora8nvnlbt9 --discovery-token-ca-cert-hash sha256:d23cc3c4aba172df962ecb8735292793cb7a101965ffa71631bf50e593c31904 --ignore-preflight-errors=...
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.1. Latest validated version: 19.03 [preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.//然后在master机器上查看（pcno5就是node节点）
kubectl get nodes
NAME STATUS ROLES AGE VERSION
pcno5 Ready <none> 13m v1.20.1
pcno6 Ready control-plane,master 74m v1.20.1//再查看master的所有pod和container
kubectl get pods -n kube-system -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-7f89b7bc75-78zxc 1/1 Running 0 16h 10.244.0.3 pcno6 <none> <none>
coredns-7f89b7bc75-xf7sh 1/1 Running 0 16h 10.244.0.2 pcno6 <none> <none>
etcd-pcno6 1/1 Running 0 16h 10.10.30.76 pcno6 <none> <none>
kube-apiserver-pcno6 1/1 Running 0 16h 10.10.30.76 pcno6 <none> <none>
kube-controller-manager-pcno6 1/1 Running 0 16h 10.10.30.76 pcno6 <none> <none>
kube-flannel-ds-6zv79 1/1 Running 0 15h 10.10.30.75 pcno5 <none> <none>
kube-flannel-ds-hw78n 1/1 Running 0 16h 10.10.30.76 pcno6 <none> <none>
kube-proxy-7nttl 1/1 Running 0 15h 10.10.30.75 pcno5 <none> <none>
kube-proxy-ntkr4 1/1 Running 0 16h 10.10.30.76 pcno6 <none> <none>
kube-scheduler-pcno6 1/1 Running 0 16h 10.10.30.76 pcno6 <none> <none>
