基于Kubernetes构建企业Jenkins CI/CD平台

1.蓝绿发布

项目逻辑上分为AB组，在项目升级时，首先把A组从负载均衡中摘除，进行新版本的部署。

B组仍然继续提供服务。A组升级完成上线，B组从负载均衡中摘除。

特点：

策略简单
升级/回滚速度快
用户无感知，平滑过渡

缺点：

需要两倍以上服务器资源
短时间内浪费一定资源成本

2.灰度发布

灰度发布：只升级部分服务，即让一部分用户继续用老版本，一部分用户开始用新版本，如果用户对新版本没有什么意见，那么逐步扩大范围，把所有用户都迁移到新版本上面来。

特点：

保证整体系统稳定性
用户无感知，平滑过渡

缺点：

自动化要求高

k8s 中的落地方式

3.滚动发布

滚动发布：

每次只升级一个或多个服务，升级完成后加入生产环境，不断执行这个过程，直到集群中的全部旧版升级新版本。特点：

用户无感知，平滑过渡

缺点：

部署周期长
发布策略较复杂
不易回滚

4.发布流程

5在Kubernetes中部署Jenkins

部署文档：https://github.com/jenkinsci/kubernetes-plugin/tree/fc40c869edfd9e3904a9a56b0f80c5a25e988fa1/src/main/kubernetes

mkdir k8s-ci/jenkins -p && cd k8s-cli/jenkins

rabc.yml

---
# 创建名为jenkins的ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
 name: jenkins
 ---
# 创建名为jenkins的Role，授予允许管理API组的资源Pod
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
 name: jenkins
rules:
- apiGroups: [""]
 resources: ["pods"]
 verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
 resources: ["pods/exec"]
 verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
 resources: ["pods/log"]
 verbs: ["get","list","watch"]
- apiGroups: [""]
 resources: ["secrets"]
 verbs: ["get"]
 ---
# 将名为jenkins的Role绑定到名为jenkins的ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
 name: jenkins
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: jenkins
subjects:
- kind: ServiceAccount
 name: jenkins

statefulset.yml

apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: jenkins
 labels:
 name: jenkins
spec:
 serviceName: jenkins
 replicas: 1
 updateStrategy:
 type: RollingUpdate
 selector:
 matchLabels:
 name: jenkins 
 template:
 metadata:
 name: jenkins
 labels:
 name: jenkins
 spec:
 terminationGracePeriodSeconds: 10
 serviceAccountName: jenkins
 containers:
 - name: jenkins
 image: jenkins/jenkins:lts-alpine
 imagePullPolicy: Always
 ports:
 - containerPort: 8080
 - containerPort: 50000
 resources:
 limits:
 cpu: 1
 memory: 1Gi
 requests:
 cpu: 0.5
 memory: 500Mi
 env:
 - name: LIMITS_MEMORY
 valueFrom:
 resourceFieldRef:
 resource: limits.memory
 divisor: 1Mi
 - name: JAVA_OPTS
 value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MAR
GIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85          volumeMounts:
 - name: jenkins-home
 mountPath: /var/jenkins_home
 livenessProbe:
 httpGet:
 path: /login
 port: 8080
 initialDelaySeconds: 60
 timeoutSeconds: 5
 failureThreshold: 12
 readinessProbe:
 httpGet:
 path: /login
 port: 8080
 initialDelaySeconds: 60
 timeoutSeconds: 5
 failureThreshold: 12
 securityContext:
 fsGroup: 1000
 volumeClaimTemplates:
 - metadata:
 name: jenkins-home
 spec:
 storageClassName: "managed-nfs-storage"
 accessModes: [ "ReadWriteOnce" ]
 resources:
 requests:
 storage: 1Gi

service.yml

apiVersion: v1
kind: Service
metadata:
 name: jenkins
spec:
 selector:
 name: jenkins
 type: NodePort
 ports:
 -
 name: http
 port: 80
 targetPort: 8080
 protocol: TCP
 nodePort: 30006
 -
 name: agent
 port: 50000
 protocol: TCP

ingress.yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: jenkins
 annotations:
 nginx.ingress.kubernetes.io/ssl-redirect: "true"
 kubernetes.io/tls-acme: "true"
 # 如果上传插件超出默认会报"413 Request Entity Too Large", 增加 client_max_body_size
 nginx.ingress.kubernetes.io/proxy-body-size: 50m
 nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
 # nginx-ingress controller版本小于 0.9.0.beta-18 的配置
 ingress.kubernetes.io/ssl-redirect: "true"
 ingress.kubernetes.io/proxy-body-size: 50m
 ingress.kubernetes.io/proxy-request-buffering: "off"
spec:
 rules:
 - host: jenkins.example.com
 http:
 paths:
 - path: /
 backend:
 serviceName: jenkins
 servicePort: 80

批量完成创建

这样创建之后就会通过 sts 控制器在node上拉起一个jenkins 容器，可以通过node ip:svc port 访问jenkins

插件默认先不安装--》创建个admin 用户---》更换默认插件源 ---》重启jenkins 容器----》安装 git pipline kubernetes插件

更改配置使用api 重启jenkins 页面访问 http://192.168.31.65:30006/restart

jenkins master/slave架构

这种架构主要解决单jenkins执行效率低，资源不足等问题，jenkins master 调度任务到 slave上并发执行任务。

提升任务执行的效率

传统添加jenkins 节点是在jenkins页面 jenins nodes 添加

那么在 k8s 中如何动态创建jenkins slave 节点呢？

jienkins中的Kubernetes插件：Jenkins在Kubernetes集群中运行动态代理插件介绍：https://github.com/jenkinsci/kubernetes-plugin

一个 slave节点具体的条件：

1.git

2.镜像构建，推送仓库（docker in docker）

3.slave.jar （jenkins slave）从jenkins 中获取 http://192.168.31.65:30006/jnlpJars/slave.jar

4.maven jdk （java 项目）

mkdir k8s-ci/jenkins-slave && cd k8s-ci/jenkins-slave

jenkins slave Dockerfile

FROM centos:7
LABEL maintainer lizhenliang
 RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \ 
 yum clean all && \
 rm -rf /var/cache/yum/* && \
 mkdir -p /usr/share/jenkins
 COPY slave.jar /usr/share/jenkins/slave.jar  
COPY jenkins-slave /usr/bin/jenkins-slave
COPY settings.xml /etc/maven/settings.xml
RUN chmod +x /usr/bin/jenkins-slave
 ENTRYPOINT ["jenkins-slave"]

📎slave.jar 📎settings.xml

jenkins-slave 用来设置这个镜像的 entrypoint 启动jenkins slave

#!/usr/bin/env sh
 if [ $# -eq 1 ]; then
 # if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image
 exec "$@"
 else
 # if -tunnel is not provided try env vars
 case "$@" in
 *"-tunnel "*) ;;
 *)
 if [ ! -z "$JENKINS_TUNNEL" ]; then
 TUNNEL="-tunnel $JENKINS_TUNNEL"
 fi ;;
 esac
 # if -workDir is not provided try env vars
 if [ ! -z "$JENKINS_AGENT_WORKDIR" ]; then
 case "$@" in
 *"-workDir"*) echo "Warning: Work directory is defined twice in command-line arguments and the environment variable" ;;
 *)
 WORKDIR="-workDir $JENKINS_AGENT_WORKDIR" ;;
 esac
 fi
 if [ -n "$JENKINS_URL" ]; then
 URL="-url $JENKINS_URL"
 fi
 if [ -n "$JENKINS_NAME" ]; then
 JENKINS_AGENT_NAME="$JENKINS_NAME"
 fi  
 if [ -z "$JNLP_PROTOCOL_OPTS" ]; then
 echo "Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior"
 JNLP_PROTOCOL_OPTS="-Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true"
 fi
 # If both required options are defined, do not pass the parameters
 OPT_JENKINS_SECRET=""
 if [ -n "$JENKINS_SECRET" ]; then
 case "$@" in
 *"${JENKINS_SECRET}"*) echo "Warning: SECRET is defined twice in command-line arguments and the environment variable" ;;
 *)
 OPT_JENKINS_SECRET="${JENKINS_SECRET}" ;;
 esac
 fi
 OPT_JENKINS_AGENT_NAME=""
 if [ -n "$JENKINS_AGENT_NAME" ]; then
 case "$@" in
 *"${JENKINS_AGENT_NAME}"*) echo "Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable" ;;
 *)
 OPT_JENKINS_AGENT_NAME="${JENKINS_AGENT_NAME}" ;;
 esac
 fi
 #TODO: Handle the case when the command-line and Environment variable contain different values.
 #It is fine it blows up for now since it should lead to an error anyway.
 exec java $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@"
fi

构建 jenkins slave 镜像并推送至私有仓库

 docker build -t 192.168.31.70/library/jenkins-slave-jdk:1.8 .
 docker push  192.168.31.70/library/jenkins-slave-jdk:1.8

测试 job

配置插件

manage jenkins --》configur system ---》拉到最底下 add cloud 选择kubernetes

测试的pipeline 脚本

从私有仓库拉取 jenkins-slave镜像，并起一个pod，完成打印123

实际过程是，jenkins 会提交一个 yaml文件给k8s 编排生成一个pod

// 公共
def registry = "192.168.31.70"
 podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
 containerTemplate(
 name: 'jnlp', 
 image: "${registry}/library/jenkins-slave-jdk:1.8"
 ),
 ],
 volumes: [
 hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
 hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker')
 ],
) 
{
 node("jenkins-slave"){
 // 第一步
 stage('拉取代码'){
 echo "123"
 }
 }
}

点击构建最终会创建出一个 jenkins-slave pod提供构建功能

6.k8s完整发布流程

添加凭据

添加harbor 凭据

添加git凭据

安装插件

Kubernetes Continuous Deploy插件：

用于将资源配置部署到Kubernetes。

插件介绍：https://plugins.jenkins.io/kubernetes-cd

支持以下资源类型:

Deployment
Replica Set
Daemon Set
StatefulSet
Pod • Job
Service
Ingress
Secret

生成config文件

mkdir admin-cert && cd admin-cert

admin-csr.json

{
 "CN": "admin",
 "hosts": [],
 "key": {
 "algo": "rsa",
 "size": 2048
 },
 "names": [
 {
 "C": "CN",
 "ST": "BeiJing",
 "L": "BeiJing",
 "O": "system:masters",
 "OU": "System"
 }
 ]
}

ca-config.json

{
 "signing": {
 "default": {
 "expiry": "87600h"
 },
 "profiles": {
 "kubernetes": {
 "expiry": "87600h",
 "usages": [
 "signing",
 "key encipherment",
 "server auth",
 "client auth"
 ]
 }
 }
 }
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

cp admin*pem /opt/kubernetes/ssl/
export KUBE_APISERVER="https://192.168.31.63:6443"
kubectl config set-cluster kubernetes   --certificate-authority=/opt/kubernetes/ssl/ca.pem   --embed-certs=true   --server=${KUBE_APISE
RVER}
 kubectl config set-cluster kubernetes   --certificate-authority=/opt/kubernetes/ssl/ca.pem   --embed-certs=true   --server=${KUBE_APISE
RVER}
 kubectl config set-credentials admin   --client-certificate=/opt/kubernetes/ssl/admin.pem   --embed-certs=true   --client-key=/opt/kube
rnetes/ssl/admin-key.pem
 kubectl config set-context kubernetes   --cluster=kubernetes   --user=admin
kubectl config use-context kubernetes

既然定义了部署时使用的yaml ，那么可以将yaml文件存放在git 仓库中。

将以下deploy.yaml复制到页面git中,这次拉群代码时也会将这个yam 拉取下来。

deploy.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: web
spec:
 replicas: 3 
 selector:
 matchLabels:
 app: java-demo
 template:
 metadata:
 labels:
 app: java-demo 
 spec:
 imagePullSecrets:
 - name: $SECRET_NAME 
 containers:
 - name: tomcat 
 image: $IMAGE_NAME 
 ports:
 - containerPort: 8080
 name: web
 livenessProbe:
 httpGet:
 path: /
 port: 8080
 initialDelaySeconds: 60
 timeoutSeconds: 5
 failureThreshold: 12
 readinessProbe:
 httpGet:
 path: /
 port: 8080
 initialDelaySeconds: 60
 timeoutSeconds: 5
 failureThreshold: 12
 ---
apiVersion: v1
kind: Service
metadata:
 name: web
spec:
 type: NodePort
 selector:
 app: java-demo 
 ports:
 - protocol: TCP
 port: 80
 targetPort: 8080
 ---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: web 
spec:
 rules:
 - host: java.example.com
 http:
 paths:
 - path: /
 backend:
 serviceName: web 
 servicePort: 80</pre>

### 创建一个登录registry的secret

<pre class="cm-s-default" style="margin: 0px; padding: 0px;">kubectl create secret docker-registry docker-regsitry-auth --docker-username=admin --docker-password=Harbor12345 --docker-server=192.16
8.31.70

完整发布应用的 pipeline 脚本

Jenkinsfile

// 公共
def registry = "192.168.31.70"
// 项目
def project = "welcome"
def app_name = "demo"
def image_name = "${registry}/${project}/${app_name}:${Branch}-${BUILD_NUMBER}"
def git_address = "http://192.168.31.61:9999/root/java-demo.git"
// 认证
def secret_name = "docker-regsitry-auth"
def docker_registry_auth = "6170ebd8-3b94-409b-ad69-2a4753701041"
def git_auth = "3ca1f434-a566-442b-b598-188ac3d07ae2"
def k8s_auth = "58cc1edc-4a58-4541-9e74-cbc29c776b9e"
 podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
 containerTemplate(
 name: 'jnlp', 
 image: "${registry}/library/jenkins-slave-jdk:1.8"
 ),
 ],
 volumes: [
 hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
 hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker')
 ],
) 
{
 node("jenkins-slave"){
 // 第一步
 stage('拉取代码'){
 checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]
]])      }
 // 第二步
 stage('代码编译'){
 sh "mvn clean package -Dmaven.test.skip=true"
 }
 // 第三步
 stage('构建镜像'){
 withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) 
 {            
 sh """
 echo '
 FROM lizhenliang/tomcat 
 RUN rm -rf /usr/local/tomcat/webapps/*
 ADD target/*.war /usr/local/tomcat/webapps/ROOT.war 
 ' > Dockerfile
 docker build -t ${image_name} .
 docker login -u ${username} -p '${password}' ${registry}
 docker push ${image_name}
 """
 }
 }
 // 第四步
 stage('部署到K8S平台'){
 sh """
 sed -i 's#\$IMAGE_NAME#${image_name}#' deploy.yaml
 sed -i 's#\$SECRET_NAME#${secret_name}#' deploy.yaml
 """
 kubernetesDeploy configs: 'deploy.yaml', kubeconfigId: "${k8s_auth}"
 }
 }
}

执行构建等待构建完成