一. 准备工作与注意事项

1. 部署的项目情况

(1) 业务架构及服务(dubbo, spring cloud)
(2) 第三方服务，例如mysql, redis, zookeeper, eruke, mq
(3) 服务之间怎么通信
(4) 资源消耗：硬件资源，带宽

2. 项目部署涉及相关k8s资源

(1) 使用namespace 进行不同项目隔离，或者隔离不同环境( test, prod, dev)
(2) 无状态应用(deployment)
(3) 有状态应用(statefulset, pv, pvc)
(4) 发布暴露外部访问( Service, ingress)
(5) 存储一些数据(secret, configmap)

3. 项目的基础镜像

4. 编排部署(镜像为交付物)

https://github.com/lizhenliang/tomcat-java-demo
https://github.com/lizhenliang/php-demo
代码仓库下都会有一个Dockerfile 镜像构建文件

(1). 持续集成与交付

项目构建(java)，CI/CD 环境这个阶段自动完成（代码拉取->代码编译构建->镜像打包 -> 推送到镜像仓库）

(2). 编写yaml文件

编写yaml文件， 使用这个镜像

5. 持续部署

kubectl -> yaml -> 镜像仓库拉取镜像 -> Service(集群内部访问)/Ingress暴露外部用户访问

二. 部署Java项目

1. NFS网络存储准备

安装NFS服务,每个node节点都安装nfs客户端：

# yum install nfs-utils -y
# cat /etc/exports
/ifs/kubernetes *(rw,no_root_squash)

# mkdir /ifs/kubernetes -p
# systemctl start nfs

2. 配置NFS动态卷供给PV

下载三个yaml文件：
https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client/deploy
class.yaml 动态创建PV
deployment.yaml 创建控制及pod等资源
rbac.yaml deployment.yaml部署的pod时，需要访问api-server权限请求k8s的PV&PVC资源
注意： deployment.yaml中定义连接NFS的地址及目录

3. 部署私有镜像仓库harbor

https://www.jianshu.com/p/7ca6c59f9882
注意：私有镜像仓库-harbor 使用http访问必须在每个node做信任,并重启docker

# cat /etc/docker/daemon.json
{
  "registry-mirrors": ["http://bc437cce.m.daocloud.io"],
  "insecure-registries":["http://10.40.6.165"]
}

4. java代码拉取

用一个node节点clone java代码：

# yum install git -y 
# git clone https://github.com/lizhenliang/tomcat-java-demo.git
# cd tomcat-java-demo/
# ll
total 24
drwxr-xr-x 2 root root    34 Jun 15 11:40 db
-rw-r--r-- 1 root root   148 Jun 15 11:40 Dockerfile
-rw-r--r-- 1 root root 11357 Jun 15 11:40 LICENSE
-rw-r--r-- 1 root root  1930 Jun 15 11:40 pom.xml
-rw-r--r-- 1 root root    89 Jun 15 11:40 README.md
drwxr-xr-x 3 root root    18 Jun 15 11:40 src

要用容器化部署自己的项目，一般在你项目代码创建一个Dockerfile文件，使用Dockerfile构建项目镜像。

# cat Dockerfile 
FROM lizhenliang/tomcat 
LABEL maintainer www.ctnrs.com
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war 

5. maven和java环境准备

java是动态语言，需要构建war包或jar包，这构建需要maven和java环境

# tar xvf apache-maven-3.6.1-bin.tar.gz 
# tar xvf jdk-8u181-linux-x64.tar.gz 
#  mv apache-maven-3.6.1 /usr/local/services/
# mv jdk1.8.0_181 /usr/local/services/
# cat /etc/profile.d/java.sh
JAVA_HOME=/usr/local/services/jdk1.8.0_181
PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME PATH

# source /etc/profile
# java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

# ln -sv /usr/local/services/apache-maven-3.6.1/bin/mvn /usr/bin/mvn

6. 项目代码编译

编译完成后会生产一个target 目录，此目录会有一个war包。

# cd tomcat-java-demo/
# mvn clean package
# ll target/
total 17840
drwxr-xr-x 5 root root       95 Jun 15 14:45 classes
drwxr-xr-x 3 root root       25 Jun 15 14:45 generated-sources
drwxr-xr-x 4 root root       37 Jun 15 14:46 ly-simple-tomcat-0.0.1-SNAPSHOT
-rw-r--r-- 1 root root 18265402 Jun 15 14:46 ly-simple-tomcat-0.0.1-SNAPSHOT.war
drwxr-xr-x 2 root root       28 Jun 15 14:46 maven-archiver
drwxr-xr-x 3 root root       35 Jun 15 14:45 maven-status

7. 构建项目镜像并推送至镜像仓库

通过Dockefile构建项目镜像

# cat Dockerfile 
FROM lizhenliang/tomcat 
LABEL maintainer www.ctnrs.com
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war

# docker build -t 10.40.6.165/project/java-demo:v1 .   ##docker build -t 镜像仓库推送地址
# docker login 10.40.6.165
# docker push 10.40.6.165/project/java-demo:v1    ##推到到镜像仓库

8. 编写部署项目yaml文件

yaml部署资源流程:
tomcat：
deployment
service
ingress

mysql：
statefuleset
healess service
pv,pvc(storageclass PV 自动供给 )

yaml文件：
namespace.yaml
deployment.yaml
ingress.yaml
service.yaml
mysql.yaml
registry-pull-secret.yaml
这些yaml文件可以统一写到一个yaml, 使用'---'分隔即可。

9. 创建项目资源

(1). 创建命名空间

# cat namespace.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: test

#kubectl create -f namespace.yaml
# kubectl get ns -n test

(2). 创建镜像仓库Secret用户验证

拉取镜像需要用户验证，先创建kubernetes 登录镜像仓库的Secret用户验证，注意拉取镜像的yaml配置文件中Secret name字段，要与此创建的一致

# kubectl create secret --help
# kubectl create secret docker-registry --help
     ....
Usage:
kubectl create secret docker-registry NAME --docker-username=user --docker-password=password --docker-email=email

指定Secret创建到 test命名空间，因为我们的项目创建在test命名空间

# kubectl create secret docker-registry registry-pull-secret --docker-username=admin --docker-password=Harbor12345 --docker-email=admin@test.com --docker-server=10.40.6.165 -n test

# kubectl get secret -n test
NAME                   TYPE                                  DATA   AGE
default-token-vqllw    kubernetes.io/service-account-token   3      30m
registry-pull-secret   kubernetes.io/dockerconfigjson        1      69s

(3). 创建项目pod

# cat deployment.yaml 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: tomcat-java-demo
  namespace: test
spec:
  replicas: 3
  selector:
    matchLabels:
      project: www
      app: java-demo
  template:
    metadata:
      labels:
        project: www
        app: java-demo
    spec:
      imagePullSecrets:
      - name: registry-pull-secret
      containers:
      - name: tomcat
        image: 10.40.6.165/project/java-demo:v1 
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        resources:
          requests:
            cpu: 0.5
            memory: 1Gi
          limits:
            cpu: 1
            memory: 2Gi
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 20
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 20

# kubectl create -f deployment.yaml
# kubectl get pod -n test

(4). 创建 service

# cat service.yaml
apiVersion: v1
kind: Service
metadata:
  name: tomcat-java-demo
  namespace: test
spec:
  selector:
    project: www
    app: java-demo
  ports:
  - name: web
    port: 80
    targetPort: 8080

# kubectl create -f service.yaml
# kubectl get svc -n test

(5). 创建ingress规则

# cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tomcat-java-demo 
  namespace: test
spec:
  rules:
    - host: java.ctnrs.com
      http:
        paths:
        - path: /
          backend:
            serviceName: tomcat-java-demo 
            servicePort: 80
# kubectl create -f ingress.yaml 
# kubectl get ingress -n test
# kubectl get pods,svc,ing -n test

10. 测试访问

# kubectl get pods -n ingress-nginx -o wide
NAME                                        READY   STATUS    RESTARTS   AGE    IP            NODE          NOMINATED NODE
nginx-ingress-controller-74cbc544f6-9v5qv   1/1     Running   0          2d3h   172.17.59.5   10.40.6.213   <none>
nginx-ingress-controller-74cbc544f6-qklwh   1/1     Running   0          2d3h   172.17.31.5   10.40.6.210   <none>

# kubectl get ing -n test
NAME               HOSTS            ADDRESS   PORTS   AGE
tomcat-java-demo   java.ctnrs.com             80      10m

绑定域名到ingress-nginx的节点IP上访问：
10.40.6.210 java.ctnrs.com
http://java.ctnrs.com

11. 创建数据库service、pod、PV和PVC

# cat mysql.yaml 
apiVersion: v1
kind: Service
metadata:
  name: java-demo-mysql
  namespace: test
  labels:
    project: java-demo
    app: mysql
spec:
  ports:
  - port: 3306
    name: java-demo-mysql
  clusterIP: None
  selector:
    project: java-demo
    app: mysql

---

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: db
  namespace: test
spec:
  selector:
    matchLabels:
      project: java-demo
      app: mysql
  serviceName: "java-demo-mysql"
  template:
    metadata:
      labels:
        project: java-demo
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:5.7 
        env: 
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
        ports: 
        volumeMounts:
        - mountPath: "/var/lib/mysql"
          name: mysql-data
  volumeClaimTemplates:
  - metadata:
      name: mysql-data 
    spec:
      accessModes: ["ReadWriteMany"]
      storageClassName: "managed-nfs-storage"
      resources:
        requests:
          storage: 2Gi 

# kubectl create -f mysql.yaml
# kubectl get svc,pods -n test

# kubectl get pv,pvc -n test
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS          REASON   AGE
persistentvolume/pvc-03054e78-8f53-11e9-8287-005056b614b3   2Gi        RWX            Delete           Bound    test/mysql-data-db-0   managed-nfs-storage            14m

NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
persistentvolumeclaim/mysql-data-db-0   Bound    pvc-03054e78-8f53-11e9-8287-005056b614b3   2Gi        RWX            managed-nfs-storage   14m

12. 数据库配置

将tables_ly_tomcat.sql sql文件导入进去，并修改数据库连接

# scp  db/tables_ly_tomcat.sql 10.40.6.201:~   ##将文件拷贝到master节点
# kubectl cp /root/tables_ly_tomcat.sql db-0:/ -n test   ## 将sql文件cp到db-0容器中
# kubectl exec -it db-0 -n test bash
root@db-0:/# mysql -uroot -p123456
mysql> source /tables_ly_tomcat.sql
Query OK, 1 row affected (0.01 sec)

Database changed
Query OK, 0 rows affected (0.02 sec)

Query OK, 0 rows affected (0.00 sec)
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test               |
+--------------------+
5 rows in set (0.01 sec)
mysql> use test
Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| user           |
+----------------+
1 row in set (0.00 sec)
mysql> desc user;
+-------+--------------+------+-----+---------+----------------+
| Field | Type         | Null | Key | Default | Extra          |
+-------+--------------+------+-----+---------+----------------+
| id    | int(11)      | NO   | PRI | NULL    | auto_increment |
| name  | varchar(100) | NO   |     | NULL    |                |
| age   | int(3)       | NO   |     | NULL    |                |
| sex   | char(1)      | YES  |     | NULL    |                |
+-------+--------------+------+-----+---------+----------------+
4 rows in set (0.02 sec)

登录项目容器测试mysql容器域名：
域名格式：pod_name.service_name.namespace

# kubectl exec -it tomcat-java-demo-5f87cd895d-4zszj -n test bash
[root@tomcat-java-demo-5f87cd895d-4zszj tomcat]# ping -c 1 db-0.java-demo-mysql.test
PING db-0.java-demo-mysql.test.svc.cluster.local (172.17.59.6) 56(84) bytes of data.

# kubectl get pod -n test -o wide
NAME                                READY   STATUS    RESTARTS   AGE    IP            NODE          NOMINATED NODE
db-0                                1/1     Running   0          20m    172.17.59.6   10.40.6.213   <none>

修改项目连接数据库配置文件 src/main/resources/application.yml

# cat src/main/resources/application.yml
server:
  port: 8080
spring:
  datasource:
    url: jdbc:mysql://db-0.java-demo-mysql.test:3306/test?characterEncoding=utf-8
    username: root
    password: 12345
    driver-class-name: com.mysql.jdbc.Driver
  freemarker:
    allow-request-override: false
    cache: true
    check-template-location: true
    charset: UTF-8
    content-type: text/html; charset=utf-8
    expose-request-attributes: false
    expose-session-attributes: false
    expose-spring-macro-helpers: false
    suffix: .ftl
    template-loader-path:
      - classpath:/templates/

更新代码后更新镜像，并推送到镜像仓库

# docker build -t 10.40.6.165/project/java-demo:v2 .
# docker push 10.40.6.165/project/java-demo:v2

修改deployment.yaml 项目镜像版本为：10.40.6.165/project/java-demo:v2
滚动更新项目：kubectl apply -f deployment.yaml
然后访问站点添加美女.

添加美女.png

今晚翻盘哪个美女.png

三. 部署PHP

部署跟java基本一致，只是代码不需要编译构建
clone代码 -> 制作代码镜像 -> 代码镜像推送至镜像仓库->编写yaml配置文件 ->创建资源对象

1. clone代码

git clone https://github.com/lizhenliang/php-demo.git

2. 构建项目镜像并推送至镜像仓库

这里的数据库就用上边java项目创建的数据库，修改php代码连接数据库的配置文件：

php代码连接数据库的配置文件
# cat php-demo/wp-config.php
      ....
/** MySQL主机 */
define('DB_HOST', 'db-0.java-demo-mysql.test');
     ....


代码目录php-demo中的Dockfile文件：
# cat Dockerfile
FROM lizhenliang/nginx-php:latest
MAINTAINER www.ctnrs.com
ADD . /usr/local/nginx/html

构建项目镜像,并推送到镜像仓库
# docker build -t 10.40.6.165/project/php-demo:1.0 .
# docker push 10.40.6.165/project/php-demo:1.0 

3. 编写yaml配置文件并创建资源对象

命名空间test 在java 项目已经创建，这里不再创建

(1). 创建pod

# cat deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: php-demo
  namespace: test
spec:
  replicas: 2
  selector:
    matchLabels:
      project: www
      app: php-demo
  template:
    metadata:
      labels:
        project: www
        app: php-demo
    spec:
      imagePullSecrets:
      - name: registry-pull-secret
      containers:
      - name: nginx
        image: 10.40.6.165/project/php-demo:1.0
        imagePullPolicy: Always
        ports:
        - containerPort: 80
          name: web
          protocol: TCP
        resources:
          requests:
            cpu: 0.5
            memory: 256Mi
          limits:
            cpu: 1
            memory: 1Gi
        livenessProbe:
          httpGet:
            path: /index.php
            port: 80
          initialDelaySeconds: 6
          timeoutSeconds: 20

# kubectl create -f deployment.yaml
# kubectl get pod -n test

(2). 创建service

# cat service.yaml
apiVersion: v1
kind: Service
metadata:
  name: php-demo
  namespace: test
spec:
  selector:
    project: www
    app: php-demo
  ports:
  - name: web
    port: 80
    targetPort: 80

# kubectl create -f service.yaml
# kubectl get svc -n test

(3). 创建ingress规则

# cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: php-demo
  namespace: test
spec:
  rules:
    - host: php.ctnrs.com
      http:
        paths:
        - path: /
          backend:
            serviceName: php-demo
            servicePort: 80

# kubectl create -f ingress.yaml
# kubectl  get ing -n test

(4). 配置数据库

php项目与java项目同用一个数据库test, test库中有同用一个user表，这里将之前java项目到user表删除掉

# kubectl exec -it db-0 bash  -n test
root@db-0:/# mysql -uroot -p123456
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test               |
+--------------------+
5 rows in set (0.00 sec)

mysql> use test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| user           |
+----------------+
1 row in set (0.00 sec)

mysql> drop table user;
Query OK, 0 rows affected (0.01 sec)

(5). 验证

绑定域名，浏览器访问

# kubectl get ing -n test
NAME               HOSTS            ADDRESS   PORTS   AGE
php-demo           php.ctnrs.com              80      14m
tomcat-java-demo   java.ctnrs.com             80      8h

本地绑定hosts:
10.40.6.210 php.ctnrs.com

访问: http://php.ctnrs.com