Just the simple way to build docker image in Jenkins pipeline on K8S?

Background

Running Jenkins job in Pod of K8S is a very popular solution for CI. In my environment, it has K8S cluster and Jeninks Kubernetes Plugin connecting to it.

By using pod template provided by that plug-in, whenever a jenkins pipeline job starts, a temporary slave pod being created for it, then destroyed automatically after job finishes.

This works pretty well. First of all, It saves resources by replying on K8S to dynamically allocate them. Moreover, to get clean and isolated environments to work, a jenkins job just need to configure containers on demand.

But things become a bit more boring when people need to build docker image. In this articale, I am listing all ways to implement it and will discuss which one is the best.

Ways to run docker cli in pipeline

Solution 1: Use host node's socket

If the k8s cluster using docker as container runtime, which means every node will have docker. All containers can use host's docker socket then.

Code example:

podTemplate(
yaml: """
apiVersion: v1
kind: Pod
spec:
  volumes:
  - name: docker-sock
    hostPath:
      path: /var/run/docker.dock
      type: File

  containers:
  - name: demo
    image: alpinelinux/docker-cli
    imagePullPolicy: Always
    env:
    - name: DOCKER_HOST
      value: unix:///var/run/docker.dock
    command: ["cat"]
    tty: true
    volumeMounts:
    - name: docker-sock
      mountPath: /var/run/docker.sock
"""
) { node(POD_LABEL) {
    container('demo') {
        sh """
            docker pull busybox
        """
    }
}}

Problems:

• Must have docker as container runtime, which is heavy. Not work if using other runtimes e.g. containerd or cri-o.
• Cannot customize docker version in pipeline.

Solution 2: Run docker as container (DIND)

docker:dind

Docker provides an official docker-in-docker image in docker hub. So we can use it directly in podTemplate from jenkins pipeline.

Note: To use docker-in-docker, the container need to be run as privileged.

Use docker in docker:dind

Here is the mini code need to run docker-in-docker.

podTemplate(
yaml: """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: dind
    image: docker:dind
    securityContext:
      privileged: true
"""
) { node(POD_LABEL) {
    container('dind') {
        sh """
            docker pull busybox
        """
    }
}}

Please note --privileged is a MUST or it will not work.

securityContext:
  privileged: true

By default, the DOCKER_HOST is not set in above dind container. So when command docker... was run, it automatically connect to docker daemon by /var/run/docker.sock.

Here it is running docker command in dind container, which has the docker-daemon running in same container also.

However what if people want to run docker command from other containers in the same pod of dind?

Share dind to other containers - TCP

By default, docker::dind has dockerd-entrypoint.sh as entry-point.

The real command that script finally invoked is like following:

dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2376 --tlsverify --tlscacert /certs/server/ca.pem --tlscert /certs/server/cert.pem --tlskey /certs/server/key.pem

So after starting up, it will be listening on tcp://0.0.0.0:2376, with tls verification enabled by default. Since all containers in pod can communicate like in same host, other containers can easily access docker-daemon by tcp://localhost:2376.

Disable TLS

TLS is boring - to connect to it the client must provide certificate. Here let's keep it simple at the beginning by removing the tls verification.

There are two ways to do it: unset environment variable or overwrite entry-point of dind.

By Environment

DOCKER_TLS_CERTDIR is being used by docker:dind to store tls cert, if set to empty, then tsl would be disabled automatically. I

In following code, the environment is set to empty and also another new container named demo, which uses alpine's docker-cli client image, is added into pod.

podTemplate(
yaml: """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: dind
    image: docker:dind
    securityContext:
      privileged: true
    env:
    - name: DOCKER_TLS_CERTDIR
      value:
  - name: demo
    image: alpinelinux/docker-cli
    env:
    - name: DOCKER_HOST
      value: tcp://localhost:2375
    command: ["cat"]
    tty: true
"""
) { node(POD_LABEL) {
    container('demo') {
        sh """
            # use docker-daemon in dind container
            # in demo there is a docker client only
            docker pull busybox
        """
    }
}}

Note: Here the tcp listening port is changed to 2375 automatically when dind gets up !! So in demo container DOCKER_HOST must be set to tcp://localhost:2375.

By Custom Entry-Point

Remember that default entry-point of docker:dind is dockerd-entrypoint.sh, this script finally call dockerd as a service.

So instead of setting environment, replacing entry-point with dockerd plus customized parameters is easy and more portable.

podTemplate(
yaml: """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: dind
    image: docker:dind
    securityContext:
      privileged: true
    command: ["/bin/sh","-c"]
    args: ["dockerd --host=tcp://0.0.0.0:8989"]
  - name: demo
    image: alpinelinux/docker-cli
    env:
    - name: DOCKER_HOST
      value: tcp://localhost:8989
    command: ["cat"]
    tty: true
"""
) { node(POD_LABEL) {
    container('demo') {
        sh """
            # use docker-daemon in dind container
            # in demo there is a docker client only
            docker pull busybox
        """
    }
}}

Here port 8989 is set as the docker-daemon's listen port by passing --host value to entry-point which has been replaced by dockerd command in code. Following it DOCKER_HOST in demo should be set to tcp://localhost:8989.

Share dind to other containers - Unix Socket

All containers in one pod are in same host so they can share files in the host. By mounting host's directory and put local socket file there, it is easy to have docker daemon accessible to all containers. Basically here two steps needed:

• Mount a directory in host for all containers in host machine
• Set host parameter pointing to socket file in above directory

def shared_mount_point="/dind-only"
def docker_host_sock = "unix://${shared_mount_point}/docker.sock"

podTemplate(cloud: 'kubernetes',
yaml: """
apiVersion: v1
kind: Pod
spec:
  volumes:
  - name: tmp
    emptyDir: {}
    
  containers:
  - name: dind
    image: docker:dind
    env:
    - name: DOCKER_HOST
      value: ${docker_host_sock}
    securityContext:
      privileged: true
    args: ["--host=${docker_host_sock}"]
    volumeMounts:
    - name: tmp
      mountPath: ${shared_mount_point}
      
  - name: demo
    image: alpinelinux/docker-cli
    env:
    - name: DOCKER_HOST
      value: ${docker_host_sock}
    command: ["cat"]
    tty: true
    volumeMounts:
    - name: tmp
      mountPath: ${shared_mount_point}
"""
) { node(POD_LABEL) {
    container('demo') {
        sh """
            docker pull busybox
        """
    }
}}

Here the code makes dind container to listen on unix-domain socket through file /dind-only/docker.sock that shared with demo container through mounting temporary directory on /dind-only .

TCP vs Unix-Domain Socket ?

There is no big difference. But local socket should have better performance because it is more like read/write local file.

Wait Docker-Daemon Up !

So far it looks pretty good. But actually there is a potential problem:

What if the dind is already up but docker-daemon not ready when the pipeline runningdocker ... command?

K8S provides start-up probe for us to solve it !

For example, to make sure docker daemon is listening on tcp port when pipeline runs into docker, just add followings into yaml ofdind container:

- name: dind
    image: docker:dind
  ...
  ...
  startupProbe:
    tcpSocket:
      port: tcp://localhost:8989
    failureThreshold: 10
    initialDelaySeconds: 5
    periodSeconds: 6

For unix-domain socket, just use stat command to check if socket file exists.

def docker_host_sock = "unix://${shared_mount_point}/docker.sock"
...
...
- name: dind
    image: docker:dind
  ...
  ...
  startupProbe:
    exec:
      command:
      - stat
      - ${docker_host_sock_file}

Security !

Running dind in privileged mode is NOT SAFE.

By replacing docker:dind by docker:dind-rootless, the dind container can run as a normal user - this makes it safer.

Note:docker:dind-rootless need --experimental option.

Summary

Docker-In-Docker is easy to run in Jenkins on K8S. The only concern is --privileged may bring security issue. Please be careful. Though in experiment phase, the docker:dind-rootless is strongly recommended at this moment for production environment.