kubebuilder和code-generator使用分享

kubebuilder和k8s.io/code-generator类似，是一个码生成工具，用于为你的CRD生成kubernetes-style API实现。目前个人使用Kubebuilder生成CRD和manifests yaml，再使用code-generator生成informers、listers、clientsets。

下载kubebuilder

# download kubebuilder and install locally.
curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

本文中使用的Kubebuilder版本为release-3，code-generator版本为v0.19.2

使用kubebuilder创建项目

go  mod init example
// --skip-go-version-check 跳过Go版本校验
kubebuilder  init --domain lgy.com --skip-go-version-check
// 设置后生成的api上层会自动创建Group名文件夹
kubebuilder edit --multigroup=true
kubebuilder create api --version v1 --kind Example--group serving

效果如下：

[root@master example]# go mod init example
go: creating new go.mod: module example
[root@master example]# kubebuilder  init --domain lgy.com --skip-go-version-check
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.7.2
Update dependencies:
$ go mod tidy
Next: define a resource with:
$ kubebuilder create api
[root@master example]# kubebuilder edit --multigroup=true
[root@master example]# kubebuilder create api --version v1 --kind Example --group serving
Create Resource [y/n]
y
Create Controller [y/n]
n
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
apis/serving/v1/example_types.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.4.1
go get: installing executables with 'go get' in module mode is deprecated.
    To adjust and download dependencies of the current module, use 'go get -d'.
    To install using requirements of the current module, use 'go install'.
    To install ignoring the current module, use 'go install' with a version,
    like 'go install example.com/cmd@latest'.
    For more information, see https://golang.org/doc/go-get-install-deprecation
    or run 'go help get' or 'go help install'.
go get: added github.com/fatih/color v1.7.0
go get: added github.com/gobuffalo/flect v0.2.0
go get: added github.com/gogo/protobuf v1.3.1
go get: added github.com/google/gofuzz v1.1.0
go get: added github.com/inconshreveable/mousetrap v1.0.0
go get: added github.com/json-iterator/go v1.1.8
go get: added github.com/mattn/go-colorable v0.1.2
go get: added github.com/mattn/go-isatty v0.0.8
go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go get: added github.com/modern-go/reflect2 v1.0.1
go get: added github.com/spf13/cobra v1.0.0
go get: added github.com/spf13/pflag v1.0.5
go get: added golang.org/x/mod v0.2.0
go get: added golang.org/x/net v0.0.0-20200226121028-0de0cce0169b
go get: added golang.org/x/sys v0.0.0-20191022100944-742c48ecaeb7
go get: added golang.org/x/text v0.3.2
go get: added golang.org/x/tools v0.0.0-20200616195046-dc31b401abb5
go get: added golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
go get: added gopkg.in/inf.v0 v0.9.1
go get: added gopkg.in/yaml.v2 v2.2.8
go get: added gopkg.in/yaml.v3 v3.0.0-20190905181640-827449938966
go get: added k8s.io/api v0.18.2
go get: added k8s.io/apiextensions-apiserver v0.18.2
go get: added k8s.io/apimachinery v0.18.2
go get: added k8s.io/klog v1.0.0
go get: added k8s.io/utils v0.0.0-20200324210504-a9aa75ae1b89
go get: added sigs.k8s.io/controller-tools v0.4.1
go get: added sigs.k8s.io/structured-merge-diff/v3 v3.0.0
go get: added sigs.k8s.io/yaml v1.2.0
/opt/lgy/example/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
[root@master example]# 
[root@master example]# 
[root@master example]# tree
.
├── apis
│   └── serving
│       └── v1
│           ├── example_types.go
│           ├── groupversion_info.go
│           └── zz_generated.deepcopy.go
├── bin
│   └── controller-gen
├── config
│   ├── crd
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_examples.yaml
│   │       └── webhook_in_examples.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── example_editor_role.yaml
│   │   ├── example_viewer_role.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── role_binding.yaml
│   │   └── service_account.yaml
│   └── samples
│       └── serving_v1_example.yaml
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
└── PROJECT

13 directories, 35 files

添加文件apis/serving/v1/rbac.go，这个文件用生成RBAC manifests：
注意修改kubebuilder:rbac:groups和resources为你的

// +kubebuilder:rbac:groups=serving.lgy.com,resources=examples,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=serving.lgy.com,resources=examples/status,verbs=get;update;patch
 
package v1

生成CRD manifests

[root@master example]# make manifests
/opt/lgy/example/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
[root@master example]#

使用code-generator

1）准备脚本

在hack目录下添加以下文件：

[root@master hack]# tree
.
├── boilerplate.go.txt
├── tools.go
├── update-codegen.sh
└── verify-codegen.sh

0 directories, 4 files

tools.go

// +build tools
package tools

update-codegen.sh
注意：
MODULE和go.mod保持一致
API_PKG=apis，和apis目录保持一致
OUTPUT_PKG=generated/serving，生成Resource时指定的group一样
GROUP_VERSION=serving:v1和生成Resource时指定的group version对应

#!/usr/bin/env bash

set -o errexit
set -o nounset
set -o pipefail

# corresponding to go mod init <module>
MODULE=example
# api package
APIS_PKG=apis
# generated output package
OUTPUT_PKG=generated/serving
# group-version such as foo:v1alpha1
GROUP_VERSION=serving:v1

SCRIPT_ROOT=$(dirname "${BASH_SOURCE[0]}")/..
CODEGEN_PKG=${CODEGEN_PKG:-$(cd "${SCRIPT_ROOT}"; ls -d -1 ./vendor/k8s.io/code-generator 2>/dev/null || echo ../code-generator)}

# generate the code with:
# --output-base    because this script should also be able to run inside the vendor dir of
#                  k8s.io/kubernetes. The output-base is needed for the generators to output into the vendor dir
#                  instead of the $GOPATH directly. For normal projects this can be dropped.
bash "${CODEGEN_PKG}"/generate-groups.sh "client,lister,informer" \
  ${MODULE}/${OUTPUT_PKG} ${MODULE}/${APIS_PKG} \
  ${GROUP_VERSION} \
  --go-header-file "${SCRIPT_ROOT}"/hack/boilerplate.go.txt \
  --output-base "${SCRIPT_ROOT}"

verify-codegen.sh
注意：
OUTPUT_PKG=generated/serving，生成Resource时指定的group一样
MODULE是domain和go.mod的结合

#!/usr/bin/env bash

set -o errexit
set -o nounset
set -o pipefail

OUTPUT_PKG=generated/serving
MODULE=lgy.com/example

SCRIPT_ROOT=$(dirname "${BASH_SOURCE[0]}")/..

DIFFROOT="${SCRIPT_ROOT}/${OUTPUT_PKG}"
TMP_DIFFROOT="${SCRIPT_ROOT}/_tmp/${OUTPUT_PKG}"
_tmp="${SCRIPT_ROOT}/_tmp"

cleanup() {
  rm -rf "${_tmp}"
}
trap "cleanup" EXIT SIGINT

cleanup

mkdir -p "${TMP_DIFFROOT}"
cp -a "${DIFFROOT}"/* "${TMP_DIFFROOT}"

"${SCRIPT_ROOT}/hack/update-codegen.sh"
echo "copying generated ${SCRIPT_ROOT}/${MODULE}/${OUTPUT_PKG} to ${DIFFROOT}"
cp -r "${SCRIPT_ROOT}/${MODULE}/${OUTPUT_PKG}"/* "${DIFFROOT}"

echo "diffing ${DIFFROOT} against freshly generated codegen"
ret=0
diff -Naupr "${DIFFROOT}" "${TMP_DIFFROOT}" || ret=$?
cp -a "${TMP_DIFFROOT}"/* "${DIFFROOT}"
if [[ $ret -eq 0 ]]
then
  echo "${DIFFROOT} up to date."
else
  echo "${DIFFROOT} is out of date. Please run hack/update-codegen.sh"
  exit 1
fi

修改文件权限：

[root@master hack]# chmod 755 update-codegen.sh 
[root@master hack]# chmod 755 verify-codegen.sh 
[root@master hack]#

2）添加注释

修改example_types.go文件，添加上tag // +genclient，这个注释内容要加在Example结构体上面，

//+genclient
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status

// Example is the Schema for the examples API
type Example struct {
        metav1.TypeMeta   `json:",inline"`

新建apis/serving/v1/doc.go
// +groupName更换成自己的

// +groupName=serving.lgy.com
package v1

新建apis/serving/v1/register.go，code generator生成的代码需要用到它

package v1
import (
    "k8s.io/apimachinery/pkg/runtime/schema"
)
 
// SchemeGroupVersion is group version used to register these objects.
var SchemeGroupVersion = GroupVersion
 
func Resource(resource string) schema.GroupResource {
    return SchemeGroupVersion.WithResource(resource).GroupResource()
}

3）更新依赖

新增 k8s.io/code-generator，需和 k8s.io/client-go版本一致

module example

go 1.15

require (
        k8s.io/apimachinery v0.19.2
        k8s.io/client-go v0.19.2
        k8s.io/code-generator v0.19.2
        sigs.k8s.io/controller-runtime v0.7.2
)

然后使用vend工具更新vendor，注意这里不要用go mod vendor命令来更新，因为k8s.io/code-generator这个依赖在项目中并没有真正被引用过，所以使用go mod vendor是无法将这个依赖更新到vendor中，要借助第三方工具vend来实现。https://github.com/nomad-software/vend
使用命令go getgithub.com/nomad-software/vend来安装，然后再项目根目录下执行vend命令
然后给generate-groups.sh添加可执行权限：

[root@master example]# chmod 755 vendor/k8s.io/code-generator/generate-groups.sh
[root@master example]#

执行hack/update-codegen.sh：

[root@master example]# ./hack/update-codegen.sh 
Generating clientset for serving:v1 at example/generated/serving/clientset
Generating listers for serving:v1 at example/generated/serving/listers
Generating informers for serving:v1 at example/generated/serving/informers
[root@master example]#

当前目录下生成example文件夹

[root@master example]# pwd
/opt/lgy/example/example
[root@master example]# tree -CL 6
.
└── generated
    └── serving
        ├── clientset
        │   └── versioned
        │       ├── clientset.go
        │       ├── doc.go
        │       ├── fake
        │       │   ├── clientset_generated.go
        │       │   ├── doc.go
        │       │   └── register.go
        │       ├── scheme
        │       │   ├── doc.go
        │       │   └── register.go
        │       └── typed
        │           └── serving
        ├── informers
        │   └── externalversions
        │       ├── factory.go
        │       ├── generic.go
        │       ├── internalinterfaces
        │       │   └── factory_interfaces.go
        │       └── serving
        │           ├── interface.go
        │           └── v1
        └── listers
            └── serving
                └── v1
                    ├── example.go
                    └── expansion_generated.go

16 directories, 13 files