在上一篇文章中,我们主要讨论了使用 operator-sdk
工具编写 Kubernetes Admission Webhook
,并针对我们的自定义资源类型创建了我们的第一个 Mutating Admission Webhook
。但是,这次我们将为 Deployment
、Pod
等核心类型创建另一种类型的&nAdmi
,称为“验证”,而不是自定义资源类型,因为我们可能并不总是有某种自定义资源类型。
一、Kubebuilder
此外,我们将使用“kubebuilder
”来搭建项目模板。为什么?因为改变总是好的。我们在 Trendyol
的平台团队开发 webhook
时正在使用这种方法
Kubebuilder
是一个使用自定义资源定义 (CRD
)构建 Kubernetes API
的框架。
与Ruby on Rails
和SpringBoot
等 Web
开发框架类似,Kubebuilder
提高了速度并降低了开发人员管理的复杂性,以便在 Go
中快速构建和发布 Kubernetes API
。它建立在用于构建核心 Kubernetes API
的规范技术之上,以提供简单的抽象来减少样板文件和繁琐的工作。
正如我之前提到的,我们熟悉 Kubernetes Operator
模式中的这些工具,但这些工具也有助于创建 Kubernetes Admiss
。
确实有很好的文档可以用来了解 kubebuilder
,但是在这篇文章中,我们将重点关注“Webhook for Core Types
”一书的特殊部分。前面我们也提到过,kubebuilder
和 operator-sdk
不支持核心类型的 webhook
脚手架,我们必须使用controler-runtime
中的库来处理它。在控制器运行时有一个例子。
现在,让我们进入Demo部分,这一次使用 kubebuilder
工具来弄脏我们的手。
二、Demo 实践
在这个Demo中,我们将根据我们标记为必需的标签来验证 Pod
。如果Pod
上存在必要的注解,那么我们允许创建 Pod
,如果没有,我们不允许。
先决条件
- Minikube v1.18.1
- Kubebuilder v2.3.2
- kubectl v1.20.5
- go v1.16.3
我将在 macOS
环境中进行此Demo,因此,您可以使用macOS
的包管理器“br
https://brew.sh/ew
”来安装上述所有工具。
三、验证 Pod 的 AdmissionWebhook
由于我们使用的是 operator-sdk
,因此我们可能熟悉 kubebuilder
命令。让我们为项目创建目录
$ mkdir -p pod-validatingwebhook
$ cd pod-validatingwebhook
在使用 kubebuilder
初始化项目模板之前,我们应该首先创建 go
模块:
$ go mod init pod-validatingwebhook
go: creating new go.mod: module pod-validationwebhook
要初始化项目模板,我们应该运行以下命令:
$ kubebuilder init --domain developer.guy --license none --owner "developer-guy"
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/[email protected]
Update go.mod:
$ go mod tidy
Running make:
$ make
go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.2.5
/Users/batuhan.apaydin/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go
Next: define a resource with:
$ kubebuilder create api
# Look at the directory structure, you should see similar to the following
$ tree -L 2 .
.
├── Dockerfile
├── Makefile
├── PROJECT
├── bin
│ └── manager
├── config
│ ├── certmanager
│ ├── default
│ ├── manager
│ ├── prometheus
│ ├── rbac
│ └── webhook
├── go.mod
├── go.sum
├── hack
│ └── boilerplate.go.txt
└── main.go
现在,我们已经准备好为 Pod
创建 ValidatingAdmissionWebhook
,但让我们先创建一个 API
。
$ kubebuilder create api — group core — version v1 — kind Pod — resource=false — controller=false ← be careful here we specified resource and controller creation as false here because we don’t need them.
Writing scaffold for you to edit…
Running make:
$ make
go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.2.5
/Users/batuhan.apaydin/go/bin/controller-gen object:headerFile=”hack/boilerplate.go.txt” paths=”./…”
Error: go [list -e -json -compiled=true -test=false -export=false -deps=true -find=false -tags ignore_autogenerated — ./…]: exit status 1: go: github.com/developer-guy/pod-validationwebhook: package k8s.io/api/core/v1 imported from implicitly required module; to add missing requirements, run:
go get k8s.io/api/core/[email protected]
# if you get the same error above , just get the dependency and run make again like the following:
$ go get k8s.io/api/core/[email protected]
$ make
go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.2.5
/Users/batuhan.apaydin/go/bin/controller-gen object:headerFile=”hack/boilerplate.go.txt” paths=”./…”
go fmt ./…
go vet ./…
go build -o bin/manager main.go
之后,创建 ValidatingAdmissionWebhook
。
# Remember here, programmatic-validation is for ValidatingAdmissionWebhook, and the --default for the Mutating one.
$ kubebuilder create webhook --group core --version v1 --kind Pod --programmatic-validation
Writing scaffold for you to edit...
api/v1/pod_webhook.go
打开项目后,您应该注意到main.go
和“api/v1/pod_webhook.go
”文件中存在某种编译时错误。


因此,为了解决这个问题,我们将遵循我上面提到的 kubebuilder
书中的“Webhook for Core Types
”部分。
让我们用下面的代码来改变pod_webhook.go
的内容,然后,我们将稍微解释一下代码。
package v1
import (
"context"
"fmt"
"net/http"
corev1 "k8s.io/api/core/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/webhook/admission"
)
// +kubebuilder:webhook:verbs=create;update,path=/validate-core-v1-pod,mutating=false,failurePolicy=fail,groups=core,resources=pods,versions=v1,name=vpod.kb.io
// podValidator validates Pods
type podValidator struct {
Client client.Client
decoder *admission.Decoder
}
func NewPodValidator(c client.Client) admission.Handler {
return &podValidator{Client: c}
}
// podValidator admits a pod if a specific annotation exists.
func (v *podValidator) Handle(ctx context.Context, req admission.Request) admission.Response {
pod := &corev1.Pod{}
err := v.decoder.Decode(req, pod)
if err != nil {
return admission.Errored(http.StatusBadRequest, err)
}
key := "example-mutating-admission-webhook"
anno, found := pod.Annotations[key]
if !found {
return admission.Denied(fmt.Sprintf("missing annotation %s", key))
}
if anno != "foo" {
return admission.Denied(fmt.Sprintf("annotation %s did not have value %q", key, "foo"))
}
return admission.Allowed("")
}
// podValidator implements admission.DecoderInjector.
// A decoder will be automatically injected.
// InjectDecoder injects the decoder.
func (v *podValidator) InjectDecoder(d *admission.Decoder) error {
v.decoder = d
return nil
}
我们还需要做一件事。打开 main.go
文件并将 55
和 59
之间的行更改为以下内容以注册我们的 webhook
服务器。
pv := v1.NewPodValidator(mgr.GetClient())
mgr.GetWebhookServer().Register("/validate-core-v1-pod", &webhook.Admission{Handler: pv})
现在,我们从代码的角度准备好了。让我们继续部署 webhook
。在构建和推送镜像之前,请从 Dockerfile
中删除#L15
处的“COPY controllers/controllers/
”,因为我们在这里没有使用控制器。
$ make docker-build docker-push IMG=devopps/pod-validationwebhook:v1
go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.2.5
/Users/batuhan.apaydin/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/Users/batuhan.apaydin/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go test ./... -coverprofile cover.out
? github.com/developer-guy/pod-validationwebhook [no test files]
? github.com/developer-guy/pod-validationwebhook/api/v1 [no test files]
docker build . -t devopps/pod-validationwebhook:v1
[+] Building 24.2s (16/16) FINISHED
...
docker push devopps/pod-validationwebhook:v1
The push refers to repository [docker.io/devopps/pod-validationwebhook]
45ed4c050068: Pushed
417cb9b79ade: Pushed
v1: digest: sha256:63f3aaf383f34af74027802316f2c58ee750076c2b03046775ad39be818a80c3 size: 739
一旦镜像推送成功,我们需要在进入部署部分之前做几件事,请删除config/default
文件夹下的3
和 9 webhookcainjection_patch.yam
l
之间的行,因为我们没有在这里创建 MutatingAdmissionWebhook
,并删除来自同一目录下的kustomization.yaml
的第 16
行。另外,从config/rbac
目录下的kustomization.yaml
中删除第 2
行,一切就绪。
我们应该在这里做的最后一件事是启用和部署证书管理器,为了做到这一点,我们应该通过取消注释标记的部分和注释来编辑“config/default/kustomization.yaml
” 文件。[WEBHOOK]和[CERTMANAGER]
我们知道 kubebuilder
也使用 cert-manager
来管理我们的 webhook
的 TLS
管理,所以,我们应该先在我们的集群中安装一个 cert-manager
,让我们这样做。
$ minikube start
minikube v1.18.1 on Darwin 10.15.7
✨ Using the virtualbox driver based on user configuration
Starting control plane node minikube in cluster minikube
Creating virtualbox VM (CPUs=3, Memory=8192MB, Disk=20000MB) …
Preparing Kubernetes v1.20.2 on Docker 20.10.3 …
▪ Generating certificates and keys …
▪ Booting up control plane …
▪ Configuring RBAC rules …
Verifying Kubernetes components…
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v4
Enabled addons: storage-provisioner, default-storageclass
Done! kubectl is now configured to use “minikube” cluster and “default” namespace by default
$ helm repo add jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories
$ helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--version v1.2.0 \
--create-namespace \
--set installCRDs=true
NAME: cert-manager
LAST DEPLOYED: Mon Apr 5 22:53:17 2021
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager has been deployed successfully!
In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).
More information on the different types of issuers and how to configure them
can be found in our documentation:
https://cert-manager.io/docs/configuration/
For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:
https://cert-manager.io/docs/usage/ingress/
$ kubectl get pods --namespace cert-manager --watch
NAME READY STATUS RESTARTS AGE
cert-manager-85f9bbcd97-pqmnv 1/1 Running 0 3m46s
cert-manager-cainjector-74459fcc56-nnpjg 1/1 Running 0 3m46s
cert-manager-webhook-57d97ccc67-8vjff 1/1 Running 0 3m46s
现在,我们已准备好部署 webhook
。
$ make deploy IMG=devopps/pod-validationwebhook:v1
go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.2.5
/Users/batuhan.apaydin/go/bin/controller-gen “crd:trivialVersions=true” rbac:roleName=manager-role webhook paths=”./…” output:crd:artifacts:config=config/crd/bases
cd config/manager && kustomize edit set image controller=devopps/pod-validationwebhook:v1
kustomize build config/default | kubectl apply -f -
namespace/pod-validatingwebhook-system created
role.rbac.authorization.k8s.io/pod-validatingwebhook-leader-election-role created
clusterrole.rbac.authorization.k8s.io/pod-validatingwebhook-proxy-role created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/pod-validatingwebhook-metrics-reader created
rolebinding.rbac.authorization.k8s.io/pod-validatingwebhook-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/pod-validatingwebhook-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/pod-validatingwebhook-proxy-rolebinding created
service/pod-validatingwebhook-controller-manager-metrics-service created
service/pod-validatingwebhook-webhook-service created
deployment.apps/pod-validatingwebhook-controller-manager created
certificate.cert-manager.io/pod-validatingwebhook-serving-cert created
issuer.cert-manager.io/pod-validatingwebhook-selfsigned-issuer created
Warning: admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 ValidatingWebhookConfiguration
validatingwebhookconfiguration.admissionregistration.k8s.io/pod-validatingwebhook-validating-webhook-configuration created
耶!!!似乎一切正常,但让我们通过创建没有必需注释的 Pod
来测试 webhook
。
# test with invalid Pod
$ kubectl run --generator=run-pod/v1 nginx --image=nginx
Found existing alias for “kubectl”. You should use: “k”
Alias tip: k run — generator=run-pod/v1 nginx — image=nginx
Flag — generator has been deprecated, has no effect and will be removed in the future.
Error from server (missing annotation example-mutating-admission-webhook): admission webhook “vpod.kb.io” denied the request: missing annotation example-mutating-admission-webhook
# test with valid Pod
$ kubectl run --generator=run-pod/v1 nginx --image=nginx --overrides='{ "apiVersion": "v1", "metadata": {"annotations": { "required-label":"foo" } } }'
Flag --generator has been deprecated, has no effect and will be removed in the future.
pod/nginx created
现在,我们证明一切都如我们预期的那样工作
参考文献
https://medium.com/trendyol-tech/getting-started-to-write-your-first-kubernetes-admission-webhook-part-2-48d0b0b1780e
你好,能否把完整的代码共享一下,按照你的步骤,总是部署不成功,最后提示如下错误
Error from server (InternalError): error when creating “deploy.yaml”: Internal error occurred: failed calling webhook “vpod.kb.io”: Post “https://pod-validatingwebhook-webhook-service.pod-validatingwebhook-system.svc:443/validate-core-v1-pod?timeout=10s”: EOF