Troubleshooting OPA Gatekeeper

Describes how to disable Open Policy Agent (OPA) Gatekeeper on a Kubernetes cluster for troubleshooting purposes, and re-enable OPA Gatekeeper after any issues have been corrected.

As part of OPA Gatekeeper deployment, HPE Ezmeral Runtime Enterprise creates an admission webhook. This webhook intercepts requests to the API server, and returns a response to the API server. Depending on the response received, and the policies currently in place, the API server decides whether the request can be fulfilled.

If OPA Gatekeeper is preventing a cluster from operating correctly, this admission webhook can be disabled to remove all OPA Gatekeeper admission checks while the issue is being fixed.

See: https://open-policy-agent.github.io/gatekeeper/website/docs/emergency/

Proceed as follows:
  1. Save the definition of the original webhook present in the system with the following command:
    kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io gatekeeper-validating-webhook-configuration -o yaml > webhook.yaml
  2. To disable the admission webhook, enter the following command:
    kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io gatekeeper-validating-webhook-configuration
    NOTE
    While the admission webhook is disabled, OPA Gatekeeper will no longer impose policies in the cluster. To reimpose policies, re-enable OPA Gatkeeper.

  3. While the admission webhook is disabled, you can correct any issues you may be experiencing with OPA Gatekeeper.

    For information on debugging OPA Gatekeeper, see: https://open-policy-agent.github.io/gatekeeper/website/docs/debug/.

    For information on OPA Gatekeeper issues, see: https://github.com/open-policy-agent/gatekeeper/issues.

  4. After you have fixed any issues, re-apply the admission webhook with the following command:
    kubectl apply -f webhook.yaml
NOTE
If you did not save the original webhook before deleting it, re-enable OPA Gatekeeper as follows:
Apply the following YAML manifest on the Kubernetes cluster:
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  creationTimestamp: null
  labels:
    gatekeeper.sh/system: "yes"
  name: gatekeeper-validating-webhook-configuration
webhooks:
- clientConfig:
    caBundle: Cg==
    service:
      name: gatekeeper-webhook-service
      namespace: gatekeeper-system
      path: /v1/admit
  failurePolicy: Ignore
  name: validation.gatekeeper.sh
  namespaceSelector:
    matchExpressions:
    - key: admission.gatekeeper.sh/ignore
      operator: DoesNotExist
  rules:
  - apiGroups:
    - '*'
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - pods
  sideEffects: None
  timeoutSeconds: 3
- clientConfig:
    caBundle: Cg==
    service:
      name: gatekeeper-webhook-service
      namespace: gatekeeper-system
      path: /v1/admitlabel
  failurePolicy: Fail
  name: check-ignore-label.gatekeeper.sh
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - namespaces
  sideEffects: None
  timeoutSeconds: 3
.