HPE Kubernetes Cluster Troubleshooting

Troubleshooting Kubernetes clusters that are running the Hewlett Packard Enterprise distribution of Kubernetes can involve examining the .service files, environment variables, and using journald to examine logs.

This article contains troubleshooting information that is specific to Kubernetes clusters that are running the Hewlett Packard Enterprise distribution of Kubernetes, which uses the containerd runtime.

Service Files

Each Kubernetes service has its own .service file. ​

  • /etc/systemd/system/kubelet.service​
  • /etc/systemd/system/kube-proxy.service​
  • /etc/systemd/system/kube-apiserver.service​

Environment Variables

Environment variables for some of the services that ezctl configures are saved at the following location: 
/opt/ezkube/bootstrap/systemd​

The environment variables are editable, but changes might be overwritten when updating Kubernetes.

Logs

Journald manages the logs for all the services.​

  • By default, all log messages are logged to: /var/log/messages​
  • To get the log for a service, enter the appropriate journalctl command.
    For example, the following command jumps to the end of the log of the kube-apiserver unit:
    journalctl -e -u kube-apiserver​

Kubernetes Audit Information

The Kubernetes audit policy is defined in the following file:
/opt/ezkube/k8s-audit-policy.yaml​
The Kubernetes audit log is contained in the following file:
/var/log/ezkube/audit/k8s-audit.log​

Kubernetes Node Bring-Up

The following configuration file is used when orchestrating the Kubernetes node bring-up: 
/var/log/bluedata/install/ezctl-config-<date>.yaml​

The following logs are related to Kubernetes node bring-up. The contents of the logs are also logged to the standard /var/log/bluedata/install/k8s_cluster_* log files:

  • /var/log/bluedata/install/ezctl-<date>.log​
  • /var/log/bluedata/install/ezctl_pod_deletes.log​