Useful Troubleshooting Commands
The following Kubernetes commands can help you gather information about the resources used byData Fabric for Kubernetes:
kubectl describe <resourcetype> <resource> -n <namespace>
kubectl get <resourcetype> <resource> -n <namespace> -o yaml
kubectl logs <pod-name> -n <namespace>
journalctl -u kubelet -r
(on the relevant node)
kubectl describe command
In this example, the kubectl describe
command displays information about
the mapr-kdfprovisioner-5dff68656-ln6vh
Pod. Note that the kubectl
describe
output includes an event section.
kubectl describe pod mapr-kdfprovisioner-5dff68656-ln6vh -n mapr-system
Name: mapr-kdfprovisioner-5dff68656-ln6vh
Namespace: mapr-system
Node: qa101-139/10.10.101.139
Start Time: Fri, 09 Feb 2018 12:58:36 -0800
Labels: app=mapr-kdfprovisioner
pod-template-hash=189924212
Annotations: openshift.ix/scc-maprkdf-scc
Status: Running
IP: 172.17.0.3
. .
. .
. .
Node-Selectors: <none>
Tolerations: <none>
Events:
Type Reason Age From Message
---- ----- --- ---- ---------
Normal Scheduled 8m default-scheduler Successfully assigned mapr-kdfprovisioner-5dff68656-ln6vh to qa101-139
Normal SuccessfulMountVolume 8m kubelet, qa101-139 MountVolume,SetUp succeeded for volume "logs"
Normal SuccessfulMountVolume 8m kubelet, qa101-139 MountVolume,SetUp succeeded for volume "timezone"
Normal SuccessfulMountVolume 8m kubelet, qa101-139 MountVolume,SetUp succeeded for volume "maprkdf-token-drqtt"
Normal Pulling 8m kubelet, qa101-139 pulling image "maprtech/kdf-provisioner:1.0.0.006_centos7"
Normal Pulled 8m kubelet, qa101-139 Successfully pulled image "maprtech/kdf-provisioner:1.0.0.006_centos7"
Normal Created 8m kubelet, qa101-139 Created container
Normal Started 8m kubelet, qa101-139 Started container
kubectl get command
In this example, the kubectl get
returns the .yaml
parameters for the test-secure-provisioner86
Pod:
kubectl get pods test-secure-provisioner86 -n mapr-examples -o yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: 2018-02-09T00:42:06Z
name: test-secure-provisioner86
namespace: mapr-examples
resourceVersion: "721689"
selfLink: /api/v1/namespaces/mapr-examples/pods/test-secure-provisioner86
uid: 0dd21274-0d32-11e8-bdd6-84a9c4fbf7cb
spec:
containers:
- args:
- sleep
- "1000000"
image: busybox
imagePullPolicy: Always
name: busybox
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: file
volumeMounts:
- mountPath: /dynvolume
name: maprfs-pvc
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-zpv69
readOnly: true
dnsPolicy: ClusterFirst
nodeName: qa108-165.qa.lab
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key:node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
-name: maprfs-pvc
Running the kubectl get
command without the -o yaml
parameter generates less output:
kubectl get pods test-secure-provisioner86 -n mapr-examples
NAME READY STATUS RESTARTS AGE
test-secure-provisioner86 1/1 Running 0 14m
kubectl logs command
In this example, the kubectl logs
command returns logged output for the
mapr-kdfprovisioner-5dff68656-ln6vh
Pod:
kubectl logs mapr-kdfprovisioner-5dff68656-ln6vh -n mapr-system
I0209 12:58:39.956822 1 controller.go:407] Starting provisioner controller 013d58b3-0ddc-11e8-b0dd-0242acl10003!
journalctl -u command
In this example, the journalctl
command returns events for the kubelet
service for the node:
journalctl -u kubelet -r
-- Logs begin at Thu 2017-12-28 06:24:47 PST, end at Thu 2018-02-08 17:01:49 PST. --
Feb 08 17:01:49 k8s-master kubelet[26521]: E0206 17:01:49,047595 26521 dns.go:121] Search Line limits were exceeded, some search paths have been omitted, the applied search line
Feb 08 17:01:45 k8s-master kubelet[26521]: E0206 17:01:45,396253 26521 dns.go:180] CheckLimitsForResolvConf: Resolv,conf file '/etc/resolve.conf' contains search line consisting
Feb 08 17:01:15 k8s-master kubelet[26521]: E0206 17:01:15,396023 26521 dns.go:100] CheckLimitsForResolvConf: Resolv,conf file '/etc/resolve.conf' contains search line consisting
Feb 08 17:00:48 k8s-master kubelet[26521]: E0206 17:00:48,047555 26521 dns.go:121] Search Line limits were exceeded, some search paths have been omitted, the applied search line
. .
. .
. .