Downloading Kubernetes Usage Details

Platform Administrators can download scripts to view Kubernetes usage details in HPE Ezmeral Runtime Enterprise.

The following two scripts allow Platform Administrators to download Kubernetes usage details:

  • k8susage.py: Collects the usage metrics from Kubernetes clusters and stores the results in a comma-delimited (.csv) file. Historical usages are collected for the specified time period and aggregated over the specified time interval (aggregation interval) or the current usage (over a 2 minute interval) is collected. Individual pod metrics and pod counts are summed up for each aggregation interval following the hierarchy Pod > Node > Namespace > All namespaces. This script collects the following metrics:
    • CPU cores limits and requests
    • Memory limits and requests
    • Ephemeral storage capacity and usage
    • Number of running and pending pods
    • Tenant storage usage (now option only)
  • k8scsv.py: Collect utilization metrics from Kubernetes clusters and stores the results in a comma-delimited (.csv) file. Historical usages are collected for the specified time period and aggregated over the specified time interval (aggregation interval) or the current usage (over a 2-minute interval) is collected. Individual pod metrics, except the CPU and memory limit percentages are summed up for each aggregation interval following the hierarchy Pod > Node > Namespace > All Namespaces. For the CPU and memory limit metrics, the per aggregation interval averages are computed for the Node, Namespace, and All Namespaces. This script collects the following metrics:
    • CPU
      • Pod used Nanocores
      • Pod usage as a percentage of the total node CPU
      • Pod usage as a percentage of the defined limit for the pod containers (or total node CPU if unlimited)
    • Memory
      • Pod total memory usage
      • Pod memory usage as a percentage of the total node allocable memory
      • Pod memory usage as a percentage of the defined limit for the pod containers (or total node allocable memory if unlimited)
      • Pod total network received (Rx) and transmitted (Tx) bytes.
      • Pod network received (Rx) and transmitted (Tx) bytes per aggregation interval (not for now option)

Location

These scripts are located on the Controller host, in the following directory: 
/opt/bluedata/common-install/scripts/monitoring

Usage

You must have Platform Administrator privileges to execute these scripts.

  • k8susage.py:

    Either:

    python k8susage.py -c <controller> -f <credentials_file> -s <start_time> -e <end_time>

    or 

    python k8susage.py -c <controller> -f <credentials_file> -n

    For example:

    python k8susage.py -f cred.json -c 10.1.32.120 -s 2021/11/01-08:00:00 -e 2021/11/10-10:11:45

  • k8scsv.py:

    Either:

    python k8scsv.py -c <controller> -f <credentials_file> -s <start_time> -e <end_time>

    or

    python k8scsv.py -c <controller> -f <credentials_file> -n

Where:

  • <controller> is one of the following:

    • The IP address of the Controller host
    • Cluster IP address (if platform HA is enabled)
    • IP address of a Gateway host.

  • <credentials_file> is the path to a JSON file that contains the username and password to use to connect to the Controller host.

    This JSON file stores the username under the key user and the password under the key password.

    For example:

    { "user": "MyUserName", "password": "MyPassword123" }

Options

Flag Description Required
-h, --help Show help message and exit.
-a, --all-namespaces Include all namespaces and not just tenant associated namespaces.
-c CONTROLLER, --controller=CONTROLLER Controller IP address. Yes
-e END, --end=END End time as either YYYY/mm/dd-HH:MM:SS or number of hours in the past. Yes, with the -s option.
-f CREDFILE, --file=CREDFILE Credentials file. Yes
-i INTERVAL, --interval=INTERVAL Interval (e.g. 10m for 10 minutes, 1h for 1 hour, etc., min: 2m).
-k K8SCLUSTER, --k8scluster=K8SCLUSTER Filter by this Kubernetes cluster name.
-l LOGFILE, --logfile=LOGFILE Log file (default is ./script_name.log)
-n, --now Collect latest usage over a 2-minute interval. Yes, without the -s -e options.
-o OUTPUTFILE, --outputfile=OUTPUTFILE Output file (default is ./script_name.csv).
-p, --list-pods List pods in the .csv file.
-q, --quiet No output to console.
-s START, --start=START Start time as either YYYY/mm/dd-HH:MM:SS or number of hours in the past. Yes, with the -e option.
-t TENANT, --tenant=TENANT Filter by this tenant name.
-x, --https Connect to the Controller host via https.

CSV File Columns

The .csv files include the following columns:

  • k8susage.csv:

    • Time period query:

      Time Window Start | Time Window End | Selector | Cluster Name | Tenant Name | Namespace | Node Name | Pod Name | CPU Limit Cores | CPU Request Cores | Memory Limit (B) | Memory Request (B) | Ephemeral Storage Capacity (B) | Ephemeral Storage Available (B) | Ephemeral Storage Used (B) | Running Pods | Pending Pods

    • Now query:

      Timestamp | Selector | Cluster Name | Tenant Name | Namespace | Node Name | Pod Name | CPU Limit Cores | CPU Request Cores | Memory Limit (B) | Memory Request (B) | Ephemeral Storage Capacity (B) | Ephemeral Storage Available (B) | Ephemeral Storage Used (B) | Tenant Storage Used (MB) | Running Pods | Pending Pods | Quota CPU Cores | Quota Memory (GB) | Quota Ephemeral Storage (GB) | Quota Tenant Storage (GB) | Cluster CPU Cores | Cluster Memory (GB) | Cluster Ephemeral Storage (GB) | Cluster Tenant Storage (GB)
  • k8scsv.csv:

    • Time period query:

      Time Window Start | Time Window End | Selector | Cluster Name | Tenant Name | Namespace | Node Name | Pod Name | CPU Usage Nanocores | CPU Usage Node (%) | CPU Usage Limit (%) Memory Usage (B) | Memory Usage Node (%) Memory Usage Limit (%) | Network Rx (B) | Network Tx (B) | Network Rx Int (B) | Network Tx Int (B)

    • Now query:

      Timestamp | Selector | Cluster Name | Tenant Name | Namespace | Node Name | Pod Name CPU Usage Nanocores | CPU Usage Node (%) | CPU Usage Limit (%) Memory Usage (B) | Memory Usage Node (%) Memory Usage Limit (%) | Network Rx (B) | Network Tx (B)

In the .csv files, The Selector column specifies the Row data:

  • Total: Total sum/average for the period.
  • Tenant: Total sum/average for the tenant/namespace.
  • Node: Total sum/average for the node.
  • Pod: Individual pod.

Limitations

  • The scripts attempt to match pods in tenant-associated namespaces with the same name on diļ¬€erent clusters to the correct tenant based on the cluster nodes assigned to the cluster during the aggregation interval. Pending pods that are not assigned to a node cannot be matched.
  • The scripts query the change history of the clusters for added/deleted hosts and try to match nodes found in the aggregation data to a specific cluster. Deleted hosts cannot be matched.
  • Historical data for deleted tenants/namespaces, nodes and pods can be retrieved using the --all-namespaces option, but cannot be matched to clusters or tenants.