Configuring Alert Forwarding

Describes how to forward alerts via SMTP or Slack and configure the alert policies by updating the alertmanager_config.yaml file.

Prerequisites:
  • Sign in to HPE Ezmeral Unified Analytics Software as an administrator.
  • Have access to a terminal where kubectl is installed to interact with your Kubernetes cluster.

Steps:

To configure alert policies and forward alerts via SMTP or Slack, complete the following steps:
  1. On the Prometheus namespace, run the following command to list secrets:
    kubectl get secrets -n prometheus

  2. In the list of secrets, locate the secret named alertmanager-prometheus-kube-prometheus-alertmanager.

  3. Extract the base64-encoded Alertmanager configuration from the alertmanager_config.yaml file:
    kubectl get secret alertmanager-prometheus-kube-prometheus-alertmanager -n prometheus -o jsonpath="{.data.alertmanager\.yaml}" | base64 -d > alertmanager_config.yaml

  4. Open the alertmanager_config.yaml file in a text editor.

  5. Update the email settings under the email_configs section and the Slack settings under the slack_configs section. You can also configure the additional notification channels as required. To learn more about these settings, see Understanding the Prometheus Alertmanager Configuration File.

  6. Delete the existing Alertmanager configuration secret:
    kubectl delete secret alertmanager-prometheus-kube-prometheus-alertmanager -n prometheus
  7. Create the updated Alertmanager configuration secret:
    kubectl create secret generic alertmanager-prometheus-kube-prometheus-alertmanager -n prometheus --from-file=alertmanager.yaml=alertmanager_config.yaml
  8. Force restart the Alertmanager pod to reload its configuration:
    kubectl delete pod alertmanager-prometheus-kube-prometheus-alertmanager-0 -n prometheus

Results:

You have updated the alertmanager_config.yaml file to forward alerts via SMTP or Slack.