Configuring Templates and Filtering Alerts

Describes the process of configuring templates and filtering alerts in Alertmanager. You can follow these steps to customize the content and structure of email notifications generated by Alertmanager.

Prerequisites:
  • Sign in to HPE Ezmeral Unified Analytics Software as an administrator.
  • Have access to a terminal where kubectl is installed to interact with your Kubernetes cluster.

Steps:

To customize the content and structure of email notifications generated by Alertmanager, perform the following steps:

  1. Create two separate template files: custom_mail_subject.tmpl and custom_mail_html.tmpl. You will use custom_mail_subject.tmpl to create a template of the subject and custom_mail_html.tmpl to edit the html.

    In custom_mail_subject.tmpl, add the following content:
    [Alerting] {{ .CommonLabels.alertname }}
    In custom_mail_html.tmpl, add the following content:
    <html>
<body>
  <h3>{{ .CommonLabels.alertname }}</h3>
  <p>{{ .CommonAnnotations.description }}</p>
</body>
</html>
  2. Create a ConfigMap in the prometheus namespace to store the templates:
    kubectl create configmap alertmanager-templates -n prometheus \
  --from-file=custom_mail_subject.tmpl \
  --from-file=custom_mail_html.tmpl
  3. Modify the prometheus-kube-prometheus-alertmanager Custom Resource (CR) in the prometheus namespace to mount the ConfigMap:
    kubectl edit alertmanager prometheus-kube-prometheus-alertmanager -n prometheus -o yaml
    In the spec section, add the following:
    spec:
  volumes:
    - name: alertmanager-templates
      configMap:
        name: alertmanager-templates
  containers:
    - name: alertmanager
      volumeMounts:
        - name: alertmanager-templates
          mountPath: /etc/alertmanager/templates
  4. Get the Secret containing the Alertmanager configuration:
    kubectl get secret alertmanager-prometheus-kube-prometheus-alertmanager -n prometheus -o yaml
  5. Extract the base64-encoded Alertmanager configuration from the alertmanager.yaml file:
    echo '<base64_config_data>' | base64 -d > alertmanager.yaml
  6. Open the alertmanager.yaml file in a text editor.
  7. Add the template reference in alertmanager.yaml:
    templates:
  - '/etc/alertmanager/templates/*.tmpl'
  8. Add or update the email_configs section in alertmanager.yaml:
    receivers:
- name: 'email_receiver'
  email_configs:
  - to: 'email@xyz.com'
    html: '{{ template "custom_mail_html.tmpl" . }}'
    headers:
      subject: '{{ template "custom_mail_subject.tmpl" . }}'
    NOTE
    This template is applied to alerts that are not specifically routed elsewhere.
  9. Add additional receivers and routes in alertmanager.yaml:
    global:
  resolve_timeout: 5m
  smtp_from: <email_from>
  smtp_smarthost: <smtp_host>
  smtp_require_tls: <true/false> 

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'kubeflow_receiver'

  routes:
    - match:
        alertname: 'Kubeflow job failing'
      receiver: 'kubeflow_receiver'
    - match:
        alertname: 'Airflow job failing'
      receiver: 'airflow_receiver'

receivers:
- name: 'kubeflow_receiver'
  email_configs:
  - to: 'kubeflow_admin@hpe.com'
    html: '{{ template "kubeflow_html.tmpl" . }}'
    headers:
      subject: '{{ template "kubeflow_html.tmpl" . }}'
- name: 'airflow_receiver'
  email_configs:
  - to: 'admin_airflow@hpe.com'
    html: '{{ template "airflow_html.tmpl" . }}'
    headers:
      subject: '{{ template "airflow_subject.tmpl" . }}'

templates:
- '/etc/alertmanager/templates/*.tmpl'
  10. Encode the updated alertmanager.yaml file to Base64:
    cat alertmanager.yaml | base64 -w0
  11. Replace the existing Base64 data in the Secret by editing the Secret:
    kubectl edit secret alertmanager-prometheus-kube-prometheus-alertmanager -n prometheus
  12. Restart the Prometheus Operator pod and then the Alertmanager pod to apply the changes:
    kubectl delete pod -n prometheus -l alertmanager=prometheus-kube-prometheus-alertmanager
kubectl delete pod -n prometheus -l app=kube-prometheus-stack-operator

Results:

You have updated the alertmanager.yaml file to forward alerts via different receivers.