Alarm Annotations & Severity

KloudMate allows you to attach custom metadata to your alarms using Annotations. Annotations are key-value pairs that provide additional context to the responders receiving the notification.

With the latest updates, annotations now support dynamic templating. This allows you to inject real-time data from the alarm evaluation directly into the notification text.

Setting up Annotations

Annotations are configured in the Alarm Details section when creating or editing an alarm.

1. Scroll to the Custom Annotations section.
2. Click Add Annotation.
3. Provide a Key (e.g., severity).
4. Provide a Value. This value can be static text or a template expression resolving the value dynamically.

Using Liquid Templates

Liquid templating allows you to dynamically populate annotation values based on the query results that triggered the alarm.

You can reference the values of queries and expressions using the {{ }} syntax.

Common Template Variables

• {{ $A.value }}: The evaluated numeric value of query/expression A.
• {{ $B.value }}: The evaluated numeric value of query/expression B.
• {{ dimensions.host_name }}: The value of the host_name dimension that triggered the alarm (useful for multi-dimensional alarms).
• {{ state }}: The current state of the alarm (e.g., FIRING, NORMAL).

Example Usage

If you have an alarm monitoring CPU utilization (Query A) grouped by host_name, you might create an annotation to provide immediate context in Slack:

• Key: Description
• Value: Host {{ dimensions.host_name }} is experiencing high CPU usage. Current value: {{ $A.value }}%.

When the alarm fires for the host web-server-01 with a CPU of 95%, the resulting annotation in the notification will read: Host web-server-01 is experiencing high CPU usage. Current value: 95%.

Dynamic Severity

You can use the reserved severity annotation key to dynamically set the severity level of the alarm based on the evaluated conditions. This is particularly useful when integrating with incident management tools like PagerDuty or Jira.

To use dynamic severity, you typically combine Liquid templating with if/else logic based on expression values.

Example: Dynamic Severity based on Thresholds

Let’s assume you have an alarm evaluating an error rate metric:

• If Error Rate > 10% (0.1), it’s critical
• If Error Rate > 5% (0.05), it’s warning

Create an annotation with:

• Key: severity
• Value:

{% if state.value > 0.1 %}
  critical
{% elsif state.value > 0.05 %}
  warning
{% else %}
  info
{% endif %}

When the notification is sent, KloudMate will evaluate the Liquid template and assign the appropriate severity string to the alert, allowing your downstream notification channels or incident routing rules to prioritize it correctly.