High CPU Usage
High CPU Usage monitors cpu_usage_percent for web-server-01. A critical alert is triggered when the value is greater than 90 for 5 minutes. Recovery occurs below 75.
| Rule | High CPU Usage | Status | Enabled |
|---|---|---|---|
| Source | Metric-based alert | Severity | Critical |
| Metric | cpu_usage_percent | Resource | web-server-01 |
| Warning | 80 | Critical | 90 |
| Recovery | 75 | Evaluation | 5 minutes |
| Notifications | Cooldown | 15 minutes |
High CPU Usage monitors cpu_usage_percent for web-server-01. A critical alert is triggered when the value is greater than 90 for 5 minutes. Recovery occurs below 75.
How Monitoring Alert Rules Improve Reliability
Monitoring alert rules turn raw operational data into actionable signals. A strong rule identifies the important metric, defines an understandable threshold, and explains what should happen after a breach. Teams can use alert rules for servers, applications, APIs, databases, cloud resources, security events, logs, and business indicators. The best rules are specific enough to catch real problems without creating unnecessary noise. Warning thresholds provide early awareness, while critical thresholds identify situations needing faster action. Recovery thresholds prevent repeated state changes when a metric moves near the boundary.
Designing Useful Alert Conditions
Start with a measurable symptom that matters to users or operations. Choose an aggregation method that matches the signal. Averages can smooth brief spikes, while maximum values expose short but severe events. Evaluation windows should be long enough to reduce false alarms without hiding real failures. Consecutive failure checks are useful for unstable measurements. Missing data also needs an explicit policy because silence can mean healthy inactivity, collection failure, or a broken service. Escalation settings should match severity and ownership. A warning may only require email, while a critical production alert may need an on-call system and secondary escalation.
Reducing Alert Fatigue
Alert quality depends on suppression and grouping as much as thresholds. Maintenance windows stop expected work from creating incidents. Deduplication prevents many identical events from producing repeated notifications. Grouping related alerts gives responders a clearer incident picture. Cooldown periods and re-notification limits control repetition. Recovery messages confirm that service health returned to normal. Runbook links can shorten troubleshooting time by giving responders immediate guidance. Review alert performance regularly using incident history. Remove rules that never lead to action. Tune rules that fire too often. Keep ownership, documentation, schedules, and escalation paths current. Reliable alerts should guide action quickly and clearly.
Example Alert Rule Data
| Rule | Metric | Condition | Evaluation | Severity | Recovery |
|---|---|---|---|---|---|
| High CPU Usage | cpu_usage_percent | > 90% | 5 minutes | Critical | < 75% |
| API Error Rate | http_5xx_rate | > 5% | 10 minutes | Major | < 2% |
| Disk Space Low | disk_free_percent | < 10% | 15 minutes | Warning | > 20% |
| Login Failure Spike | failed_logins | > 100 | 5 minutes | Critical | < 25 |
Frequently Asked Questions
What is a monitoring alert rule?
It is a condition that watches a signal and changes state when defined criteria are met.
How should I choose thresholds?
Use historical behavior, service objectives, capacity limits, and incident experience instead of arbitrary numbers.
What is an evaluation window?
It is the time period used to examine data before deciding whether an alert should fire.
What causes alert fatigue?
Too many low-value, duplicate, unstable, or unactionable alerts can overwhelm responders.
How does deduplication work?
A deduplication key lets related events share one incident instead of opening repeated incidents.
Why use a recovery threshold?
It creates separation between trigger and recovery points, reducing rapid alert state changes.
Should missing data trigger an alert?
It depends on the signal. Missing telemetry can be harmless or indicate collection failure.
What is a maintenance window?
It is a scheduled period when selected alerts are suppressed during expected operational work.
How often should alerts repeat?
Repeat only often enough to prevent missed incidents while avoiding unnecessary responder interruption.
What is the difference between warning and critical?
Warnings indicate developing risk. Critical alerts indicate urgent conditions needing faster action.