Alerting

Alerting is the mechanism that notifies engineers when a system requires human attention — turning monitoring data into actionable notifications delivered to the right person at the right time. Effective alerting distinguishes between symptoms (user-facing impact) and causes (internal metrics), prioritizes by severity, and follows escalation paths from the on-call engineer to team leads to incident commanders. Poor alerting creates alert fatigue, where teams ignore pages because most are false alarms or non-actionable. SLO-based alerting and burn-rate calculations represent the modern approach to meaningful, low-noise alerts.

Why This Matters

Monitoring without alerting is just data collection. Alerts are the bridge between detecting a problem and having a human fix it. But alerting is deceptively hard to get right. Naive threshold alerts — page when error rate exceeds 1% for 5 minutes — generate noise during brief spikes and miss slow degradation. Google's SRE practice introduced burn-rate alerting: calculate how fast your error budget is being consumed and alert when the burn rate predicts budget exhaustion. A 14x burn rate over 1 hour means you will exhaust your monthly budget in 2 days — worth an immediate page. A 2x burn rate over 6 hours means gradual degradation — worth a ticket. This approach reduces noise by 90%+ while catching every real incident. Combined with proper on-call rotations and escalation policies, alerting ensures incidents are handled promptly without burning out engineers.

The Building Blocks

• Alert Rules: Conditions defined against metrics that trigger notifications — threshold-based (CPU > 90%), anomaly-based (3 standard deviations), or burn-rate-based (error budget consumption)
• Severity Levels: Classification of alerts by urgency and impact — P1 (page immediately, user-facing outage), P2 (page during business hours), P3 (ticket, address this week)
• On-Call Rotations: Scheduled assignment of engineers responsible for responding to alerts, typically weekly rotations with primary and secondary responders for redundancy
• Escalation Policies: Rules for what happens when an alert is not acknowledged — escalate from primary to secondary on-call, then to team lead, then to engineering manager after defined intervals
• Runbooks: Documented procedures linked to each alert, providing step-by-step diagnosis and remediation instructions so any on-call engineer can handle the incident

Under the Hood

Modern alerting systems operate through a pipeline: metric evaluation, alert routing, notification delivery, and incident management. Prometheus Alertmanager evaluates rules against time-series data, groups related alerts to reduce noise (a cluster of 10 failing pods produces one alert, not 10), and routes alerts through PagerDuty, OpsGenie, or Slack based on severity and team ownership.

SLO-based alerting represents the most effective approach. Instead of setting arbitrary thresholds on internal metrics, you define alerts based on error budget burn rates. The multi-window approach uses two time windows per severity: for immediate pages, check if the last 1 hour exceeds 14x burn rate AND the last 5 minutes exceeds 14x (to avoid stale alerts). For ticket-based alerts, check if the last 6 hours exceed 6x burn rate AND the last 30 minutes exceeds 6x. This dual-window approach catches both sudden outages and gradual degradation while virtually eliminating false positives.

Alert fatigue management is an ongoing operational practice. Teams should track alert-to-incident ratios: if fewer than 50% of pages result in an actual incident requiring action, the alerts need tuning. Regular alert review meetings examine each alert that fired, classify it as true positive, false positive, or not actionable, and refine thresholds accordingly. The goal is that every page wakes an engineer for a genuine, user-impacting problem that requires human judgment to resolve.

How Companies Actually Do This

PagerDuty Processes billions of alert events and provides intelligent noise reduction, grouping related alerts and using ML to predict incident severity, used by over 20,000 organizations

Google Pioneered SLO-based alerting and error budget burn rates in their SRE practice, reducing false-positive pages by over 90% compared to traditional threshold-based alerting

Atlassian Built Opsgenie for alert management with on-call scheduling, escalation policies, and integration with Jira for incident tracking across engineering organizations

Common Pitfalls

1. Alerting on causes instead of symptoms — alerting when CPU exceeds 80% is less useful than alerting when request error rate exceeds the SLO because users do not experience CPU, they experience errors
2. Not having runbooks for every alert — on-call engineers get paged at 3 AM for an alert with no documentation, wasting time figuring out what the alert means and how to remediate
3. Creating too many P1 alerts so everything pages immediately — when everything is urgent, nothing is; reserve P1 for user-facing outages and use P2/P3 for degradation

Interview Questions Worth Practicing

1. How does SLO-based burn-rate alerting reduce alert fatigue compared to simple threshold alerts?
2. How would you design an on-call rotation and escalation policy for a team of six engineers?
3. What is the difference between alerting on symptoms versus causes, and why does it matter?

The Tradeoffs

• Sensitivity vs. Fatigue: Lower thresholds catch more issues but generate more false positives that train engineers to ignore pages
• Speed vs. Confidence: Shorter evaluation windows detect issues faster but are more prone to transient spike false alarms
• Automation vs. Human Judgment: Auto-remediation handles known issues faster but novel incidents require human diagnosis and creative problem-solving

How to Explain This in an Interview

Here is how I would explain Alerting in a system design interview:

Alerting notifies engineers when systems need human attention. I follow Google SRE principles: alert on user-facing symptoms (error rate, latency) not internal causes (CPU, memory). I use multi-window burn-rate alerting — if the error budget is being consumed at 14x the sustainable rate over the last hour, that is an immediate page; at 2x over 6 hours, that is a ticket. This eliminates most false positives. Every alert needs a severity level (P1 pages immediately, P2 pages business hours, P3 creates a ticket), a clear runbook, and an owning team. On-call rotations should have primary and secondary responders with escalation policies. I track alert-to-incident ratios — if less than half of pages lead to real incidents, alerts need tuning.

Related Topics

• Monitoring
• Observability
• Availability
• Failover

The Real-World Incident That Made This Famous

Understanding Alerting became critical after multiple high-profile production incidents at major tech companies. When systems handle millions of users, even small misunderstandings about Alerting can lead to cascading failures that cost millions in lost revenue and erode user trust. Companies like Netflix, Google, Amazon, and Meta have all invested heavily in mastering Alerting because they learned the hard way that ignoring it leads to outages.

The key lesson from these incidents: Alerting is not just a theoretical concept — it is a practical skill that separates engineers who build resilient systems from those who build fragile ones. Every major outage report from the past decade involves at least one Alerting-related design decision that was either implemented incorrectly or overlooked entirely during the initial architecture review.

How Senior Engineers Think About This

Senior engineers approach Alerting differently from textbook definitions. Instead of memorizing rules, they build mental models. They ask: "What problem does Alerting solve? When does it fail? What are the alternatives?" This problem-first thinking leads to better design decisions because every system has unique constraints.

When evaluating Alerting in a system design context, experienced engineers consider the failure modes first. What happens when this component goes down? How does the system degrade? Is the degradation graceful or catastrophic? These questions reveal more about your understanding than any textbook definition.

The key difference between junior and senior engineers when it comes to Alerting: juniors focus on the happy path, while seniors design for what happens when things go wrong. They consider operational cost, team expertise, monitoring requirements, and how the decision will look six months from now when traffic has grown 10x.

Common Interview Mistakes

Mistake 1: Giving a textbook definition without context. Interviewers want to see you connect Alerting to real systems and real problems. Instead of reciting definitions, explain when and why you would use Alerting in the system you are designing.

Mistake 2: Not discussing trade-offs. Every design decision involving Alerting has trade-offs. Discuss what you gain and what you give up. Acknowledge the downsides and explain why the benefits outweigh them for your specific use case.

Mistake 3: Overcomplicating the solution. Start with the simplest approach to Alerting that meets the requirements, then add complexity only when justified. Many candidates jump to complex implementations when a simpler solution would work perfectly.

Production Checklist

• Define clear metrics for measuring the effectiveness of your Alerting implementation
• Set up monitoring and alerting that specifically tracks Alerting-related failures
• Document your Alerting design decisions in Architecture Decision Records (ADRs)
• Test failure scenarios related to Alerting in staging before production deployment
• Review and update your Alerting implementation quarterly as system requirements evolve
• Train new team members on the specific Alerting patterns used in your system
• Establish runbooks for common Alerting-related incidents and recovery procedures

Practical Implementation for .NET Developers

In .NET, alerting is typically configured at the infrastructure level rather than in application code. Prometheus Alertmanager evaluates rules against metrics from .NET services. Azure Monitor Alerts can trigger on Application Insights metrics, log queries, and health check failures from ASP.NET Core health endpoints. For programmatic alerting, the PagerDuty.ApiClient NuGet package creates incidents via their API. Alerting on .NET-specific signals includes EventCounters (GC pressure, thread pool starvation) via dotnet-counters and custom health check failures surfaced through Microsoft.Extensions.Diagnostics.HealthChecks.

ASP.NET Core setup: Create a service class that encapsulates the logic, register it with dependency injection, and inject it into your controllers or minimal API endpoints. The built-in DI container handles lifecycle management.

Entity Framework Core: For database interactions, EF Core provides the ORM layer. Use migrations for schema management and raw SQL for performance-critical queries. Consider Dapper for read-heavy paths where EF Core overhead matters.

Azure integration: If deploying to Azure, leverage managed services — Azure Cache for Redis, Azure SQL, Azure Service Bus, Azure Cosmos DB. These eliminate operational overhead and provide built-in monitoring through Application Insights.

Testing: Use xUnit with Testcontainers for integration tests that spin up real databases in Docker. Mock external dependencies with NSubstitute. The WebApplicationFactory class lets you test your entire HTTP pipeline in-process.

Monitoring: Add Application Insights telemetry to track request latency, dependency calls, and custom metrics. Use structured logging with Serilog to make production debugging possible:

Log.Information("Processing {Operation} for {ResourceId}", operation, resourceId);

This gives you searchable, structured logs in Azure Monitor or Seq.