July 2024: CrowdStrike releases an update that brings down 8.5 million Windows machines worldwide. Airports, hospitals, banks — everything stops. Total estimated damage: $5.4 billion. Teams with a well-established incident management process recovered in hours. Those without it took days.
The difference isn't in technology or team size. The difference is whether you have a process that automatically activates when something breaks: who responds, what they do, how they communicate, how they learn from mistakes.
Incident management isn't about firefighting. It's about a system that prevents fires from starting and makes the ones that do happen short and controlled.
Incident Lifecycle
Every incident goes through five phases. Skipping any of them either prolongs the current incident or increases the chance of the next one.
Phase 1: Detection
The faster you know about a problem, the faster you can fix it. Here are sources of detection, ranked by speed:
Automated monitoring (seconds): Uptime monitoring detects downtime in 30-60 seconds. Synthetic checks from multiple regions confirm issues, excluding false positives. The alert goes to the on-call engineer.
Internal detection (minutes): An engineer notices an anomaly in logs, an unusual metric in Grafana, or an error in deployment. Slower than automation, but covers scenarios monitoring doesn't track.
User reports (minutes to hours): A support ticket, a complaint on Twitter, or a message on Slack. The slowest source by far. If you learn about problems mostly from users, your monitoring is insufficient.
Goal: minimize MTTD (Mean Time to Detect). With properly configured monitoring, 30-90 seconds for critical services. Key checks: HTTP endpoints, SSL certificates, DNS, TCP ports, database, and heartbeat cron jobs.
Phase 2: Triage
Not all incidents are equal. Triage is a quick assessment: how serious the problem is, who it affects, and what response level is needed.
P1 — Critical: Core functionality unavailable for all or most users. API completely down. Payments failing. Data lost. Response: Immediately, all hands on deck.
P2 — High: Significant degradation or subset of users affected. API works but with 20% errors. Search doesn't work, everything else does. Response: Within 15 minutes, on-call plus backup.
P3 — Medium: Non-critical feature impaired, workaround exists. Report export is slow but works. Notifications delayed 5 minutes. Response: During business hours, service owner.
P4 — Low: Cosmetic issue, minimal impact. Button renders incorrectly. Typo in email. Response: In backlog, next planning session.
Severity is determined by user impact, not technical complexity. A one-line fix that blocks all payments is P1. A major refactoring that users don't notice is P4.
Phase 3: Response
This is where diagnosis and fixing happen. Three parallel tracks:
Technical work: Diagnosis → root cause identification → fix implementation → deploy → verification. Engineers work on the problem.
Communication: Status page updates every 15-20 minutes. Incident progresses: Investigating → Identified → Monitoring → Resolved. Users are informed.
Coordination: For P1/P2, an incident commander manages the process. Not fixing the issue themselves, but coordinating: who does what, what additional resources are needed, are tasks properly prioritized.
Phase 4: Recovery
A deploy of the fix doesn't mean the incident is closed. Recovery is observing stability after the fix. How long to monitor depends on severity:
P1: Minimum 1-2 hours of monitoring after deploy. Verify metrics returned to baseline, error rate at zero, response time normal.
P2: 30-60 minutes of active monitoring.
P3/P4: Standard post-deployment checks.
Common mistake: closing the incident immediately after deploying the fix. Then 20 minutes later the problem returns because the fix was incomplete or caused side effects. The recovery phase is insurance against this.
Phase 5: Learning
Post-mortem. Analysis of the incident after it's closed. Not for finding blame — for improving the system. More details below.
Alerting: How to Notify Correctly
Monitoring detects the problem. Alerting gets it to the right person. Between "check fires" and "engineer starts work," alerting fills the gap.
Channels by Severity
Not all alerts are equally urgent. Notification channels should match severity:
P1: Phone call / SMS + Push notification + Slack escalation. Wakes engineer at 3am. Can't wait.
P2: Push notification + Slack/Telegram. Needs attention within 15 minutes, but not a phone call at 3am (unless SLA demands it).
P3: Slack / Email. Business hours, no escalation.
P4: Jira ticket / Email digest. Doesn't need immediate reaction.
Alert Fatigue: When There Are Too Many Alerts
If an on-call engineer gets 50 alerts a day, they'll start ignoring them. In a week, they'll miss a real P1 because they're used to noise. Alert fatigue is the enemy of incident management.
Remedies:
Grouping: When 10 checks of one service fail at once, one alert, not ten.
Multi-region confirmation: Alert only when 2+ of 3 regions confirm the issue. One failed check isn't an alert.
Proper thresholds: Not too sensitive, to avoid triggering on every spike, but sensitive enough for real problems.
Regular review: Once a month, check which alerts were actionable, which were noise. Delete the noise or retune.
Escalation
Alert went to on-call. On-call didn't respond within 5 minutes. What next? Without escalation, nothing. With escalation:
0 min: Alert → primary on-call (Slack + push notification)
5 min: No response → secondary on-call (Slack + push + SMS)
10 min: No response → engineering lead (phone call)
15 min: No response → VP Engineering (phone call + "all hands")
Escalation must be automatic. If it depends on someone manually calling the next person, it doesn't work at 3am.
On-Call: Organizing Your Rotation
If your service must run 24/7 (and a 99.9% SLA implies this), you need an on-call rotation.
Principles of Healthy On-Call
Rotation: Weekly, among all engineers on the team. One person on-call constantly burns out in a month.
Compensation: On-call duty is work. Pay for it or give time off. Unpaid on-call kills morale and causes turnover.
Minimum 2 people: Primary + secondary. If primary is unavailable, secondary automatically takes over.
Runbooks: The on-call engineer doesn't need to know every microservice by heart. A runbook for each alert: what to check, what commands to run, when to escalate.
Realistic SLAs: If one person is on-call and you promise MTTA under 5 minutes, remember people sleep, shower, drive. Plan for 10-15 minutes for night incidents.
On-Call as Feedback Loop
On-call isn't just "sitting by the phone." It's a feedback loop for service quality. Teams that run their own code write more reliable code. The principle "you build it, you run it" (Werner Vogels, Amazon) works because if you know you'll be woken by your own bug, you test more carefully.
Metric: if an on-call engineer gets more than 2 pager alerts in a night, that's a service reliability problem, not a monitoring problem. Fix the service, not the thresholds.
Incident Communication
Technical work is half of incident management. The other half is communication. With users, with leadership, between teams.
External Communication: The Status Page
Your status page is the primary channel for communicating with users during incidents. Updates every 15-20 minutes. Be specific instead of vague. Stages: Investigating → Identified → Monitoring → Resolved.
Additional channels: email to status page subscribers, posts on Twitter/X for major incidents, banner in app. Detailed guide on incident communication in a separate article.
Internal Communication
During an incident, there's chaos. Who's working on what? What's the ETA? Has this been escalated? Internal communication prevents that chaos.
War room (Slack channel / Zoom call). All responders in one channel. Real-time updates. "I found the issue: database is at 100% CPU. Scaling now." "Deployment in progress, 2 minutes."
Clear incident commander. One person coordinating. Prevents "who's doing what" confusion.
Handoff protocol. When incidents last 8+ hours, people rotate. Explicit handoff: here's what we found, here's what we tried, here's what's next.
Post-Mortem: Learning from Incidents
The incident is over. Now you figure out how to prevent the next one. Post-mortems are essential.
When to Do a Post-Mortem
Every P1/P2 incident. For P3, optional but encouraged if patterns emerge. For P4, skip.
Blameless Culture
This is critical. "Dave deployed bad code" is useless. "The deployment pipeline lacked automated testing for this edge case" is actionable. Blame discourages people from reporting problems; systemically finding root cause actually prevents recurrence.
Post-Mortem Structure
Summary: What happened, in 1-2 sentences.
Timeline: Detection → diagnosis → fix → recovery. Include key timestamps.
Impact: How many users? How long? Revenue loss? Data loss?
Root cause: The underlying systemic issue, not "Dave made a typo."
Contributing factors: Things that made the incident worse. Maybe monitoring wasn't granular enough. Maybe runbook was outdated.
Action items: Specific, measurable steps to prevent this. "Improve monitoring" is vague. "Add database CPU metric with threshold of 80% to catch connection pool exhaustion earlier" is actionable. Assign owners, set deadlines.
Metrics: MTTA, MTTR, and Severity Distribution
You can't improve what you don't measure. Track these:
MTTA (Mean Time to Acknowledge): Average time from alert to engineer starting work. Target: under 5 minutes for P1.
MTTR (Mean Time to Resolve): Average time from detection to full recovery. Target: under 30 minutes for P1, under 2 hours for P2.
Incident frequency: How many P1s, P2s, P3s per week/month? If frequency is increasing, something is degrading.
Repeat incidents: How many incidents are the same root cause happening again? If the same problem recurs, your post-mortem action items aren't being completed.
Common Mistakes
No escalation. Alert goes to on-call, on-call doesn't respond, and nothing happens. Hours pass. Automate escalation.
No status page updates. Users don't know what's happening, assume the worst, and leave. Update every 15-20 minutes, even if you have nothing new to say.
No post-mortems. Same incident happens again three months later. You never learned.
Blame-focused post-mortems. "Dave should have been more careful." Dave leaves. You still have the same problem.
On-call burnout. One engineer on-call for months. They get tired, make mistakes, leave. Rotate.
Building Your Incident Management Process
Start minimal. AtomPing makes this straightforward — you can have monitoring and incident management running in 15 minutes:
1. Set up monitoring of critical endpoints. HTTP checks, SSL expiry, DNS.
2. Configure incident thresholds in AtomPing: if X checks fail in Y regions, create incident automatically.
3. Set up a status page. Link it to your monitoring — incidents auto-create status updates.
4. Pick an on-call: who responds to P1 alerts? How do they escalate?
5. Write a post-mortem template. Use it after the first real incident.
6. Review quarterly. Are severities right? Are MTTA/MTTR improving? Are people burned out?
Incident management is a system, not a checklist. It evolves. Start simple, add complexity only when you need it. The goal is a team that responds fast, communicates clearly, and learns from every incident.
FAQ
What is incident management?
Incident management is the process of detecting, responding to, and resolving service disruptions. It covers the full lifecycle: monitoring detects a problem, alerting notifies the right people, a structured response process kicks in (triage, diagnosis, fix, verification), and a post-mortem prevents recurrence. The goal is to minimize both the duration and impact of incidents.
What's the difference between incident management and incident response?
Incident response is the immediate reaction: detect, triage, fix. Incident management is broader — it includes response plus everything around it: defining severity levels, maintaining on-call schedules, writing runbooks, conducting post-mortems, and continuously improving the process. Response is a phase; management is the system.
How do I prioritize incidents?
Use severity levels tied to user impact, not technical complexity. P1/Critical: core functionality down for all users. P2/High: significant feature degraded or subset of users affected. P3/Medium: non-critical feature impaired, workaround available. P4/Low: cosmetic or minor issue. Severity determines response time, escalation path, and communication level.
Do I need an on-call rotation?
If your service needs to be available outside business hours — yes. Without on-call, a 2am outage waits until 9am for response. That's 7 hours of downtime. On-call doesn't mean 24/7 monitoring by one person. Rotate weekly across the team, compensate fairly, and automate as much as possible so on-call engineers only wake up for real incidents.
What is MTTA and MTTR?
MTTA (Mean Time to Acknowledge) measures how long between an alert firing and an engineer starting work on it. MTTR (Mean Time to Resolve) measures total time from incident detection to full resolution. Both are key operational metrics. MTTA tells you if your alerting and on-call process is fast enough. MTTR tells you if your overall incident response is effective.
Should post-mortems be blameless?
Yes. Blameless doesn't mean accountability-free — it means focusing on systemic causes rather than individual mistakes. 'Dave deployed bad code' isn't actionable. 'The deployment pipeline lacked a canary stage, so the faulty release reached 100% of traffic instantly' is actionable. Blame discourages reporting; systemic analysis prevents recurrence.
How often should I review my incident process?
After every P1/P2 incident (post-mortem), and quarterly for the overall process. Quarterly reviews cover: are severities calibrated correctly? Is MTTA improving? Are action items from post-mortems getting completed? Are there patterns across incidents? This is how the process evolves from 'we have a runbook' to 'we rarely have serious incidents.'
What tools do I need for incident management?
At minimum: monitoring (to detect), alerting (to notify), a status page (to communicate), and a post-mortem template (to learn). As you scale: on-call scheduling (PagerDuty, OpsGenie), incident tracking, automated runbooks, and integration between all of these. The tools matter less than the process — a team with good habits and basic tools outperforms a team with expensive tools and no process.