Incident Response Best Practices

The incident lifecycle

Every incident goes through phases. Understanding these phases helps you optimize each one:

Phase 1: Detection

The goal is to detect incidents before customers report them.

Best practices

• Monitor proactively – Don't wait for customer complaints. Use uptime monitoring with appropriate check intervals.
• Check from outside – Internal health checks miss network and DNS issues. External monitoring sees what users see.
• Use multiple signals – Combine uptime monitoring with error rates, latency metrics, and business metrics.
• Reduce noise – False positives cause alert fatigue. Tune thresholds and use confirmation checks.

Phase 2: Triage

Quickly assess what's happening and who needs to be involved.

Severity levels

Define clear severity levels so everyone knows how to respond:

Best practices

• Don't over-escalate – Every SEV1 burns team energy. Reserve it for true emergencies.
• Assign an incident commander – One person coordinates for major incidents.
• Communicate early – Update your status page even before you know the cause.

Phase 3: Response

Focus on restoring service first, then root cause.

Best practices

• Restore first, debug later – A rollback that fixes the issue is better than a perfect diagnosis during an outage.
• Communicate progress – Regular status updates reduce customer anxiety and support tickets.
• Document as you go – Note what you tried, what worked, what didn't. This helps the postmortem.
• Don't make it worse – Be careful with fixes during incidents. A bad change can extend the outage.

Phase 4: Recovery

Verify the fix and communicate resolution.

Best practices

• Verify from multiple angles – Check monitoring, test manually, confirm customer-facing functionality.
• Watch for recurrence – Stay alert for a period after resolution. Some fixes are temporary.
• Update status page – Clear communication that the incident is resolved.
• Thank your team – Incident response is stressful. Acknowledge the effort.

Phase 5: Learning (Postmortem)

The most important phase. Without learning, you'll repeat the same incidents.

Postmortem principles

• Blameless – Focus on systems and processes, not individuals. People make mistakes; systems should catch them.
• Timely – Conduct postmortems within days while memory is fresh, not weeks later.
• Action-oriented – Every postmortem should produce specific, assigned action items.
• Shared – Publish postmortems internally so other teams can learn.

Postmortem template

A good postmortem covers:

• What happened (timeline)
• Impact (users affected, duration, business impact)
• Root cause (why did this happen?)
• What went well (detection, response)
• What could be improved
• Action items (with owners and due dates)

See our postmortem template for a ready-to-use format.

Building incident response capability

Runbooks

Document how to diagnose and fix common issues. When the pager goes off at 3 AM, you don't want to be searching for answers.

On-call rotation

Spread the load fairly. Burnout from constant on-call leads to turnover. Use tools like PagerDuty or Opsgenie for scheduling and escalation.

Practice

Run game days or chaos engineering exercises. Teams that practice incident response handle real incidents better.

---

Related resources

• What is MTTR?
• Postmortem template
• Status page templates
• PagerDuty integration