Triage, command, mitigation và recovery verification
Incident goal đầu tiên là giảm user harm; root cause có thể điều tra sau khi hệ thống ổn định và evidence được giữ.
First minutes
Confirm user impact/SLI, start time/scope/severity, assign incident commander/ops/communications and investigation streams, freeze risky changes, capture recent deploy/config/traffic. Open timeline/hypothesis log.
Stabilize
Rollback/disable flag, shed load, open circuit, reduce retries, isolate tenant/AZ, scale only if bottleneck supports, failover with fencing. Prefer reversible low-risk actions. Preserve logs/traces/profiles before restart if safe.
Investigate
Compare healthy/unhealthy by version/AZ/tenant/endpoint; RED → trace → dependency/resource USE → logs/profile. State hypothesis, predicted evidence, test; avoid dashboard wandering and multiple simultaneous changes.
Communication
Impact, affected scope, mitigation/current status, next update time; avoid speculative cause. Internal technical channel separate stakeholder updates. Handoffs explicit ownership/context.
Recovery
Verify user SLI/error budget, backlog/drain, retries/DLQ, data consistency and dependencies; monitor after rollback. Reconcile unknown/duplicate/stuck transactions. Declare resolved only when stable and follow-up owner assigned.