Incident response

An incident has a summary, priority, service, asset, assignee and times for opening, acknowledgment and resolution. Its state separates newly opened work from acknowledged, investigating, resolved and closed work. The Monitoring Operator records the first report against the affected service. The Incident Responder accepts ownership, investigates and records what happened in attributed updates. Response events mark paging, acknowledgment, escalation, mitigation and resolution with the responder and event time.

A high priority incident should carry enough context for the next person to act. Suppose an availability check fails while a server sample shows high memory use. The operator opens an incident for that service and asset. The assigned responder acknowledges it, records a response event and writes an update describing the check result and what they are testing. Later, a second update records the mitigation and the evidence used to assess recovery. The original observations and the responder’s statements serve different purposes; the team should keep both.

• Queue state. Use the incident state and priority to show which work is open, acknowledged or under investigation.
• Responsibility. Assign the incident to a person and record each response event with its actual responder.
• Chronology. Add updates at the time of the investigation so later readers can follow the sequence.

Define who may open an incident, how priorities are selected and when the assignee must acknowledge. Agree whether a page, escalation or mitigation event requires an update and who may close a resolved incident. Map each service to an owner and a planned on-call responder. A shift carries a start, end and escalation level; it gives the team context for routing but should be reviewed against actual coverage before the handoff is considered complete.

Rehearse an incident that begins with a failed availability check. Ask the Monitoring Operator to create the record, set a priority and assign the responder. Have the responder acknowledge it, enter the first response event and add an attributed update. Then pass the case to another on-call responder with a new event and update. The Service Owner should be able to see who acted at each point, what remained unresolved and when recovery was reported.

Test closure separately. A mitigation event records an action; it does not mean the service has recovered. The responder should review a new successful check and any outage restoration time before marking the incident resolved. The Service Owner can then review the history and close it. Keep the incident’s original opening and acknowledgment times for later response analysis.

• Open Incidents — Work the current queue by priority, service and assignee.
• My Incidents — See records assigned to the current responder.
• High Priority Incidents — Focus attention on urgent service interruptions.

See incident response for the handoff and service recovery for verification and closure.