---
name: observability-incident-response
description: Diagnose production incidents using metrics, logs, traces, error tracking, health checks, deployment history, and infrastructure state. Use with Grafana, Prometheus, Loki, OpenTelemetry, Sentry or Bugsink, Uptime Kuma, Coolify, Docker, HTTP, DNS, TLS, SMTP, PostgreSQL, Redis, and related systems. Produces evidence-backed timelines and safe remediation plans before any restart, rollback, or configuration change.
compatibility: Read-only observability access is preferred. Restarts, rollbacks, alert changes, dashboard writes, log deletion, and production configuration changes require separate authorization.
license: Apache-2.0
---

# Observability incident response

Use telemetry to reduce uncertainty. Do not begin by restarting services or changing configuration unless immediate containment is required and explicitly authorized.

## Incident workflow

1. Define the symptom, affected users, start time, current severity and business impact.
2. Confirm whether the problem is ongoing, intermittent or already recovered.
3. Establish a timeline from deploys, configuration changes, alerts, errors and infrastructure events.
4. Compare healthy and affected windows using the same queries and dimensions.
5. Follow the request path across edge, application, queue, database, storage and external dependencies.
6. Form competing hypotheses and identify evidence that would disprove each one.
7. Select the smallest reversible mitigation.
8. Verify recovery using user-facing probes and telemetry, not only a successful command response.
9. Record root cause confidence, contributing factors and follow-up work.

## Evidence sources

- uptime and synthetic checks;
- HTTP status, latency, saturation and error ratios;
- Prometheus metrics and alert history;
- Loki or application logs with request and trace identifiers;
- OpenTelemetry traces and service maps;
- Sentry/Bugsink issues, releases, regressions and affected users;
- Coolify deployments and container restarts;
- database connections, locks, replication and slow queries;
- Redis memory, evictions and latency;
- DNS, TLS certificate, SMTP and external API health.

## Query discipline

Use bounded time windows and narrow dimensions. Prefer summaries, percentiles and representative samples before fetching complete dashboards or large log bodies. Redact tokens, cookies, credentials, message bodies, personal information and environment values.

## Change gates

Read-only diagnostics may proceed. The following require explicit intent:

- restart, redeploy or rollback;
- scale changes;
- alert muting or deletion;
- dashboard or datasource modification;
- database failover;
- queue purge;
- cache flush;
- firewall, DNS or certificate changes.

Before a change, state expected impact, affected resources, rollback, verification query and maximum observation window.

## Incident report

Return:

1. severity and customer impact;
2. precise timeline with time zone;
3. observed symptoms and decisive evidence;
4. hypotheses considered and rejected;
5. likely root cause and confidence;
6. mitigation performed or proposed;
7. recovery verification;
8. remaining risks;
9. preventive actions with owners and measurable completion criteria.