# Diagnostic playbook

## Contents

1. Evidence collection
2. Build and image failures
3. Deploy and runtime failures
4. Health, routing, TLS, and DNS
5. Storage and database dependencies
6. GitHub workflow
7. Verification and rollback

## 1. Evidence collection

Collect only what the current layer needs. Redact values, authentication headers, registry URLs containing credentials, and secret-bearing environment output.

```bash
docker version
docker compose version
docker buildx version
docker info --format '{{json .ServerVersion}}'
docker ps --no-trunc --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Networks}}'
```

Use Coolify MCP for infrastructure identity and status. Start with `get_infrastructure_overview`, then use the relevant list and get tools. The built-in MCP is read-only and its documented tool set does not promise build logs.

For logs, request the failing build step with 50-100 surrounding lines. For a container, use a bounded time window and tail rather than dumping all history:

```bash
docker logs --since 15m --tail 300 <container>
docker inspect <container> --format '{{json .State}}'
```

## 2. Build and image failures

Check in this order:

1. Build context and Dockerfile path.
2. `.dockerignore` exclusions and required generated files.
3. Dockerfile syntax, stage names, and `COPY --from` targets.
4. Required build arguments versus runtime environment variables.
5. Lockfile and package-manager version consistency.
6. Base-image tag/digest availability and target architecture.
7. Registry authentication, rate limits, and image naming.
8. BuildKit cache behavior, disk/inode pressure, and OOM evidence.
9. Native dependency toolchain and libc/architecture compatibility.
10. File ownership, executable bits, and non-root user transitions.

Match the reproduction to Coolify:

```bash
docker compose -f <compose-file> config -q
docker compose -f <compose-file> config --format json
docker buildx build --progress=plain --platform <platform> --target <target> -f <dockerfile> <context>
```

Do not invent missing secret build arguments. Use BuildKit secrets for build-time credentials and Coolify environment variables/secrets for runtime values.

## 3. Deploy and runtime failures

Use container state before logs. Distinguish an image that never starts from a process that starts and exits.

For `Stopped after reaching restart limit`, missing containers after a crash loop, or uncertainty between Docker and PM2 restarts, follow [restart-loops.md](restart-loops.md).

| State | High-value evidence |
| --- | --- |
| Created, never running | Entrypoint/command, mounts, permissions, missing network, runtime constraints. |
| Exited | Exit code, OOM flag, termination signal, final bounded logs. |
| Restarting | Restart policy plus first crash, not the repeated tail alone. |
| Running, unhealthy | Healthcheck command, start period, timeout, in-container dependency and listening port. |
| Running, healthy, unreachable | Network membership, app bind address, Traefik router/service, DNS/TLS. |

Check resolved configuration rather than reading only the source:

```bash
docker compose -f <compose-file> config
docker inspect <container> --format '{{json .Config.Entrypoint}} {{json .Config.Cmd}}'
docker inspect <container> --format '{{json .Mounts}}'
docker inspect <container> --format '{{json .State.Health}}'
```

## 4. Health, routing, TLS, and DNS

Verify inside-out:

1. Process listens on `0.0.0.0:<container-port>` rather than only `127.0.0.1`.
2. Healthcheck uses an available binary and correct path, scheme, and port.
3. The target answers from its own network namespace or a disposable peer on the same private network.
4. The target and Traefik share the effective public network.
5. Router rule, entrypoint, middleware chain, service association, and load-balancer port are loaded.
6. Public DNS resolves to the intended server.
7. Ports 80/443 and the selected ACME challenge are reachable.
8. An external request returns expected status, headers, and body.

Interpret common HTTP results carefully:

- 404 from Traefik: routing did not match or load.
- 502: Traefik selected a backend but could not speak to it correctly.
- 503: the router exists but no usable server is available.
- 504: the selected backend address is unreachable or does not respond in time; investigate multi-network selection early.

## 5. Storage and database dependencies

- Confirm bind source existence, ownership, mount type, and target before changing permissions.
- Avoid `chmod -R 777` and broad ownership changes. Match the image's runtime UID/GID.
- Never delete or recreate a volume to test a hypothesis without a verified backup and explicit authorization.
- Confirm database DNS name, port, TLS mode, migrations, and readiness separately from application health.
- Treat upgrade/downgrade and internal PostgreSQL procedures as migrations requiring official Coolify guidance and rollback planning.

## 6. GitHub workflow

1. Resolve the exact repository, branch/PR, and commit deployed by Coolify.
2. Use GitHub connector data for repository files, PR patches, comments, checks, and workflow-run metadata.
3. For failing Actions checks, inspect the failed job and step logs with the GitHub CI workflow or `gh` when necessary.
4. Compare the failing commit with the last known-good deployment; avoid assuming the newest diff is causal.
5. Patch locally, validate, and show the focused diff.
6. Commit, push, rerun, or open a draft PR only when explicitly authorized.

## 7. Verification and rollback

Record before/after evidence for the layer that failed. A useful handoff includes:

- exact root cause and why alternatives were rejected;
- files/settings changed;
- tests and commands run with outcomes;
- deployment and external request status;
- unverified layers;
- rollback commit/config and any database or volume caveats.

If a production fix fails verification, stop further mutations, restore the known-good state where authorized, and report the observed regression.