# Restart-loop diagnosis

## Contents

1. Identify the loop
2. Resolve Coolify and Docker identifiers
3. Capture a retained crashed container
4. Recover when Coolify removed the container
5. Interpret the evidence
6. Inspect startup code
7. Apply the narrow fix
8. Verify without burning the restart limit

## 1. Identify the loop

Do not use "restart" as a single diagnosis.

| Loop | Decisive evidence | Typical owner |
| --- | --- | --- |
| Docker container restart | The same container ID has an increasing `.RestartCount`; PID 1 exits. | Entrypoint, startup dependency, runtime crash, signal, OOM. |
| PM2 worker restart | Docker `.RestartCount` stays zero while PM2 reports worker exits/restarts. | Node process exception, memory threshold, worker lifecycle. |
| Coolify redeployment loop | New container names/IDs and deployment records appear; each individual container may have restart count zero. | Auto-deploy webhook, CI/API trigger, rolling-update behavior. |

Coolify's `Stopped after reaching restart limit (N/N)` status means it observed crash restarts and stopped the application after the configured maximum. Verify the behavior against the installed Coolify version because restart handling is version-sensitive.

## 2. Resolve Coolify and Docker identifiers

Keep these identifiers separate:

- **Coolify project UUID:** groups environments/resources and appears in UI/API paths. It is not normally the Docker Compose project label.
- **Coolify resource UUID:** identifies the deployed application/service. Coolify commonly passes it as `COOLIFY_RESOURCE_UUID` and `docker compose --project-name`.
- **Compose project name:** stored as `com.docker.compose.project` and used to find the stack reliably.
- **Compose service name:** stored as `com.docker.compose.service`, such as `app`.
- **Generated container name:** includes deployment-specific suffixes and may change or disappear.

Derive the Compose project value from the deployment command or runtime labels. Do not guess it from the Coolify project UUID.

```bash
docker ps -a --no-trunc \
  --filter "label=com.docker.compose.project=<resource-uuid>" \
  --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Image}}'
```

If the value is unknown, inspect all retained Coolify Compose containers without printing environment values:

```bash
docker ps -a --no-trunc \
  --format '{{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Label "com.docker.compose.project"}}\t{{.Label "com.docker.compose.service"}}'
```

## 3. Capture a retained crashed container

Select by stable labels, then inspect before reading broad logs:

```bash
APP_ID="$(docker ps -aq \
  --filter "label=com.docker.compose.project=<resource-uuid>" \
  --filter "label=com.docker.compose.service=app" | head -n1)"

test -n "$APP_ID" || { echo 'No retained app container found'; exit 1; }

docker inspect "$APP_ID" \
  --format 'Name={{.Name}} Status={{.State.Status}} ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}} RestartCount={{.RestartCount}} Error={{json .State.Error}} StartedAt={{.State.StartedAt}} FinishedAt={{.State.FinishedAt}} Health={{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}'

docker logs --since 30m --tail 300 "$APP_ID"
```

Request redaction of credentials, database URLs, authentication headers, private keys, and tokens. Preserve the first error, not just the final repeated shutdown.

When health evidence matters:

```bash
docker inspect "$APP_ID" \
  --format '{{if .State.Health}}{{range .State.Health.Log}}{{.End}} exit={{.ExitCode}} {{printf "%s" .Output}}{{println}}{{end}}{{else}}no healthcheck{{end}}'
```

## 4. Recover when Coolify removed the container

After the restart limit, Coolify may stop the application and the expected container may no longer exist. If an ID variable is empty, do not run `docker inspect ""` repeatedly. First show the label search result and explain that the target is absent.

Docker events may retain lifecycle evidence, though not application stderr:

```bash
docker events \
  --since 2h \
  --until "$(date -Iseconds)" \
  --filter type=container \
  --filter "label=com.docker.compose.project=<resource-uuid>" \
  --format '{{.Time}} {{.Action}} {{index .Actor.Attributes "name"}} exit={{index .Actor.Attributes "exitCode"}}'
```

For a controlled capture, obtain explicit authorization to start or redeploy the production resource. Begin capture immediately and stop after the first decisive crash:

```bash
for attempt in $(seq 1 30); do
  APP_ID="$(docker ps -aq \
    --filter "label=com.docker.compose.project=<resource-uuid>" \
    --filter "label=com.docker.compose.service=app" | head -n1)"
  test -n "$APP_ID" && break
  sleep 1
done

test -n "$APP_ID" || { echo 'App container did not appear'; exit 1; }

docker logs --follow --since 1m "$APP_ID"
```

Use the Coolify UI or its approved deployment workflow to start the resource. Do not bypass Coolify with an improvised `docker compose up` against guessed environment files.

## 5. Interpret the evidence

| Evidence | Meaning | Next check |
| --- | --- | --- |
| Exit code `1`, `OOMKilled=false` | Application/startup failure. | First stderr/stack trace; migration and secret checks. |
| Exit code `126` | Command exists but cannot execute. | Mode bits, mount flags, user permissions, architecture. |
| Exit code `127` | Command or runtime file missing. | Entrypoint, PATH, standalone output, copied scripts. |
| Exit code `137` or `OOMKilled=true` | SIGKILL, usually memory pressure. | Container limit, host OOM events, worker count, peak RSS. |
| Exit code `143` | SIGTERM. | Coolify/deployment events, graceful shutdown, external stop. |
| `[Migrations] Failed` before server logs | Startup migration caused PID 1 to exit. | Database DNS/TLS/auth/schema plus retry policy. |
| PM2 shows worker restarts, Docker count zero | Worker loop inside stable container. | Worker stack trace, PM2 settings, memory, cluster compatibility. |
| Running container is only unhealthy | Liveness/readiness failure, not proof of process exit. | Health log and whether Coolify or another controller restarts it. |
| New IDs with count zero | Redeployment, not Docker restart. | GitHub webhook, API/CI trigger, auto-deploy history. |

Do not treat build-only warnings, cleanup of a short-lived Coolify helper, or a proxy 502/504 after the app died as the root cause of a startup crash.

## 6. Inspect startup code

Read these files at the exact deployed commit:

1. Dockerfile `ENTRYPOINT`, `CMD`, runtime user, copied files, and runtime-stage environment.
2. Compose `command`, `entrypoint`, `restart`, healthcheck, dependencies, and resource limits.
3. Startup scripts for `set -e`, migrations, DNS calls, secret validation, file writes, and backgrounding.
4. PM2 ecosystem/CLI settings, worker count, memory restart thresholds, and `pm2-runtime` usage.
5. Health route status codes and dependency semantics.

High-priority pattern:

```text
set -e
one-shot database connection or migration without retry
server starts only after it succeeds
restart policy enabled
```

This pattern converts a transient dependency outage into a crash loop. Confirm it with runtime logs before patching.

## 7. Apply the narrow fix

- **Transient database unavailability:** add bounded retry with backoff around connection establishment. Keep permanent migration/schema errors fatal. Prefer a dedicated pre-deploy migration job when the platform workflow supports it.
- **Invalid database/auth configuration:** correct the Coolify secret or URL without printing it. Do not mask the error with infinite retries.
- **OOM:** confirm OOM evidence, then test one worker, set realistic memory limits/reservations, and profile peak use. Do not assume PM2 is the cause merely because cluster mode is enabled.
- **PM2 worker crash:** fix the worker exception. Use `pm2-runtime` as PID 1 and an ecosystem file when restart limits, backoff, or merged logs need explicit configuration.
- **Missing runtime artifact:** fix the runner-stage `COPY`, workdir, executable bit, or standalone trace. Rebuild the exact target.
- **Healthcheck only:** decide whether the endpoint is liveness or readiness. Return a non-success code only for dependencies that should remove the instance from traffic.
- **Redeploy loop:** stop the triggering webhook/API/CI loop rather than changing Docker restart policy.

Do not raise or disable the Coolify restart limit as the first fix. It suppresses protection without addressing the crash.

## 8. Verify without burning the restart limit

1. Reproduce the startup path locally or in a disposable environment with the same target and required dependencies.
2. Validate the fixed failure mode explicitly, such as delayed database availability.
3. Start one production attempt only after local/static checks pass and authorization is clear.
4. Capture state and logs from the first attempt.
5. Require stable running time beyond the previous crash window, a successful healthcheck, an internal request, and an external domain request.
6. Confirm Docker restart count and PM2 worker restart counters remain stable.
7. Record the deployed commit and Coolify resource UUID, not only the project UUID.