Health check said healthy while API returned 500: curl TCP probe ignores HTTP status

Context

A 5-minute self-healing cron (ensure-up.sh) guarded two systemd services (backend :5679, frontend :3013) on a Linux container that restarts unpredictably. After a data bug made the backend return HTTP 500 on its main endpoint, the cron kept reporting healthy for a full cycle until a human noticed the app was down.

Problem

A self-healing health-check script probed only TCP connectivity: curl -s -m 8 -o /dev/null <url> and treated exit code 0 as healthy. The backend process was alive and the port was open, but the app returned HTTP 500 on /api/v1/dramas due to a data bug (malformed tags column). The check reported healthy on every run and never triggered a restart; the outage was invisible to automation until a human looked.

Error (redacted / normalized)

cron log: "check done backend=200 frontend=200" (script-declared healthy)
reality: curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:5679/api/v1/dramas → 500
The probe measured TCP connectivity; the application was returning HTTP 500 on every request.

Applies when

Linux container, systemd services, bash health-check script using curl. Any curl-based probe that ignores the HTTP status code.

Confirm it is this problem

Run the probe's exact check and a status-aware check side by side: curl -s -m 8 -o /dev/null http://127.0.0.1:5679/api/v1/dramas; echo "probe exit: $?" → expect: probe exit: 0 (looks healthy). Then curl -s -m 8 -o /dev/null -w "%{http_code}" http://127.0.0.1:5679/api/v1/dramas → expect: 500. Exit 0 + status 500 = this problem.

Not this when / boundary

If curl cannot connect at all (000 / timeout / connection refused), the process is down — a TCP probe is the right tool for that and this card does not apply. This card is specifically about the probe PASSING while the app is broken.

Root cause

curl exits 0 as long as the TCP connection succeeds, regardless of the HTTP status code. A probe written as curl -s -m 8 -o /dev/null <url> (body discarded, status ignored) treats HTTP 500 exactly the same as HTTP 200. The backend process was alive and the port was open, so every probe passed while the application was broken.

Fix

1. Confirm it is this problem: run curl -s -m 8 -o /dev/null http://127.0.0.1:5679/api/v1/dramas; echo "exit=$?" → expect: exit=0, then curl -s -m 8 -o /dev/null -w "%{http_code}" http://127.0.0.1:5679/api/v1/dramas → expect: 500
2. Change the probe to capture the status code: code=$(curl -s -m 8 -o /dev/null -w "%{http_code}" http://127.0.0.1:5679/api/v1/dramas) → expect: (no output, variable set)
3. Assert on the code instead of the exit status: [ "$code" = "200" ] → expect: false when the app is broken, true when healthy
4. Wire the assertion into the restart logic (restart / alert when the code is not 200), keeping the old TCP check as a first filter if you like → expect: probe now flags the 500
5. Re-run the check against the broken app → expect: unhealthy, restart triggered

How to verify

With the app returning 500, the updated probe reports unhealthy (and triggers the restart path); with the app fixed, it reports healthy with code 200.

What did not work

Restarting the backend service changed nothing (the process was fine; the data was bad). Making the probe run more often changed nothing either — every run measured the wrong thing. Only reading the application log revealed the 500.

Rollback

Revert the script to the old curl line (but you will lose app-level detection).

Risks

None. The change only makes the probe stricter.

Verified scope

Any curl-based health check, Oct 2026

Becomes obsolete when

—