502s from one pod: PHP-FPM, CLI OPcache and Kubernetes under load
This is a write-up of two problems behind the same symptom, on a fleet of Laravel sites running on Kubernetes (EKS) with PHP 8.5, PHP-FPM behind nginx, and Horizon queue workers. Under heavy use, some requests started coming back as 502 Bad Gateway. Most pods were fine, and a few weren't.
It turned out to be two unrelated problems that look the same from the outside:
- Nodes running out of memory, because of OPcache in CLI processes.
- Single pods whose PHP-FPM workers kept crashing, while the pod still looked healthy to Kubernetes.
Problem 1: CLI OPcache eats the node
The queue-worker nodes kept climbing to over 90% memory and going NotReady. When a node goes NotReady, Kubernetes starts moving everything off it, including unrelated web pods that happened to share the node. Those show up as 502s and 503s until they're running again somewhere else.
Each worker pod used 1.3–1.6 GB, and about 1 GB of that was OPcache.
Why
OPcache keeps compiled PHP in shared memory. For PHP-FPM that's a great deal: the master process creates one segment and every worker of the pool uses it. Fifty workers, one cache.
The CLI has no master process. With opcache.enable_cli = 1, every CLI process creates its own private
segment, compiles the code into it and shares it with nobody. A typical Laravel worker pod runs several CLI
processes:
- the Horizon supervisor,
- each Horizon worker process,
schedule:work, plus a freshschedule:runevery minute.
That was about 11 processes per pod, each with its own 85–93 MB cache: roughly a gigabyte per pod, spent on caches that can't be shared.
The fix
before: 11 CLI processes × ~90 MB OPcache ≈ 1 GB per worker pod
after: opcache.enable_cli = 0 ≈ 0, php-fpm keeps its one shared cache
[opcache]
; php-fpm: one shared cache for every worker of the pool. Keep it, and the JIT.
opcache.enable = 1
opcache.jit = tracing
opcache.jit_buffer_size = 100M
; CLI: off. Every CLI process (queue workers, the queue supervisor,
; schedule:work, each schedule:run) would build its own private copy of the
; compiled code in its own shared memory segment. Nothing is shared between them.
opcache.enable_cli = 0
; If one php.ini serves both, and you do want CLI OPcache, give the CLI a small
; cache instead in a CLI-only ini file, e.g.:
; opcache.enable_cli = 1
; opcache.memory_consumption = 64
; opcache.jit_buffer_size = 0
- Turn OPcache off for the CLI. Keep it, and the JIT, on for PHP-FPM, where one cache serves every worker.
- It costs almost nothing. Horizon workers are long-running, so without OPcache they compile each file once per process and then keep it in normal memory for their whole life. The win was never there in the first place.
- If you really want it for the CLI, lower it instead of switching it off: a separate CLI ini with a small
opcache.memory_consumptionand no JIT buffer. - Your health checks may need adjusting. If something checks that the JIT is enabled from a CLI script, it
will now fail. Check with
php -d opcache.enable_cli=1 …, which still proves that the settings PHP-FPM uses are valid.
After the change the worker pods dropped by about a gigabyte each, and the nodes stopped going NotReady.
Problem 2: one pod quietly returning 502s
The second problem was nastier, because it looked like everything was fine. On one pod of a site, PHP-FPM workers kept dying:
WARNING: [pool www] child 1234 exited on signal 11 (SIGSEGV)
nginx in that pod answered 502 for requests that hit a dying worker. But the readiness probe (/up) kept
passing, because enough workers were alive to answer it. So the pod stayed Ready, the Service kept sending it a
share of traffic, and for one site about 22% of requests failed. Every other pod running the same image was
fine. Deleting the bad pod fixed it every time; a fresh pod didn't have the problem.
A probe that only proves "something on this pod can answer one request" can't catch "a fifth of the requests on this pod fail". There were two fixes: one hides the failure from users straight away, and the other removes the pod.
Fix 1: let the ingress retry on another pod
ingress-nginx can retry a failed request on a different pod. By default it retries only connection errors and
timeouts. Adding http_502 covers this case:
# ingress-nginx Helm values
controller:
config:
# The default retries only connection errors and timeouts. Adding http_502
# sends the request to another pod when one answers 502.
proxy-next-upstream: "error timeout http_502"
# Total attempts, including the first (3 is the default).
proxy-next-upstream-tries: "3"
# nginx never retries POST/PATCH/LOCK once the pod received them
# (non_idempotent is not in the list above), so a write is never sent twice.
nginx won't retry non-idempotent requests (POST, PATCH) once the pod has received them, so a form post is never sent twice. Those still fail, but every GET that lands on the bad pod now quietly succeeds on a healthy one.
Fix 2: evict the pod that keeps failing
Retries hide the problem, so you also need to get rid of the cause. A small controller now runs every minute:
- It reads the ingress access log (from Loki) and counts the 502s per pod over the last 5 minutes.
- If a pod is over the threshold (20 in 5 minutes) and another Ready pod of the same Deployment is under it, it evicts the bad pod through the Kubernetes Eviction API. The Deployment starts a fresh one.
The guardrails matter more than the eviction:
- Nothing is evicted when every pod is failing. That's the app or a dependency, and evicting pods would only make it worse.
- One eviction per Deployment per 10 minutes, and at most two per run.
- Only 502s logged after the pod started count. Pod IPs get reused, so an old pod's errors mustn't be blamed on its replacement.
- It uses the Eviction API, not delete, so PodDisruptionBudgets are respected.
- A retried request still counts. Its access-log line shows the first pod's failure (
$upstream_statusreads502, 200), so the bad pod is found even when no user saw an error. - Alerts go out when a pod is evicted, when one is over the threshold but couldn't be evicted (and why), and when the 502 rate that clients actually see goes up.
Where it stands
| Symptom | Cause | Fix |
|---|---|---|
| Nodes NotReady, pods rescheduled, 502/503 | OPcache in every CLI process, ~1 GB per worker pod | opcache.enable_cli = 0 (or a small CLI-only cache) |
| One pod answers 502 while staying Ready | PHP-FPM workers crash with SIGSEGV on that pod | ingress retries http_502 on another pod; auto-evict pods that keep failing |
Still open:
- A real PHP-FPM liveness probe. Check the FPM status or ping endpoint rather than a single page, so a pod with crashing workers gets restarted by Kubernetes itself.
- The crash itself. It only ever happened on single, freshly started pods, and it went away with a new pod. The
OPcache JIT is the prime suspect, but that's not confirmed yet. If you see
SIGSEGVin PHP-FPM logs, also make sure you're on a patched PHP: crashing workers are exactly what php2shell probing looks like on an unpatched runtime.