The 256Mi limit that took checkout down on payday
- Impact
- 18% of checkout requests failed for 41 minutes during peak traffic.
- Symptom
- One replica of checkout-api sat in CrashLoopBackOff while the other stayed healthy, so dashboards showed a partial outage that looked like a network blip.
- Root cause
- A connection-pool bump raised memory use past the container's 256Mi limit. The pod was OOMKilled (exit 137), restarted, warmed its pool, and died again — a loop that only triggered under payday load.
What it teaches
Requests and limits are capacity decisions, not defaults to copy. Alert on OOMKill events directly; error-rate alone hid which pod was at fault.