Pod lifecycle mistakes rarely look dramatic in YAML. They show up as rollout stalls, connection resets, stuck draining, traffic going to the wrong pod, or a service that passes liveness but still is not truly ready.
That is why this topic matters. The question is not “should I add a readiness probe?” The real question is whether the platform and the application agree on when a pod is safe to start receiving traffic and when it has fully stopped being part of the serving path.
Quick Summary
| Mechanism | What it should mean | Common misuse |
|---|---|---|
startupProbe |
the process is not ready for health judgment yet | omitted for slow-starting apps, causing liveness kills during boot |
readinessProbe |
the pod can safely accept new traffic now | treated as a generic health check instead of a serving gate |
livenessProbe |
the process is unhealthy enough to restart | used for dependency checks and causing restart storms |
preStop |
give shutdown logic time to stop accepting work | used as a magical sleep with no real drain behavior |
terminationGracePeriodSeconds |
upper bound for graceful exit | set shorter than real request or drain time |
The core rule is simple: readiness is about traffic admission, liveness is about recovery, and graceful termination is about stopping new work before killing the old process.
What This Post Is Really About
Most teams do not fail because they forgot probes exist. They fail because these parts are configured independently:
- application startup behavior
- probe thresholds
- ingress or service endpoint removal
- long-running request handling
- shutdown hooks
Kubernetes then does exactly what the manifests say, even if the application lifecycle story is incomplete.
That is why pod lifecycle design should be treated as one end-to-end contract, not five unrelated fields.
A Better Mental Model
Think about pod lifecycle in four stages:
flowchart LR
A[Process booting] --> B[Ready to receive traffic]
B --> C[Serving steady-state traffic]
C --> D[Draining and shutting down]
Each stage needs a different signal. Trying to use one probe for all four stages is where trouble begins.
Startup, Readiness, and Liveness Do Different Jobs
startupProbe
This exists to protect slow-starting applications from premature liveness failure. If the service needs time for:
- JIT warmup
- large cache initialization
- schema checks
- plugin discovery
then startup should absorb that behavior explicitly.
Without it, liveness can kill a process that was still booting correctly.
readinessProbe
Readiness should answer one narrow question: is this pod safe to receive new traffic right now?
That is different from:
- “the JVM is alive”
- “the health endpoint returns 200”
- “the database is reachable in this instant”
A good readiness signal reflects serving ability, not generic process existence.
livenessProbe
Liveness is a restart decision. That is a strong action. Use it when the process is stuck or irrecoverably unhealthy, not merely because a downstream dependency is slow.
If a database hiccup causes liveness failure, Kubernetes can turn a dependency issue into a restart storm.
Graceful Termination Is a Serving Contract
Graceful termination is about more than receiving SIGTERM.
The application must stop accepting new work quickly enough that the platform can remove it from rotation before hard kill.
That often means:
- fail readiness or stop advertising availability
- stop accepting new requests
- finish or hand off in-flight work
- exit before the grace period expires
If the app keeps accepting traffic after SIGTERM, termination is not really graceful even if the process exits politely.
Why preStop Is Often Misused
preStop is useful when it supports a real drain strategy.
It is not a universal substitute for application shutdown logic.
Bad pattern:
- sleep for a few seconds
- hope the load balancer catches up
Better pattern:
- flip readiness quickly
- let the application stop taking new work
- use
preStoponly to help coordinate timing around that behavior
If the app itself has no drain behavior, a sleep hook is usually theater.
A Practical Baseline Manifest
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-api
spec:
template:
spec:
terminationGracePeriodSeconds: 45
containers:
- name: app
startupProbe:
httpGet:
path: /health/startup
port: 8080
failureThreshold: 30
periodSeconds: 2
readinessProbe:
httpGet:
path: /health/ready
port: 8080
periodSeconds: 5
livenessProbe:
httpGet:
path: /health/live
port: 8080
periodSeconds: 10
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 5"]
That manifest is not a universal answer. It is a reminder that each endpoint must map to a real lifecycle meaning inside the application.
Failure Modes That Keep Showing Up
Readiness says “yes” before the app is truly warm
This causes cold pods to receive production traffic too early.
Liveness checks downstream dependencies
Now a remote outage restarts healthy pods instead of preserving scarce capacity.
Grace period shorter than request reality
Long uploads, batch handlers, or queue drains do not finish in time and get cut off mid-flight.
No operator visibility into drain behavior
Teams know a pod terminated, but not whether it stopped traffic first, finished inflight requests, or got killed at grace-period expiry.
A Better Rollout Drill
Before trusting pod lifecycle settings, test them under real-ish load:
- send steady traffic
- trigger a rollout or pod deletion
- watch endpoint removal timing
- observe whether new requests still reach the draining pod
- verify whether in-flight work finishes cleanly
If the only validation is “the rollout completed,” the lifecycle policy is under-tested.
What the First Dashboard Should Show
At minimum, expose:
- readiness transitions
- liveness restarts
- pod termination reason
- in-flight request count during shutdown
- request failures during rollout windows
- duration between
SIGTERMand process exit
That is what lets operators answer: “Did the pod fail, or did our lifecycle contract fail?”
Key Takeaways
- Startup, readiness, and liveness are different decisions and should stay different.
- Graceful termination means stop receiving new work before the process dies.
preStopis helpful only when it supports a real drain strategy.- Pod lifecycle settings should be validated under load, not only by successful deployment completion.
Categories
Tags