Kubernetes sees infrastructure objects
Pods. Replicas. Nodes. Device assignments. Placement and health.
About / Why just Kubernetes is not enough?
Why just Kubernetes is not enough.
Kubernetes is excellent at managing infrastructure. But pods, replicas, and GPU assignments are too coarse to be the complete resource model for modern LLM inference.
About / The abstraction mismatch
A pod tells you where a process runs.
It does not tell you how much useful inference capacity is available right now.
Inference depends on live state
Model residency. KV cache. Memory pressure. Compute and bandwidth. Request mix and cache locality.
The problem is not Kubernetes. The problem is asking Kubernetes objects to describe live inference state they were never designed to own.
About / Why pod-level scaling is too coarse
More pods do not automatically mean more useful capacity.
Scaling can be slow
A new worker can require scheduling, model loading, engine initialization, and warming before it is useful.
Routing does not create capacity
A router can choose among existing endpoints. It cannot by itself reshape the memory, compute, or state inside them.
Replacement can throw work away
Replacing a worker can lose valuable local cache and model residency, forcing reload or recomputation.
About / The missing control layer
Inference needs decisions closer to the live model.
See what is actually happening
Understand model state, cache state, resource pressure, and the shape of incoming work.
Coordinate routing with resources
The best destination depends on more than endpoint health. It depends on the state and capacity behind that endpoint.
Keep adapting
Inference capacity changes after startup. The control layer must keep responding as traffic and state change.
That control layer can run on Kubernetes. It just cannot be reduced to Kubernetes alone.
About / What Kubernetes should keep doing
Kubernetes still has an essential job.
Nodes and placement
Put durable runtime infrastructure on the right machines and failure domains.
Networking, identity, and policy
Provide the enterprise infrastructure controls teams already know how to operate.
Deployment and recovery
Handle rollout, restart, cluster operations, and coarse failure recovery.
Coarse device allocation
Assign accelerators and infrastructure resources to the runtime layer that will use them.
Kubernetes should remain the infrastructure manager. A specialized inference layer should manage the live models, state, routing, and resource economics inside and across those allocations.
About / The servescale.ai view
Use Kubernetes. Don’t stop at Kubernetes.
servescale.ai is building around this division of responsibility: Kubernetes manages infrastructure; the inference layer manages models, state, routing, resource pressure, and economics.
The result is a cleaner operating model: keep the infrastructure platform enterprises already trust, and add the inference-specific control that modern model serving requires.