Kubernetes Dynamic Resource Allocation: Scheduling GPU Inference for Spring Boot AI Microservices
Kubernetes 1.34 promoted Dynamic Resource Allocation to stable, replacing device plugins for GPU scheduling. A walkthrough of migrating a Spring Boot fraud-scoring service from nvidia.com/gpu limits to ResourceClaims, with a working manifest and health/readiness wiring for shared GPU pools.
I spent a Friday evening in September staring at a GPU utilization graph that made no sense. Four nodes, each with a single A10G, each pinned to one fraud-scoring pod through nvidia.com/gpu: 1, and every single one of them sitting at around 20% utilization during peak transaction windows. The pods weren't starved — they were just small. A risk model that scores a card transaction in eleven milliseconds doesn't need an entire accelerator to itself, but the device plugin model Kubernetes shipped for years only understood whole units. You asked for a GPU, you got a GPU, and whatever you didn't use sat there idle while the node next to it queued pods waiting for a device that would never free up because the first pod was never going away.
That's the problem Dynamic Resource Allocation was built to solve, and as of Kubernetes 1.34 it's no longer an alpha feature you enable with a feature gate and pray — it's stable, it's in the default API surface, and if you're running any kind of shared GPU pool for inference workloads, it changes how you should be writing deployment manifests starting now. I want to walk through what actually changed, why it matters more in fintech ML pipelines than almost anywhere else, and how I migrated a Spring Boot inference sidecar off device plugins without taking down the service that was scoring live transactions while I did it.
Why Device Plugins Were Never Going to Scale for Inference
The device plugin API, which has been the standard way to expose GPUs to Kubernetes since around 1.10, treats an accelerator as an opaque, indivisible resource. You declare nvidia.com/gpu: 1 in your pod spec, the kubelet asks the device plugin for an available device, and the scheduler places your pod on a node that has one free. That model works fine when your workload is a training job that genuinely wants the whole card for six hours. It falls apart for inference, where a single fraud-scoring call, a risk-tiering lookup, or a document-classification pass against a small transformer model might need a fraction of a GPU's memory and compute for a few milliseconds at a time.
The workarounds everyone reached for — NVIDIA's MPS (Multi-Process Service), time-slicing configs in the device plugin, or just running multiple model replicas crammed onto one physical card and hoping the scheduler didn't notice — all shared the same flaw. None of them let the scheduler itself reason about fractional or shared GPU capacity. The device plugin told Kubernetes "this node has one GPU, it's either taken or it isn't." Everything past that point was outside the scheduler's view: memory partitioning, MIG slice selection, which workload got priority when two pods wanted the same physical die. You ended up hand-rolling bin-packing logic in Helm values and node labels, which is exactly the kind of fragile, tribal-knowledge configuration that breaks the week after the person who wrote it leaves.
DRA inverts that. Instead of a resource being a static integer the kubelet counts, it's a claim — a first-class API object the scheduler evaluates alongside CPU and memory, with a driver underneath that understands the actual topology of the hardware: how many MIG slices exist, how much memory each one has, which slices are already claimed by another pod. The scheduler doesn't just see "GPU: yes/no." It sees a ResourceClaim with constraints, and it can place pods based on what's structurally available, not what a label said was available three deployments ago.
The Core Objects: ResourceClaim, DeviceClass, ResourceClaimTemplate
Three API objects matter here, and they map fairly cleanly onto concepts Spring Boot engineers already know from dependency injection, which is the analogy I used when I explained this to my team.
A DeviceClass is roughly a bean definition — it describes a category of hardware (an A10G, an H100 MIG slice, a specific driver version) and the selection rules for matching one. A ResourceClaim is the actual injected dependency — a concrete request for a device matching that class, scoped to a namespace, that the scheduler resolves at pod placement time. A ResourceClaimTemplate is what you use when you want every replica of a Deployment to get its own claim stamped out from the same template, rather than manually writing a ResourceClaim per pod.
Here's the DeviceClass and ResourceClaimTemplate I used for the fraud-scoring GPU pool, which runs on a node group with four A10G cards, each split into two MIG-equivalent memory partitions so two inference pods can share a physical card without stepping on each other's memory:
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: fraud-scoring-gpu-slice
spec:
selectors:
- cel:
expression: |
device.driver == "gpu.nvidia.com" &&
device.attributes["gpu.nvidia.com/memory"].quantity >= quantity("12Gi")
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
name: fraud-scorer-gpu-claim-template
namespace: risk-inference
spec:
spec:
devices:
requests:
- name: inference-slice
deviceClassName: fraud-scoring-gpu-slice
count: 1
That cel expression is doing real work — it's filtering candidate devices by an attribute the DRA driver publishes about actual available memory on the partition, not a label someone set by hand. If the node's driver reports a slice has less than 12Gi free because another claim already took the rest, the scheduler skips it, no manual bin-packing required.
Migrating the Spring Boot Deployment
The service in question is a Spring Boot 3.x app that doesn't run the model itself — it calls into a Triton inference server sidecar over gRPC in the same pod, which is a pattern I'd recommend over embedding a model runtime directly in the JVM process almost every time, since it keeps the Java heap and the GPU memory management as two separate failure domains you can reason about independently. The old pod spec requested the GPU directly on the sidecar container with nvidia.com/gpu: 1. The new one replaces that resource limit with a claim reference:
apiVersion: apps/v1
kind: Deployment
metadata:
name: fraud-scoring-service
namespace: risk-inference
spec:
replicas: 4
selector:
matchLabels:
app: fraud-scoring-service
template:
metadata:
labels:
app: fraud-scoring-service
spec:
resourceClaims:
- name: inference-gpu
resourceClaimTemplateName: fraud-scorer-gpu-claim-template
containers:
- name: risk-api
image: registry.internal/risk/fraud-scoring-api:2.4.1
ports:
- containerPort: 8080
resources:
requests:
cpu: "500m"
memory: "768Mi"
limits:
cpu: "1"
memory: "1Gi"
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
- name: triton-sidecar
image: nvcr.io/nvidia/tritonserver:25.09-py3
resources:
claims:
- name: inference-gpu
request: inference-slice
ports:
- containerPort: 8001
readinessProbe:
grpc:
port: 8001
initialDelaySeconds: 10
periodSeconds: 5
Notice the claims field under resources on the Triton container — that's the mechanism that binds a specific container to a specific named request inside the pod-level claim, which matters once you have more than one GPU-aware container in a pod and need to be explicit about who gets what. The Spring Boot container itself requests no GPU resources at all; it never touches the accelerator directly, it just needs the sidecar to be alive and the model loaded before it starts routing traffic to it.
Wiring Readiness So the JVM Doesn't Lie About Being Ready
The failure mode I actually hit during migration wasn't scheduling — DRA placed pods correctly from the first rollout. It was a readiness gap: Spring Boot's default /actuator/health would report the app as healthy the moment the embedded Tomcat context finished starting, which happened well before the Triton sidecar had finished loading the model weights into the claimed GPU memory. Under device plugins this race existed too, but it was shorter, because the GPU was guaranteed exclusive and model load was fast and predictable. Under a shared, claim-based GPU, load time became more variable — if another pod's claim was mid-teardown on the same physical card, the driver sometimes took an extra second or two to hand off the freed memory region.
I added a custom HealthIndicator that actually checks the sidecar's readiness before reporting the Spring Boot container ready, rather than trusting Kubernetes to sequence the two containers correctly on its own:
@Component
public class InferenceSidecarHealthIndicator implements HealthIndicator {
private final GrpcHealthClient tritonHealthClient;
private final AtomicBoolean modelConfirmedLoaded = new AtomicBoolean(false);
public InferenceSidecarHealthIndicator(GrpcHealthClient tritonHealthClient) {
this.tritonHealthClient = tritonHealthClient;
}
@Override
public Health health() {
if (modelConfirmedLoaded.get()) {
return Health.up().withDetail("model", "fraud-risk-v3").build();
}
try {
ModelReadyResponse response = tritonHealthClient.checkModelReady("fraud-risk-v3");
if (response.isReady()) {
modelConfirmedLoaded.set(true);
return Health.up().withDetail("model", "fraud-risk-v3").build();
}
return Health.down()
.withDetail("reason", "gpu-model-not-loaded")
.withDetail("gpuMemoryBound", response.getMemoryBindingState())
.build();
} catch (StatusRuntimeException ex) {
// Sidecar unreachable usually means the claimed device hasn't
// finished binding to this pod yet — treat it as not-ready,
// not as an error, so the probe keeps retrying instead of
// tripping a restart loop on a pod that's just waiting its turn.
return Health.down().withDetail("reason", "sidecar-unreachable").build();
}
}
}
I cache the "ready" result once it's confirmed true, because polling Triton's model-ready endpoint on every readiness check once the model is loaded is pointless latency on the hot path — the thing you're actually worried about is the cold-start window, not steady state. This indicator feeds the readiness group in application.yml:
management:
endpoint:
health:
group:
readiness:
include: readinessState, inferenceSidecar
health:
inferenceSidecar:
enabled: true
With that wiring in place, the pod doesn't get traffic routed to it by the Service until the GPU-backed sidecar has confirmed the model is actually bound into claimed device memory — which is the part device plugins gave you for free through exclusivity, and DRA makes you earn through an explicit readiness contract, because exclusivity is exactly the thing you gave up to get the sharing.
Why This Is a Bigger Deal in Fintech Than Fintech People Think
The utilization story is the obvious win and it's real — we went from four A10Gs running at roughly 20% to two cards running two fraud-scoring replicas each at closer to 55-60%, which let us drop the other two nodes from the pool and redirect that spend toward a second model doing document-fraud classification on uploaded KYC images, work that previously would have needed its own dedicated node group to justify the GPU cost. That's the kind of unit-economics argument a platform team can take to a budget review without hand-waving.
But the quieter win, and the one that matters more for anyone running regulated inference pipelines, is auditability. I've written before about DORA and NIS2 compliance evidence with policy as code, and one of the recurring headaches in that work is proving, after the fact, exactly what compute resource served a given model inference for a given transaction — which matters when an auditor asks why two models running on shared hardware didn't cross-contaminate state. Device plugin allocation left almost no trail; the kubelet handed out a device and nobody recorded which logical workload got which physical card beyond whatever you manually scraped from nvidia-smi logs. ResourceClaim objects are real, queryable, versioned API resources with a status field showing exactly which device was allocated, when, and to which pod. That's an audit trail that didn't exist before, sitting right there in kubectl get resourceclaims -o yaml, for free.
The Migration Checklist I Actually Used
If you're running Kubernetes 1.34 or later and still have device-plugin-based GPU requests in production, here's roughly the order I'd do this in, because doing it out of order is how you take down a live inference path:
First, confirm your GPU driver vendor ships a DRA driver, not just a device plugin — NVIDIA's k8s-dra-driver needs to be running as a DaemonSet before any DeviceClass will resolve to real hardware; the API objects will create fine without it, they just won't ever bind. Second, stand up the DeviceClass and ResourceClaimTemplate in a non-production namespace first and watch kubectl describe resourceclaim to confirm devices are actually being attributed correctly against your real hardware topology, not just scheduling optimistically. Third, migrate one replica of a Deployment at a time by running old-style and new-style pods side by side behind the same Service, rather than cutting the whole Deployment over in one rollout — DRA's scheduling behavior under load is different enough from device plugins that I wanted to watch real traffic hit a DRA-scheduled pod before trusting it with the full fleet. Fourth, add the readiness wiring above before you need it, not after a page at 2 AM tells you a pod took traffic before its model was bound.
None of this is especially exotic once you've done it once, but it's new enough that documentation is thin and most of what's written about DRA online is about training workloads, not low-latency inference services sitting behind a JVM. If you're building out this kind of shared-GPU pattern for a fintech ML pipeline and want to talk through the specifics, my experience running production inference infrastructure and project work in this space are good starting points, or just reach out directly.
Related Articles
- DORA and NIS2 Policy as Code: Compliance Evidence for Spring Boot and Kubernetes
- Observability and Guardrails for Agentic AI Backends in Spring Boot
- Building an MCP Server with Spring Boot and Spring AI
Pull up your own GPU utilization dashboard before you touch any of this — if your cards are already running above 70% under device plugins, DRA won't move the needle much and the migration risk isn't worth it yet; if they're sitting where mine were, start with the DeviceClass in a staging namespace this week.