Kubernetes 1.37 gang scheduling beta for AI training

Kubernetes 1.37 Garhwal, released August 26, delivers gang scheduling (KEP-4671) in beta, finally giving AI training workloads a native way to avoid the partial-placement deadlock that wastes GPU resources. But there's a catch: the feature is disabled by default and requires explicit configuration. For teams running distributed training on Kubernetes, this release also brings HPA scale-to-zero (enabled by default) and stable Dynamic Resource Allocation device taints, alongside a planned migration from IPVS to nftables.

Gang scheduling beta: all-or-nothing pod placement for training jobs

Gang scheduling guarantees that a named PodGroup is only scheduled when at least minCount pods can be placed simultaneously. If the cluster cannot seat that many, none are bound, and the job waits cleanly in the unschedulable queue rather than consuming partial resources. This prevents the common failure where a distributed training job requiring 64 GPUs across eight nodes gets half its pods placed, then stalls because other workloads have claimed the remaining nodes. The partially scheduled pods sit idle, wasting GPU capacity.

The feature is defined via PodGroup and Workload objects in the scheduling.k8s.io/v1alpha3 API group. Operators configure placement guarantees using a minCount field. For elastic training jobs, minCount can be set lower than the total desired pod count, allowing progress with a subset of workers. The field is mutable, so you can adjust it without recreating the PodGroup for already-running jobs.

HPA scale-to-zero and DRA device taints cut GPU costs

HPA scale-to-zero graduated to Beta and is enabled by default for workloads that set spec.minReplicas: 0. It allows a HorizontalPodAutoscaler to drive replica counts to zero when demand disappears and restore them when demand returns. For GPU-backed inference endpoints and batch processing queues, the potential cost reduction is significant: an inference service that sits idle overnight can fully release its GPU capacity. The feature works only with external or object metrics (e.g., queue depth, Prometheus custom metrics) because CPU and memory metrics require live pods to measure. Cold-start latency from zero is the primary tradeoff to evaluate before enabling scale-to-zero for latency-sensitive workloads.

Dynamic Resource Allocation (DRA) device taints and tolerations (KEP-5055) graduated to Stable. This mirrors the node taint model: a degraded or misbehaving GPU can be marked with a device taint, flagging it as unavailable for new workloads without removing it from the cluster or draining its entire node. Administrators can also apply taints based on which driver manages a device, enabling targeted exclusion during driver maintenance windows.

nftables migration and other changes

Kubernetes 1.37 continues the migration from iptables and IPVS toward nftables for cluster traffic handling. IPVS mode is formally deprecated (KEP-5495), with planned disablement in Kubernetes 1.40 and full removal in Kubernetes 1.43. Clusters that do not explicitly configure a kube-proxy mode will receive deprecation warnings as of v1.37. nftables supports incremental rule updates, offering significantly lower latency in large clusters with thousands of services.

Two security features reached Stable: Pod Certificates (KEP-4317) allow pods to request short-lived X.509 certificates directly from the Kubernetes API, and ClusterTrustBundles (KEP-3257) provide a cluster-scoped mechanism to distribute trust anchors to pods through projected volumes. Together, they make mutual TLS between services possible without external tooling for basic use cases.

What to do before upgrading

Test gang scheduling in non-production environments first. Enable the GenericWorkload feature gate and the scheduling.k8s.io/v1alpha3 API group. Audit your kube-proxy mode configuration; if no mode is explicitly configured, deprecation warnings will appear. Evaluate nftables migration before Kubernetes 1.40 to avoid forced migration under pressure. Review DRA device taint policies for clusters with heterogeneous or aging GPU hardware. For inference serving, set spec.minReplicas: 0 and configure external metrics to drive scale-up, but measure cold-start latency first.

Gang scheduling is Beta and disabled by default. External schedulers like Volcano and Kueue still add multi-tenant quota management, fair-sharing, and queue borrowing that native gang scheduling does not provide. HPA scale-to-zero relies on external metrics, and cold-start latency is a real tradeoff. The IPVS removal timeline gives operators time to migrate, but the clock is ticking.

FAQs

Gang scheduling guarantees all-or-nothing pod placement for a named PodGroup, preventing the partial-placement deadlock that wastes GPU resources in distributed training. When the cluster cannot seat at least minCount pods from the group simultaneously, none are bound, and the job waits cleanly in the unschedulable queue rather than consuming partial resources. This is critical for AI training jobs that require all workers to start together.