Kubernetes v1.36 Enhances Memory Management with Tiered Protection and Opt-In Reservations

Introduction

Kubernetes continues to refine its memory management capabilities with the latest v1.36 release, which brings significant updates to the Memory QoS feature (still in alpha). Originally introduced in v1.22 and refined in v1.27, this feature leverages the cgroup v2 memory controller to provide the kernel with clearer guidance on how to handle container memory. The v1.36 enhancements focus on giving administrators more granular control over memory allocation and protection, reducing the risk of out-of-memory (OOM) kills while maintaining performance. This article explores the key additions: opt-in memory reservation, tiered protection based on Quality of Service (QoS) classes, new observability metrics, and a warning for older kernel versions.

What's New in v1.36

The latest iteration of Memory QoS introduces several pivotal changes that allow cluster operators to fine-tune memory behavior. The most notable is the separation of memory throttling from memory reservation, along with a tiered protection scheme that aligns with Kubernetes QoS classes. Additionally, new metrics offer deeper visibility into memory operations, and a kernel version warning helps avoid configuration pitfalls.

Opt-In Memory Reservation with memoryReservationPolicy

Prior to v1.36, enabling the Memory QoS feature gate automatically set memory reservations for all containers with memory requests using memory.min. This approach could lead to excessive hard reservations, especially for BestEffort or even Burstable pods, potentially starving system daemons or critical workloads. In v1.36, the throttling mechanism (using memory.high) remains enabled by default when the feature gate is active, but memory reservation is now controlled by a separate kubelet configuration field: memoryReservationPolicy.

• None (default): No memory.min or memory.low is written. Only throttling via memory.high is applied, allowing administrators to test throttling behavior before committing to hard or soft reservations.
• TieredReservation: The kubelet writes tiered memory protection based on the pod's QoS class. This policy ensures that Guaranteed pods receive hard protection, Burstable pods get soft protection, and BestEffort pods remain fully reclaimable.

Tiered Protection by QoS Class

The TieredReservation policy maps memory requests to appropriate cgroup v2 parameters depending on the pod's QoS class:

• Guaranteed Pods: Receive hard protection via memory.min. The kernel will not reclaim this memory under any circumstances. If the guarantee cannot be honored, the OOM killer is triggered on other processes to free pages. For example, a Guaranteed pod requesting 512 MiB results in memory.min set to 536870912 (512 MiB in bytes).
• Burstable Pods: Get soft protection via memory.low. Under normal memory pressure, the kernel avoids reclaiming this memory, but may reclaim it if necessary to prevent a system-wide OOM. The same 512 MiB request on a Burstable pod sets memory.low to the same value.
• BestEffort Pods: Receive neither memory.min nor memory.low. Their memory is fully reclaimable, making them the first candidates for eviction under pressure.

This tiered approach prevents the over-reservation that plagued earlier versions. Guaranteed pods are truly protected, Burstable pods have a safety net, and BestEffort workloads are flexible, improving overall cluster resilience.

Comparison with v1.27 Behavior

In Kubernetes v1.27, enabling the Memory QoS feature gate immediately wrote memory.min for every container with a memory request, regardless of QoS class. This could be problematic on nodes with high aggregate requests. Consider a node with 8 GiB of RAM where Burstable pod requests total 7 GiB. Under v1.27, all that 7 GiB would be locked as memory.min, leaving minimal headroom for the kernel, system daemons, or BestEffort workloads, increasing OOM risk.

With v1.36's TieredReservation, those Burstable requests use memory.low instead of memory.min. Under normal pressure, the memory is still protected, but under extreme conditions, the kernel can reclaim part of it to avoid system-wide OOM. Only Guaranteed pods use memory.min, which keeps the total hard reservation lower. The memoryReservationPolicy field allows operators to enable throttling first (with None) to observe workload behavior, then opt into reservation once node headroom is confirmed.

Observability Metrics

v1.36 exposes two new alpha-level metrics on the kubelet /metrics endpoint, providing visibility into the memory QoS mechanisms:

• kubelet_memory_qos_node_memory_min_bytes: Total amount of memory reserved via memory.min across all pods on the node.
• kubelet_memory_qos_node_memory_low_bytes: Total amount of memory reserved via memory.low across all pods on the node.

These metrics allow administrators to monitor how much memory is protected, differentiate between hard and soft reservations, and adjust policies accordingly. For example, if the memory.min total approaches node capacity, operators might reconsider request limits or adjust the memoryReservationPolicy.

Kernel Version Warning for memory.high

An important addition is a warning when the kernel version does not support memory.high properly. The memory.high parameter, used for throttling, behaves differently on older kernels, particularly those before version 5.4. Kubelet now logs a warning if it detects an unsupported kernel, reminding administrators to upgrade to ensure Memory QoS works as intended.

Conclusion

Kubernetes v1.36 marks a significant step forward in memory management for containerized workloads. By separating throttling from reservation and introducing tiered protection aligned with QoS classes, the update minimizes the risk of OOM kills while allowing flexible resource allocation. The opt-in memoryReservationPolicy gives operators a safe way to introduce memory protection gradually, and the new metrics provide the data needed to make informed decisions. As the Memory QoS feature matures, these improvements bring Kubernetes closer to providing robust, production-ready memory guarantees for all pod types.