Memory management plays a crucial role in ensuring optimal performance and resource efficiency in Azure Kubernetes Service (AKS). With the release of AKS 1.29, significant changes have been introduced to improve memory allocation and pod stability. In this blog, we will explore these changes and provide practical insights on how to optimize memory management in AKS 1.29. But first lets go through the basics of resource management in k8s.
In Kubernetes you can define CPU and memory (RAM) requirements for each container in a Pod. Resource Requests indicate the minimum amount of a resource that a container needs to run properly. The kube-scheduler uses this information to assign Pods to suitable nodes. Resource Limits, set by users, these limits prevent containers from consuming more resources than allocated. They are enforced to maintain fairness and resource efficiency within the cluster. Read more in kubernetes docs
If a node has surplus resources, a container might use more than its requested amount, subject to its defined limit. Consider a container with a 4GiB memory limit. If the container attempts to exceed this limit, the kubelet intervenes. This can result in process terminations within the container, often manifested as Out of Memory (OOM) errors.
However Kubernetes also includes eviction policies managed by the kubelet. These policies trigger Pod eviction if the node's available resources fall below specified thresholds. For instance, if the node's available memory dips below a defined threshold, the kubelet may evict Pods to reclaim resources, even though they are within their resource limit which brings us to the topic of this post.
Challenges of Memory Management Pre-AKS 1.29:
In previous versions of AKS, memory management posed challenges due to a higher eviction threshold and a complex reservation formula. These factors often led to inefficient resource allocation and pod instability.
Higher Eviction Thresholds: Nodes would start evicting pods when the available memory dipped below 750 MiB, a mechanism to prevent system instability due to low memory.
Complex Memory Reservation: A regressive formula determined the amount of memory reserved for system processes and Kubernetes components, which impacted the allocatable memory for pods.
Exploring the Changes in AKS 1.29:
AKS 1.29 introduced significant changes in memory management, optimizing how resources are allocated and used.
Reduced Eviction Threshold: The new threshold is set at
memory.available<100Mi, allowing for more efficient resource allocation and better pod stability.Simplified Memory Reservation: The new formula for
kube-reservedmemory is the lesser of20MB * Max Pods + 50MBor25% of the total system memory. This change aims to simplify and potentially increase the allocatable memory.
Example for an 8GB VM with 30 Max Pods:
Previous AKS versions
Eviction Threshold:
memory.available<750MiThe regressive formula typically reserved:
25% of the first 4GB = 1GB (1024 MiB)
20% of the next 4GB (4GB to 8GB) = 0.8GB (819.2 MiB)
Total Reserved: 1024 MiB + 819.2 MiB = 1843.2 MiB
Allocatable Memory:
8GB - 1.8432GB - 0.75GB ≈ 5.41GB
AKS 1.29
Eviction Threshold:
memory.available<100MiMemory Reservation:
20MB * 30 + 50MB = 650MB(approximately 0.65GB)Allocatable Memory:
8GB - 0.65GB - 0.1GB = 7.25GB
Example for a 4GB VM with 70 Max Pods:
Previous AKS versions
Eviction Threshold:
memory.available<750MiThe regressive formula for a 4GB node typically reserved:
- 25% of the first 4GB = 1GB (1024 MiB)
Total Reserved: 1024 MiB
Allocatable Memory:
4GB - 1GB - 0.75GB = 2.25GB
AKS 1.29
Eviction Threshold:
memory.available<100MiMemory Reservation: Capped at
25% * 4GB = 1000MB(1GB), as it's lesser than20MB * 70 + 50MB = 1450MB.Allocatable Memory:
4GB - 1GB - 0.1GB = 2.9GB
For a detailed understanding of resource reservations in AKS, refer to Microsoft's official documentation.
Conclusion
The transition from the regressive formula in previous versions to the new calculation method in AKS 1.29 can result in different amounts of memory being reserved for kubelet and system processes. In AKS 1.29, the changes generally lead to a more predictable and potentially higher percentage of allocatable memory, particularly for larger VM sizes. The reduced eviction threshold in AKS 1.29 also signifies a more proactive approach to managing memory availability, enhancing node stability.
To effectively manage memory in AKS 1.29, stay updated with new changes, apply best practices, and continually adjust to your application needs. Efficient memory management is an ongoing journey that enhances your AKS cluster's performance.