PROVISIONING
The provisioning of a paid Kyma runtime presents a number of different options related to the machine size and scaling options including:
"autoScalerMax": 10,
“autoScalerMin": 2,
"machineType": "Standard_D8_v3",
The value autoScalerMin denotes the minimum amount of virtual machines or nodes that will be available in the cluster. This value can not be set lower than two, which ensures that enough capacity is available for all Kyma/K8s components and additional workloads can be added to the cluster before any additional machines are added by the autoscaler. Two nodes also ensures a level of high availability in the case a machine failure occurs. Depending on the workloads to be added to the cluster and the scaling expectations, in productive scenarios it may be preferred to have this set at a higher number based on the use case(s) that would be running on the cluster.
The autoScalerMax denotes the maximum amount of virtual machines or nodes that will be available in the cluster. Defaulting to 10, this value can be adjusted with a maximum value of 40. The basis of this value should be dependent on the cluster’s use, i.e. production or development etc, to restrict max utilization/cost. The process of adding additional nodes is automated and is dependent on the workloads running in the cluster. We will look further into this later in the blog.
The MachineType refers to virtual machine size. in Kubernetes/Kyma each one of the machines is referred to as a node. There are a number of options for the machine type with the minimum consisting of 8 CPU and 32GB Ram, depending on the provider this translate to the machine type options.
There are also a number of other machine type options available with the max being 48 CPU and 192GB Ram. In most cases the minimum is preferred. As additional machines are added to the cluster, a smaller machine increases capacity at a lesser rate, resulting in a more efficient resource/cost utilization. In the case of a machine failure, applications spread across a number of four smaller machines would also cope better vs if only two larger machines were used. In the case you have an application that requires a high amount of memory or CPU, such as Big Data/AI scenarios, the performance a larger machines offers may be preferred.
This is best understood by provisioning a cluster and reviewing the tools available to understand the process. With that being said, let's review a newly provisioned Kyma runtime cluster using the default settings for Microsoft Azure
"autoScalerMax": 10,
“autoScalerMin": 2,
"machineType": "Standard_D8_v3",
kubectl get nodes
This returns the nodes of the system, which currently is equivalent to the autoScalerMin value of two. To see the current resource usage we can then run
kubectl top nodes
This shows that the system is currently only using 6% of CPU and 15% memory of its capacity. To get details of each node we can then run describe to return the details of each node
kubectl describe node shoot--kyma--c-7dea047-cpu-worker-0-z1-6dbb4-hxssg
This returns a number of different pieces of information, such as the node capacity which displays the CPU and memory details. Once again this is based on the machineType, Standard_D8_v3 which provides the Capacity of 8 CPU and 32GB Ram, requested when the cluster was provisioned minus what the system itself has reserved, the Allocatable amount.
This also lists all of the pods on the node and their set CPU/memory Requests/Limits as well as the overall node Allocated resources.
The Allocated resources details the total amount of CPU/memory that is currently reserved of the node. Each workload added to the cluster has the option to reserve resources by setting resource request for memory and CPU which guarantees this amount for the workload. As workloads are then added, they can only use/request the CPU/memory that is free. In this case 39% of CPU is available.
To summarize, the system is currently only using ~6% of CPU and ~15% memory, but has ~61% of CPU and ~22% memory reserved. The runtime will run in this capacity until workloads are applied, or existing workloads attempt to scale out. As this happens there may be a point where the workload can not be scheduled on a node, which will cause the autoscaler to create another node.
Node Scaling overview
If workloads fail to be scheduled due to insufficient resources, the autoscaler will automatically kick in and provision an additional node. This will hold true as more additional workloads are added, additional nodes will be added to meet capacity until it maxes out based on the autoScalerMax. If the workloads are then removed, the additional machines will be de-provisioned, back down to the minimum, if necessary.
As workloads are added to the cluster, the available capacity will be consumed. This is based on either the amount of resources the workload is using or what it has reserved. The reservation of resources is accomplished by setting memory/CPU requests on the workload which guarantees this amount. This is part of the spec when defining a deployment for example. Each workload can also be set to have a limit on the amount of CPU/memory that it can utilize, resulting in an out of memory error if exceeded. If these are not defined then the actual usage of the workload will have no reservation or constraints placed on it unless the namespace it exists in has a default limit. In this case the namespace default limit would be applied to the workload.
The requests and limits are on a per container basis, there can be a number of containers within a single pod. By default any workload added to Kyma will have two containers in it, your workload and the istio-proxy which provides the service mesh functionality.
kubectl run nginx1 --image=nginx --requests='cpu=2000m'
kubectl get pod nginx1 -o=jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}'
istio-proxy
{"limits":{"cpu":"1","memory":"1Gi"},"requests"{"cpu":"100m","memory":"192Mi"}}
nginx1
{"requests":{"cpu":"2"}}
At this point the node will have about 7843m, 7.8 cores, of CPU allocated, 2 cores from our nginx container and 1 from the istio-proxy container
If we now add another pod that requests 2000m of CPU
kubectl run nginx2 --image=nginx --requests='cpu=2000m'
The pod will be scheduled on the other node which results in both nodes being close to 90% committed for CPU requests. This can be seen by running describe on each node to view the Allocated resources.
Adding one more pod with the same CPU request will cause the cluster to scale an additional node because the amount of requested resources is not available, which causes the scheduling of the pod to fail until the new node is added.
kubectl run nginx3 --image=nginx --requests='cpu=2000m'
kubectl get pods
kubectl get events
Now after a few minutes we see that another node has been added to the cluster
kubectl get nodes
And if it is then deleted
kubectl delete pod nginx3
In about an hour the node will be marked for deletion and removed from the cluster which is shown in the events
Tuning workloads
Just as we did with the nodes running we can see the resource usage using
kubectl top pod
This of course only provides the current resource usage. A historical view can be found in the Kyma dashboard within the namespace pod view as well as within Grafana. This information can be used to define appropriate settings for the memory/CPU requests of the workload when a normal amount of load is applied. Of course there are some other aspects to consider such as the number of replicas defined to run as well as a Horizontal Pod Autoscaler which allows the workload to scale up and down. For the sake of brevity, we will leave that for another discussion.
After reading this blog I hope that you better understand the Kyma provisioning process and how scaling is determined. For more information on this topic please reference the official SAP help documentation and stay tuned for more blogs.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
26 | |
24 | |
21 | |
13 | |
9 | |
9 | |
9 | |
9 | |
8 | |
8 |