TL;DR: We have added support for configuring the load-balancing algorithm of routes in SAP BTP, Cloud Foundry runtime and environment. See the links in additional resources for details on how to configure it.
SAP BTP, Cloud Foundry Runtime
Cloud Foundry is an open-source PaaS designed to run Twelve-Factor applications. Those applications can then be called via HTTP by attaching a route to them. A route is basically a domain and a path like service.example.com/api. The routing stack of Cloud Foundry then routes any incoming request to the appropriate application. Most applications will run with multiple instances to increase their availability and scale to accommodate the increasing load from clients. To decide which instance of an application should receive a given request, the routing stack implements load-balancing algorithms.
Load-Balancing
In general, the goal of load-balancing is to distribute load as evenly as possible across the available instances, as imbalances can lead to some instances being overloaded while others are idling, causing disruptions / degradation of the service. There are several strategies to distribute requests:
Round-Robin
Probably the most popular load-balancing algorithm is round-robin. In round-robin, the load balancer iterates over all backends in order, sending one request at a time to each.
Being a very simple algorithm, it has its limits when load patterns are heterogeneous, but it's widely available and is the default algorithm used in SAP BTP, Cloud Foundry runtime and environment.
Least-Connection
An alternative to round-robin is least-connection. As the name implies, when deciding where to send a request, the load-balancer first determines how many active connections it has to each backend and sends the request to the backend with the least. This is beneficial if an application supports long-running requests as well as rather short ones. In such cases, with round-robin, there is a chance that long-running requests accumulate in some instances, while others receive only short requests, resulting in an uneven distribution of load. The least-connection method mitigates this by assuming that the number of active connections correlate with load. It then tries to even out load by directing requests to instances with fewer connections.
We have now added the option for developers to configure this algorithm for their application routes.
Hash-Based
In some cases, it might be desirable to have a certain stickiness of requests to application instances, for example ensuring that all requests from the same client IP are routed to the same instance. A common way to achieve this is to use a hash algorithm, which is an excellent way of turning arbitrary complex input into evenly distributed numbers. By properly specifying the input value used to calculate the hash, you might be able to increase the number of cache hits, if some data has to be loaded per client but not per request.
Update: We have added support for hash-based routing to Cloud Foundry, read the blog post for all details.
Others
There are many other ways of balancing load between instances, including mechanisms to inform the load-balancer of the instance state via out-of-band communication. Such algorithms require tight integration between the load-balancer and the service.
As mentioned before, by default, every route uses round-robin as load-balancing algorithm. We have now released an extension to the routing-stack which allows users to choose either round-robin or least-connection as the load-balancing algorithm for their route.
Configuring the Load-Balancing Algorithm
Depending on how you deploy your applications, there are several options to configure your route. The most straightforward way for existing routes is to configure them using the cf-CLI (you'll need version v8.10.0 or newer):
cf update-route example.com -n service --path /api -o loadbalancing=least-connection
This will set the load-balancing algorithm for the route service.example.com/api to least-connection.
An alternative approach is to configure the algorithm as part of the app manifest which is a common way of deploying applications to Cloud Foundry:
applications:
- name: example
routes:
- route: service.example.com/api
options:
loadbalancing: least-connection
Choosing an Algorithm
The first thing to note is that, if your application is working fine, you can just leave it as-is.
If you experience issues with uneven load distribution across your application instances, you could try switching from the default round-robin to least-connection as an alternative. However, this always has the risk of causing unintended side effects. When making such a change, ensure to properly test it before rolling it out to production environments.
The least-connection algorithm may produce better results if you use the Application Autoscaler service. This algorithm will direct the traffic to the new instances that the Application Autoscaler creates when scaling out, because they don't have any active connections. By contrast, with the round-robin algorithm traffic would even out more slowly as connections created before the new instances became available finish.
Additional Resources
Bluesky
