Why You Shouldn't be Afraid of SSL Performance

[sufw]

I’ve often wondered why I see so many SAP systems which don’t offer SSL connections for their client applications, be they traditional SAP GUI clients, browsers running BSP or Web Dynpro applications or the IE runtime embedded into Netweaver Business Client. To me this has always seemed strange because SAP applications contain a lot of information I thought companies would be interested in keeping safe and secure: customer data, accounting information, salary information, etc.

Every time I asked the question of a Basis consultant though, I received an answer along the lines of “SSL introduces too much overhead”, or it “slows things down”.

But hang on, it’s 2013! Google searches are now all done over HTTPS by default, for 38,000 search queries per second! So how hard can it be?

Let’s try to get a handle on this by looking at the impact of switching to SSL to both network performance and server-side CPU load. I’m not going to worry about client-side CPU impact, as there are many more clients than servers, and they’re all multi-core systems running idle most of the time anyhow :-).

To keep things simple, I will only look at HTTP rather than the DIAG protocol used by SAP GUI. HTTP - through Web Dynpro and NWBC - is the future anyhow ;-).

The vast majority of SAP’s products - from old R/3 systems to HANA - use the Internet Communication Manager as the bridge between incoming HTTP traffic and the applications. This is nice because the ICM is more powerful than usually appreciated, and findings here with one system are broadly applicable to many of SAP’s products.

Network Impact

Let’s first recall the OSI model to understand where SSL (really, TLS) sits relative to other networking protocols we need for our HTTP communication example:

Layer	Name	Examples
7	Application	DNS, FTP, HTTP, SMTP
6	Presentation	MIME, XDR
5	Session	PPTP, TLS/SSL
4	Transport	TCP, UDP
3	Network	IP, ARP, ICMP
2	Data Link	ATM, IEEE 802.2, IEEE 802.3
1	Physical	IEEE 802.11, DSL, Bluetooth

Because there are turtles all the way down, I’ll only focus on the highlighted sections above. The other layers are interesting and very important to understand (mobile app connectivity, anyone?), but they’re not overly relevant right now.

So, let’s assume we have obtained an IP address (i.e. everything up to Layer 3 is done) and want to make a simple HTTPS GET request to an SAP system via SSL. Let’s also assume we are using HTTPS from the outset by specifying a URL starting with https://, rather than starting with HTTP and letting the server upgrade the protocol to HTTPS. I think that’s reasonable for the usual portal/NWBC-based interactions with SAP systems; I don’t think many users type WebDynpro URLs directly into their browser...

This scenario results in the following flow of messages:

Click the image above for a full-size version

Messages 1, 2 and 3 establish a TCP connection at Layer 4 in the OSI model. This is the typical three-way TCP handshake which establishes a reliable TCP session, and is required regardless of whether SSL is used to encrypt communications or not. In the OSI model, higher-level protocols such as HTTP or SSL build upon the foundations established by the lower layers.

The next 4 messages shown in orange are where the real work to set up an SSL session happens. This is where the client verifies the server’s identity, and both parties independently derive a shared session key used to encrypt subsequent communication. I will leave the details of this public/private-key cryptographic “magic” to the links at the end, but a few things are important here:

• There are “only” two round-trips across the network. This isn’t so terrible even across a WAN.
  • (to be precise, these are Layer 4 round-trips. If the server is using large 4096 or 8192 bit certificates, a long certificate chain, etc. then the network packets will be too big and cause the TCP congestion windows to overflow, resulting in further round-trips at Layer 3 in the stack)
• Asymmetric public/private-key encryption is used to establish this shared encryption key. Asymmetric encryption is relatively slow and CPU-intensive, so the handshake is expensive.
• Once the key is established, both parties will use fast and efficient symmetric encryption (e.g. 3DES, AES, RC4 or any other algorithm agreed upon during the handshake) to encrypt data. This is usually fast.

Finally the client can send its HTTP request message over the SSL session established earlier. The server will respond to the request across the same pipe, which is secure from eavesdropping by outsiders. Nice!

Server CPU Impact

You know, Moore’s Law is great. Every 18 months, computing capacity is expected to double.

But over the lifetime of an SAP system, it’s astonishing! Applied over 10 years, we would see computing power increase by a factor of 100! One of my current clients has been running SAP ERP since 1998, or for 15 years. That’s long enough to witness a factor 1000 improvement!!

Source: Twitter

So maybe the slow, asymmetric encryption used to derive session keys during an SSL handshake was an issue when many experienced Basis consultants set up their first systems. But today? Definitely not!

My 2008-era MacBook Pro can sign 2048bit RSA keys in 9.1 milliseconds, my 2012 MacBook Air takes 6.1 ms, and the Xeon-based systems we run various SAP apps on take 3.5 ms.

Once the SSL session is established, data travelling across it is encrypted using more efficient symmetric algorithms such as AES or RC4. Their throughput is even greater: 173 MB/s and 650 MB/s respectively on the same Xeon-based servers.

Based on these figures, the performance impact of moving to SSL should be between small and negligible. However it’s always a good idea to confirm theories by way of a test.

Let’s Test!

The exact setup is described below in a lot more detail, but here’s the short version:

• 50 parallel threads in soapUI sending GET requests to a resource in the ICM memory cache
• 200 requests per thread. No delay between requests.
• 500ms thread start-up delay (i.e. ramp-up takes 25s)
• Pre-emptive HTTP Basic authentication (i.e. credentials are sent as part of initial GET request rather than after receiving a 401 response from the server asking for authentication)

We will use three scenarios:

1. Plain-text HTTP. Enough said.
2. HTTPS, by changing the URL used in the test to https://
3. HTTPS as above but with the “Close Connection between requests” option selected, forcing a TCP connection and SSL handshake for every request. In the graphs, I’ve called this “SSL All-New”

Results

Client Impact

The main concern here is response time. If SSL under load takes a significantly long time, then this will translate into a poor experience for the user. SoapUI recorded the response times in milliseconds for each test scenario, and Excel created the following histogram:

Click the image above for a full-size version

	Throughput	Response Time (ms)
	requests/sec	Median	95th percentile	99th percentile
Plain-text	93	17	20	33
SSL	93	22	36	40
SSL All-new	75	66	82	>1000

Here the cost of SSL traffic does become visible. However the overhead is small for the ‘normal’ SSL scenario at less than 10ms - not noticeable to end users.

The ‘three peaks’ distribution of the “all-new” scenario does seem a little strange and could merit further investigation into profile and ICM parameters, or even repetition using a different client tool to eliminate issues on the soapUI side.

Slightly more concerning are the approx. 2% of requests which took longer than 200ms to complete in the “all-new” scenario. Most of these took over 1 second, which will definitely be noticeable to end users. If the “all-new” scenario of 75 SSL handshakes a second sounds like something your ICM server could experience, then further tests and tuning would definitely be a good idea. I can’t begin to speculate the cause of this phenomenon.

Server Impact

I used a simple ‘top’ command on the server to record the CPU usage of the icman process, which is what the ICM is called at the operating system level. Here are the results, smoothed using a 15 second moving average:

Click the image above for a full-size version

Although the ‘normal’ SSL scenario consumed around a third more CPU than the same test using plain-text HTTP, it still stayed below 5% of CPU usage on a single core. The ‘worst case’ test of establishing a new connection for every request consumed around 3 times more CPU than the plain-text base case. But even then, the 15-second moving average did not go above 10% of a single CPU core.

Basically, even at ~75 SSL handshakes per second, the impact on a multi-core system would be barely noticeable.

ICM Workload

Snapshots of the ICM workload taken during the test also back up this picture. Even in the “all-new” scenario, the number of ICM worker threads reached a peak of 64 - again nothing to worry about on a sandbox system with a maximum of 250!

Network Impact

Using SSL means encrypting packets between the client and the server. Intermediary proxy caches, WAN optimisers and other network infrastructure often deployed in global organisations to squeeze more out of their expensive WANs thus won’t be able to look into those packets and cache them for other users. So if you’re using SSL to encrypt icons, stylesheets, and other static, non-private resources, then there will be a performance cost as clients have to fetch them from the server every time. Depending on your topology, this may be expensive and sub-optimal. However I’m testing the SAP server-side impacts here, so I’m calling this out of scope.

Using SSL also means at least one additional round-trip to the server to perform the initial handshake. Being around ~2ms away from the server, this won’t really be noticeable. I’m guessing it also won’t be noticeable loading a Web Dynpro app with dozens of HTTP resources and hundreds of kilobytes of data. But it might make a difference to high-volume or low-latency API consumers (REST clients, anyone?), so it’s something to consider.

Test Setup

Server Setup - Code

On an SAP system, HTTP requests go through the ICM which acts as both a HTTP server and reverse proxy in front of the ABAP application server. The ICM also terminates SSL traffic, and this is where the pain of many SSL handshakes should be felt most acutely.

On the SAP server side, I am reusing a HTTP handler which I had previously written for another load testing exercise and which contains a very bad implementation of a Fibonacci sequence generator. It’s “very bad” on purpose in order to generate a lot of CPU load; if I knew what I was doing with ABAP, I probably could have written a terrible or even horrendous implementation.

The full code of the IF_HTTP_EXTENSION~HANDLE_REQUEST method is available here as a Gist. Suggestions towards an “improved” v2 are always welcome! 😉

In order to separate the server load of processing HTTP and SSL traffic from the load caused by actually processing request messages and generating responses, we need to enable caching of responses in the ICM memory cache. This is in-memory computing without HANA and really a cool feature! Setting this will cause the ICM to serve responses very quickly and without going back to the ABAP system and consuming CPU cycles in the process.

ICM caching is easily enabled by adding the following line into the IF_HTTP_EXTENSION~HANDLE_REQUEST method of your HTTP handler: server->response->server_cache_expire_rel( expires_rel = '300').

Alternatively, it would also be possible to use some other resources which are currently cached by the ICM. The current contents of the ICM’s cache can be seen via transaction SMICM > Goto > HTTP Plug-In > Server Cache > Display. Usually it will contain image files or CSS style sheets belonging to the Web Dynpro or UI5 frameworks by default.

Server Setup - SSL

In order to test SSL performance, the SAP system needs to be given some SSL certificates via transaction STRUST and the ICM be configured with an SSL port and relevant config. If you want to test SSL with client authentication, then client certificates will also need to be issued, installed in the client’s certificate store, and their fingerprints mapped to an SAP user account using table USREXITD - all things which I won’t go into for fear of making this blog even longer...

Client Setup

There are a large variety of client tools which can generate HTTP traffic to load test a server. jMeter, cURL or Httperf would all do the job extremely well, but I’m using soapUI here mostly because it’ll be good enough for the job and it’s a tool I use quite a bit so there’s no learning curve.

SoapUI lets me create a test script which I can then execute using its built-in load testing functionality using multiple parallel threads. For this test, I’m aiming to measure the CPU utilisation of the SAP system’s ICM process under repeatable load: in this case, 50 parallel threads sending a new request as soon as a response to the previous request has been received.

I’ll use two scenarios here:

1. “Normal” SSL. Each thread makes 10 requests to the server before exiting. (soapUI will immediately restart it to continue the test). While the thread is running though, it will reuse the TCP connection with the server according to the server’s instructions in the keep-alive header. This means it also won’t drop the SSL session, and thus perform fewer ‘expensive’ SSL handshakes.
2. “All-new” SSL. The connection with the server is closed after each request, meaning a new SSL handshake needs to be performed for every request. This isn’t a realistic scenario as no well-behaved client such as web browsers would do this, but it helps in placing additional load on the SSL components on the server.

The soapUI project file is also on Github if you're interested in repeating the tests on your own system.

Further Reading

A whole raft of blogs and presentations have been very useful in devising these tests. If you're interested in SSL from a performance perspective, I would highly recommend the following as ideal primers to what quickly becomes a deep dive into the multiple layers of any IT stack:

http://www.semicomplete.com/blog/geekery/ssl-latency.html

http://www.imperialviolet.org/2010/06/25/overclocking-ssl.html

http://en.wikipedia.org/wiki/Secure_Sockets_Layer