The 3-hours to 7-Second Story
We had a weekly data integration form ABAP in Cloud, via an OData service, that took about 3 hours(10800 seconds) and nearly 600 requests to finish. After a small redesign, the same data now arrives in ~7 seconds(instead of 10800 seconds) using just 3 requests. So a dramatic x1500 reduction. No new servers, no fancy tools, just smarter packaging of the data, but of course with some drawback.
The full PDF paper, and code is available in this GitHub repo: https://github.com/legonmarian/abap-btp-api-optimization
Quick Context
The Simple Change
Why not OData for this job:
OData is a really good tool for interactive reads: $filter, $expand, small pages, typed entities, also for Fiori applications, but our use case was the opposite: one flat dataset, all of it, as fast as possible.
With OData we’d still pay the cost of many small pages and per-entity overhead the client didn’t need. And most of the consumers wanted a simple file payload they could ingest with generic tools.
Besides that the biggest restriction why OData cannot be used for this use case is that, OData handles out-of-the-box the data transfer, so you have no control on the data shape, or compression(which proves to be a big player further).
Why a plain HTTP service instead
On the other hand an HTTP Service with an HTTP handler class gives us full control over the wire format (JSON/CSV), headers, and compression.
And this is the main reason why we moved from an out-of-the-box nice and elegant OData service but slow, to an HTTP service with a custom handler but fast.
Why “just HTTP” still wasn’t enough
So the HTTP service doesn't provide any other functionality out of the box other than full control over the request and the response objects. That means that all the other hard work needed in order to send back in the response a DB table should be implemented by the developer, this includes:
- pagination
- serialization
- compression(optional)
Besides that when it comes to custom code, and big tables, the developer must ensure that the program's complexity isn't surpassing the allocated resources and constraints, both in terms of time and memory.
Therefore we explored(and documented in the code) multiple ways of doing this. For more details on these please check the PDF in here. But here are some of our conlusions:
- If compression is not suitable for your use case, moving from JSON to CSV is a huge advantage
- Compressing is the best solution, this way you can keep your JSON schema
- There're several compression algorithms but the most suitable for us in ABAP in cloud proved to be GZIP with out-of-the-box support. We examined the compression / decompression performance of multiple algorithms in the PDF paper
- The fastest JSON serializer is
CALL TRANSFORMATION being thousands of times faster than the xco library - Reading a DB table in an internal table and serializing it requires memory, and we ran out of it when trying to serialize the full table.
- An alternative to this was to read chunks of it, serialize and compress it and then repeat for next chunk. But this is a bit less client-friendly from a decompression perspective
- Streaming data is not possible with the current ICF of ABAP in Cloud
- One last performance optimization is parallelization, this can cut the load time in 3-4 times, but use it carefully
- Always profile for bottlenecks
- Always stress test to find the right parameters like page size, concurrent calls, etc.
What we landed on (the pattern)
Before vs After
TL;DR: We didn’t change the data, only the delivery: coarse pages, fast JSON, and gzip over a plain HTTP contract.
| Parameter | Before | After |
| Dataset | ~3,000,000 rows (flat, ~12 columns) | Same |
| Page size | 5,000 rows/page | ~1,000,000 rows/page (tunable) |
| Number of requests | ~593 | 3 |
| End-to-end time | ~3 hours (sequential pulls) | ~6-7 seconds (3 parallel pulls) |
| Total transfer | ~6000 MB | ~9 MB |
| Payload per page | ~1.15 MB per 5k rows (raw) | ~3 MB per 1M rows (gzipped) |
| Protocol | OData | Plain REST |
| Compression | None | Content-Encoding: gzip (single member) |
| Client pattern | Sequential loop | Fetch pages in parallel, then merge |
How to reproduce this in three simple steps
Heads-up: in here I will refer to an Appendix, you can find it in the detailed PDF paper about the optimization, the paper and some code is available on GitHub
Step 1. Define a tiny HTTP contract
{
"number_of_records": 2496434,
"batch_size": { "maximum": 1500000, "recommended": 1000000 },
"recommended_pages": [
"/entity?offset=0&count=1000000",
"/entity?offset=1000000&count=1000000",
"/entity?offset=2000000&count=1000000"
]
}
This keeps clients simple and lets them plan parallel pulls.
Step 2. Build one big page, serialize, and gzip it on the server
a) Fast JSON generation with CALL TRANSFORMATION
Appendix C shows the lean serializer that perform the best, for the performance comparasion please check the PDF paper
METHOD convert_json_transformation.
DATA(lo_writer) = cl_sxml_string_writer⇒create( type = if_sxml⇒co_xt_json ).
CALL TRANSFORMATION id
SOURCE itab = data
RESULT XML lo_writer.
string = lo_writer->get_output( ).
ENDMETHOD.
Use this to turn your internal table into JSON quickly.
b) Minimal HTTP handler that serves one gzipped page
Appendix I demonstrates the pattern: set headers, read a deterministic slice, serialize, gzip once, send bytes.
METHOD gzip_json_single_page.
response->set_status( 200 ).
response->set_content_type( 'application/gzip' ).
response->set_header_field(
i_name = 'Content-Disposition'
i_value = |attachment; filename="data_subset.gz"| ).
response->set_compression(
options = if_web_http_response⇒co_compress_none ).
response->set_header_field(
i_name = 'Content-Encoding'
i_value = |deflate| ).
SELECT column_1, column_2, ... , column_12
FROM dbtable
ORDER BY column_2
INTO TABLE @DATA(page)
UP TO @page_size ROWS.
cl_abap_gzip⇒compress_binary(
EXPORTING raw_in = convert_json_transformation( page )
IMPORTING gzip_out = DATA(gzip) ).
response->set_binary( gzip ).
ENDMETHOD.
c) Where the handler is wired
Appendix F shows the entry point choosing which implementation to run:
METHOD if_http_service_extension~handle_request.
" choose one of these
gzip_json_single_page( CHANGING request = request response = response ).
" only for demonstration
"gzip_csv_single_page( CHANGING request = request response = response ).
" only for demonstration
"gzip_csv_multiple_pages( CHANGING request = request response = response ).
ENDMETHOD.
Step 3. Let the client pull a few big pages in parallel
Optional: CSV variant from the appendix
If you ever need CSV, Appendix H shows the single-page CSV + gzip flow. The JSON path above stayed as our final choice because gzip erases most of JSON’s key overhead while keeping tooling friendly. If you'd like to read more on why the JSON -> CSV conversion becomes obsolete after gzipping please check PDF paper.
When to use this pattern and when not to
Use it when
Think twice when
Gotchas to avoid
For the full PDF paper, and code please check this GitHub repo: https://github.com/legonmarian/abap-btp-api-optimization
Bluesky
