What Do We Need To Size
The SAP Datasphere Object Store consists of three features:
- The object store to hold data, that includes the data in the tables, metadata logs, and back-up snapshots.
- The Apache Spark engine as the compute for all operations, like creating delta tables, transformations, and housekeeping.
- SAP HANA Native SQL on Files the SQL on files engine in SAP HANA, as an bridge between the object store and the relational world of SQL, that is used when reporting on data in the object store, when previewing it, when querying it in any fashion and more.
The SAP Datasphere Object Store meters three distinct, but dependent measures.
- Object Store Storage - is the sum of all files and folders stored in the object store of an SAP Datasphere space.
- Object Store Compute - is the total amount of 4 GB Memory hours (which combined CPU, Memory and temporary disk storage) used by all Spark jobs.
- Object Store Requests - occur during all interactions with the object store, e.g. reading a file (get OPEN), writing a file (put CREATE), deleting a file (del DELETE) and many more (see documentation).
Data Volumes and Compression
We must be aware that the same records stored in a raw format, local tables (HANA) SAP HANA, or local tables (file) (stored as delta tables in SAP HANA Data Lake Files) use different amounts of storage. Compression rates can vary based on the actual data and its data types. One assumption that can be used is:
1 GB CSV (Raw) -> ~1/8 GB in HANA
1 GB HANA -> 1.35 GB in delta tables
For the performance calculation, we will use the SAP HANA Data Lake Files (HDLF) data volume.
Object Store Storage
The total data footprint consists of the initially loaded data plus the incremental delta loads accumulated over time.
When data is loaded into a local table (file) using a Replication Flow, it is first written to an inbound buffer. Then a merge task has to be completed so that the data is moved into the underlying delta table.
Data files in delta tables are immutable. Any update, delete, or merge operation generates new files and record ‘remove file’ actions for the old ones. As a result, overwriting data increases storage usage because previous file versions must be retained. These older files are needed for delta extraction and must remain available until all delta subscribers have consumed their data. They must be removed later by a vacuum job. The amount of storages used by these old versions depends on whether the source provides change data (updates/deletes) or only new inserts
The object store storage can be calculated by combining the initially loaded data and adding the periodic load volume that is added per load, times the frequency of the periodic load and the version retention period. In the worst-case scenario, all files receive an update during a delta load, which would cause the table to be re-written with a slightly larger size.
Here, SAP is working on multiple improvements to help minimize this growth, see the Roadmap Item
The best-case scenario is when only inserts are delivered and files are never changed, in which case the table storage only grows by its added volume of data during a merge.
The object store keeps a backup of all deleted files for disaster recovery over a period of 14 days.
The versions and backup should level out eventually, when the retention periods are reached and a new version comes in, as an old version leaves the system.
Storage is metered hourly. That results in the fact that the monthly CU charged is not the size at the end of the month time hours in the month, it is rather the integral over time. For a linear graph, it’s the the average data size multiplied by the number of hours in the month (for estimations we assume 730h per month). Or in other words we need to estimate the average active data size in our system.
Object Store Compute
Object Store Compute is measured in blocks, based on 4 GB/h. This is because all Apache Spark configurations currently have a 1 CPU to 4 GB memory ratio.
Therefore, we need to estimate how many hours Apache Spark will run. This is the sum of hours spent on:
- Task runs in the object store (spaces with storage type "SAP HANA Data Lake Files")
- Deployments in the object store (spaces with storage type "SAP HANA Data Lake Files")
- System Tasks
- Statistic Updates
- Housekeeping jobs (like cleanup of logs)
For merge and transformation flow tasks, the required object store compute depends on many factors:
Data Characteristics
- Volume - total data size drives memory and parallelism needs
- Cardinality - high-cardinality columns (e.g. user IDs) affect shuffle size in groupBy/join
- Skew - uneven key distribution causes some tasks to process far more data than others
- Schema width - wide rows (many columns) increase serialization overhead
- Write pattern - insert-only workloads produce fewer redundant files than upsert/delete-heavy ones, so Optimize is cheaper after them.
Transformation Type
- Narrow vs. wide transformations - filter, map, select stay within a partition; groupBy, join, distinct, repartition cause shuffles (expensive)
- Shuffle operations - triggered by wide transformations; move data across the network and write intermediate results to disk
- User defined functions (UDF) - python UDFs especially are slow (JVM ↔ Python serialization); prefer built-in Spark SQL functions
Workload Configuration
- Executor memory/cores - determine parallelism and whether spill to disk occurs
- Dynamic allocation - lets Spark scale executors based on workload
For optimize, the required object store compute depends on:
- Number of small files - the main cost driver. Each write operation (merge, load, upsert) appends new files and creates new versions. The more fragmented the table, the more spark workload optimize must use to compact them. This makes optimize costs grow with write frequency, not just data size.
- Table size (total data volume) - larger tables take longer to compact even at the same file count.
- Z-ORDER BY - if column co-location (Z-ordering) is applied, Optimize must physically reshuffle data across files to group related rows. This is significantly more compute-intensive than simple compaction.
For an overview on best practices on optimizing consumption of the object store see the SAP Note 3722983
Object Store Requests
Object store requests are created during any read and write interaction with the object store. Besides the Spark-based tasks we need to add the object store requests from the Replication Flow writing into the inbound buffer table and we need to consider data access by SQL on File (SoF) calls.
Object store requests will be the combination of:
- Task runs in the object store (spaces with storage type "SAP HANA Data Lake Files")
- Replication Flow writes into Inbound Buffer
- Deployments in the object store (spaces with storage type "SAP HANA Data Lake Files")
- SQL on Files (e.g. cross-space sharing to HANA spaces or Data Preview)
- System services (e.g. update table statistics that run daily)
Factors that impact object store requests:
- Access method to local tables (file) (delta/active) - delta access requires additional access to logs and versions, while the volume in delta is typically smaller
Merge
- Data volume in Inbound Buffer - determines how many target files are candidates for rewrite
- Insert-only vs update/delete - insert-only merge can append new files; updates/deletes require rewriting matched files. When only inserts occur, it skips expensive target file rewrite
SQL on File reads into HANA
- Total number of files in local table (file) - fewer large files vs. many small files (influenced by frequency of merge or optimize runs)
- Partition pruning - filters on partition columns eliminate entire directories → fewer calls
- Column pruning - only requested columns are fetched (fewer bytes, same file count)
HANA can be used to cache data for reads, especially for SAP Analytics Cloud. Related best practice is to create data marts in spaces with SAP HANA Cloud storage (local tables (HANA) or persisted views) for reporting scenarios with frequent use and SLAs requiring high-performance. There are ways to mitigate the full reads in SAP Datasphere persisted views . You can partition views and lock partitions that don’t change, preventing unnecessary data read. To enable this, ensure that the partitions defined in the view are aligned with the partitions defined in the Local Table (File), so that partition locking effectively avoids reading the underlying files.
Another alternative is to use local tables (HANA) instead of view persistency. This allows the system to load only the deltas from the Local Tables (File), so it reads only the changes. As a result, the number of object store requests is significantly reduced.
SQL on Files object store requests run by queries (such as MDS queries, previews, etc.) are even more difficult to estimate, as there is typically no available data to base these calculations on.
For an overview on best practices on optimizing consumption of the object store see the SAP Note 3722983
Input for sizing
- Find your average active data size
- Estimate how many small (<5GB), medium (5GB-50GB) and large (> 50GB) tables you have and what percentage of tables have changes, and which are insert-only
- Define data availability and update frequency for tables based on end user requirements and object store consumption. More frequent merge runs increase object store consumption.
- Determine the number of potential re-initializations per year, for example required for incompatible schema changes in connected sources
- Make an assumption about data retention (housekeeping on data performed with delete and subsequent vacuum processes).
If you need support to come up with an estimate, reach out to your SAP contact (Sales, Pre-Sales, COE)
Bluesky
