Mastering the Data Deluge: A Comprehensive Guide t...

Amin_Omidy · ‎2023 Mar 08

A new update on April 2025

Abstract
Are you struggling to manage the ever-increasing amount of data in your SAP ERP system? Look no further! this blog provides a comprehensive overview of data strategy, Information Lifecycle Management (ILM), data archiving, and data volume management techniques specifically tailored for SAP customers.

In this blog, I've gathered known methods and techniques in one place to help you explore different options that can fulfill your specific needs. I've also included references and links to SAP training courses at the end of this blog for those who are interested in earning SAP official certificates or training in this field. And if you need expert support, I've even included a link to SAP's payable on-demand consultant services.

After receiving multiple inquiries from customers about data management and information lifecycle management options available in SAP, whether in SAP ERP or SAP PaaS and SaaS, I decided to write this blog as an overview to assist them with their operational tasks or projects needs.

Target Readers:

SAP consultant /SAP Architect who is looking to start working on data volume management or data strategy
As a motivated professional you want to extend your career in this field
A customer would like to get general information about this topic to choose the right approach
SAP team and partners would like to provide an overview of data archiving and ILM to their clients

We understand that keeping up with data growth and separating relevant data from our operational system can be overwhelming. On the other hand, adding additional storage approach is not an optimal solution too. My goal is to simplify the process and provide a high-level summary of existing options that can help you to choose the right method to achieve your business objectives.

If you are in SAP shop and dealing with one of below objectives to address your business essential ,then data volume management, data archiving, ILM, or data retention can be the right choice to consider in your venture:

Looking for TCO reduction without losing auditing and reporting capability
Looking for one central data retention and repository for multiple SAP systems
Dealing with legal compliance, risk, and auditing acceptability; putting legal holds on data that is relevant for legal cases to prevent early destruction.
Leverage Inbuilt SAP programs to automate destroying the documents its retention reached, and all legal requirements are met.
On-demand SAP data retrieval after its archiving
Tax auditing, product liability, data privacy or data protection and the mandate to delete of personal or business partner data or GDPR
Facing performance degradation in your existing SAP systems because of unpredictable database growth with high memory and CPU consumption
Dealing with a corporate merger or acquisition, an in-house consolidation of multiple systems, or the transition to the Intelligent Enterprise
Migration from ECC to S/4 HANA in cloud and striving to reduce the future cost and memory footprint

Some terminologies ,concepts, and solutions:

Document Management services (DMS):
A Document Management Service (DMS) is designed to manage documents efficiently without the challenges associated with physical storage. A well-implemented DMS relies on a Content Management Service (CMS) to provide the necessary infrastructure for storing and retrieving documents in a structured manner.

Content Management Service(CMS):
CMS is responsible for document storage and technical management. It includes functionalities such as:

Version Control: Managing multiple versions of documents.
Recovery Mechanisms: Restoring documents when needed.
Storage Management: Optimizing storage utilization.

SAP Content Server serves as an SAP-certified CMS application, storing data in SAP MaxDB in a separate instance, ensuring secure and efficient document management.

ECM/EIM: Enterprise Content Management has been a part of companies for quite some time and in recent years ECM has been transforming into Enterprise Information Management (EIM).

Basically, due to the evolution of technology (such as cloud and hardware) and modernizing organizations, content now is more robust asset that can unify document management, BPM, data governance, etc.

ECM provides organizations with the ability to categorize and structure their content in a way that is both meaningful and easily searchable. It is not only enhancing user experience but also offers important analytical capabilities that can drive better decision making.

ECM implies an enterprise wide solution that is fully integrated across various applications to capture, store, manage, preserve and deliver information based on the business context.

ERP vs ECM: Enterprise Resource Planning (ERP) systems are primarily designed to handle structured data and transactions, serving as the system of record for an organization, especially when it comes to financial data and reporting. ECM support unstructured data too. On the other hand, Enterprise Content Management You can link a purchase order number (structured) to its scanned invoice (unstructured) in the same ECM context.

Integrating these two systems can give significant business benefits, such as providing a common information interface for users across the business. This allows for better management and organization of both structured and unstructured data, resulting in more streamlined and efficient processes.

SAP Document Management System (DMS) for S/4 HANA Cloud:
is an offering to cover DMS as a part of ECM in S/4 HANA Cloud.

For an attachment uploaded in S/4 HANA Cloud the metadata are stored in SAP DB tables but the unstructured data content or the actual content can be stored outside S/4 HANA in Content Megamonument Interoperability Service (CMIS) based repository in BTP which used CMIS protocol. As you may notice S/4 HANA Cloud is more advanced when it comes to DMS or CMIS.

DMS main options in S/4 HANA Cloud:

Operational Data Provisioning
Batch Input Sessions
Data Destruction
Blocking Distraction
Blocking and Deletion of Personal data
Information Lifecycle Management

For more detail, please check the links below for further details:
Third-Party Repository Integration with SAP S/4HANA Cloud
Content Management Interoperability Services (CMIS)
SAP Blog: Using Document Management as a Content Repository with SAP S/4 HANA
SAP Blog: Document Management in SAP S/4HANA Cloud
SAP S/4HANA Cloud Enterprise Content Management/ Document Management Offerings Video

SAP Document Center: (decommissioned , AKA SAP Mobile Document): was a sort of cloud Content Server (SAP cloud Neo environment) for Mobile content management that provided a secure file shared and access corporate content from any devices similar to Dropbox. This solution is now part of SAP new offering by SAP DMS.
Link:
SAP Document Center User Guide

SAP Document Management Service(the latest offering):
is a cloud based (Cloud Foundry) content management solution to help managing your documents with underlying hyperscaler platform support in BTP. It provides services such as archiving, manage metadata, mange document version, and mange file and folders.

Document Management Service is the successor for SAP Cloud Platform Neo and the SAP Document Center Service in Neo.
Document Management Service has two types of service plan, Free, and Standard

Different offerings of this service:

Document Management, integration option : develops a customized document management layer for business applications
Document Management, application option : is a standalone ready-to-use, Fiori-based, document management web application for content collaborations
Document Management, repository option : can securely store files and folders

Links:
SAP Document Management Service
SAP Document Management Service pricing (SAP Discovery Center)

SAP blog explained SAP BTP DMS:
SAP Blog: Start with SAP BTP Document Management Service (DMS)
Store: SAP Document Management Service

SAP DVM: An extradentary tool and methodology provided by SAP to its customers to be able to control the data growth in SAP landscape. Everyone who is planning to mange data volume and implementing data reduction in SAP landscape must know and leverage it!
SAP DVM support:

S/4HANA,
Suite-on-HANA
SAP_FIN
SAP BW4HANA

DVM Dashboard

You can access to DVM dashboard via SAP Launchpad or SAP for me (https://me.sap.com/dataoverview). After you added the "Data Volume Management" tile to your home page , you need to make sure you have the right authorization to the DVM Cloud app data before start.

It is worth to mention DVM data collection for HANA system is included in the EWA data collection. Basically this assures that DVM data additionally is available via DVM dashboard.
Links:
Data Volume Management
DVM Dashboard introduction presentation
2698930 - No Data is visible when using the DVM Cloud application - Authorisations
2868449 - Data Volume Management App - FAQ
2716655 - How To Use the Data Volume Management APP - Step by Step guide

ILM-Certified WebDAV server: is a storage of archive data to grantee non changeability of the data and protects it from impulsive destruction. It used WebDAV protocol to provide a framework for users to create, change and move documents on the server.

S/4 HANA Data Aging: feature can move less frequently accessed data (warm/cold data) from expensive in-memory storage to lower-cost disk-based storage (e.g., SAP IQ, Hadoop, or cloud storage) .By moving large amounts of data within a database you can gain more memory although data still will be part of database and its backup.
To enable data aging use Transaction Code, SFW5 for DAAG_DATA_AGING and set parameter abap/data aging on if your backend database supports this feature.
Links:
2869647 - Guidance for use of Data Aging in SAP S/4HANA
2416490 - FAQ: SAP HANA Data Aging in SAP S/4HANA

Data Archiving: is used to archive any complete business transactions which are no longer relevant for system operation tasks. Archive data has a subsequent delete phase that archived process removes from the database and can be stored in archive files outside of database. System can have read-only access to it and this data will not be part of database backup anymore.

ArchiveLink: is a service in the SAP Application Server for linking archived documents to support incoming and outcoming documents including print lists and archive files. It can be used to store unstructured data like pdf files from your SAP system to SAP Content Server. ArchiveLink also can enable the integration of external storage solutions, such as ShaePoint, for document archiving purposes.

Archive Development Kit (ADK): is a function to archive structured, transactional data. It used to archive structured, transactional data that is no longer required in the live database .It is an application program interacts between SAP applications and related archive files where the data is stored. Basically, ADK confirms data archiving is independent of hardware and release changes and provides the required tools for data archiving. ArchiveLink is part of the SAP NetWeaver stack and does not require SAP ILM to function.

ADK vs Archivelink:

Residence period: is the time the data stays in ERP system before to be archived.

Retention Management: called end-of-life data is the process to manage the lifecycle of data up to the point it can be destrcted.it used retention policy management and legal holds to control the retention of data in its entire lifecycle.

Blocking: The method of preventing access to a personal data that is no longer necessary in relation to the purpose it was collected.

Retention Period: Is the time it is required by law the personal data must be retained.

Destruction/Delete: Irreversible and adequate deletion of data after its retention period is reached.

The diagram below illustrates archiving, retention, blocking and deletion concepts:

Residence, Retention, and Deletion

E-Discovery: is the process can be used to determine any documents had legal cases that carried legal holds via ILM function.

DMLT Solutions: Data Management and Landscape is a consultant service delivered by SAP that includes:

Analysis of your archiving strategy
Application of relevant retention policies
Extraction of data as well as knowledge transfer for e-Discovery
Legal case hold and BW reporting configuration

Links:
Information Management & Governance
Strategies to Reduce Cost and Minimize Risk pdf

Content Servers in Cloud: in recent years more customers interested to have their Customer Data Management (CDM) server in the cloud connects to SAP system by benefiting lower costs, scalability, High availability, and less complexity in public cloud.

When it comes to ILM, data archiving, and data retention, there are a few options we can consider moving the historic data from SAP ECC or S/4 HANA systems to a separate system (ECM), database or file system for future access.
The diagram below showcases Data Volume management options:

As you can see by using only archiving method you can have a significant reduction in your system data consumption which will directly obtained a better performance and low TCO. Furthermore by employing ILM approach you can even reach a steady data volume in your system.

System Decommissioning:
Is the process involves utilizing SAP ILM functionality to transfer data from SAP legacy systems to a separate system. This enables the independent retrieval of data for reporting purposes.

System Decommissioning and Scenarios

To simplify the concept, you can use the logic below to decide your data management option:

Below are the most commonly used methods that customers can utilize based on their requirements for archiving ,managing data growth, and ILM:

1-SAP Classical Archiving
The Classical Archiving has been around for years in SAP world, and it can be mainly performed by Transaction Code: SARA and planed job can be managed by SM36 job control.

When data writes out on files the file extension will be ADK(Archive Development Kit).

This method is not intended to provide a destruction of transactional or master data. A simple destruction or deletion of ADK type files may cause data inconsistency.

The primary tasks involved in Data Archiving are:
Write: To create archive files
Delete: Removed the objects from the SAP DB after archived
Read: Read and review archived objects
Restored: Restore the Archive files back to SAP DB, if needed be

Here are the high-level steps involved in archiving data:

Define the archiving objects: An archiving object is a group of related data tables that you want to archive together. For example, you might define an archiving object for all sales orders that are more than 4 years old. Archiving objects are defined using the Archive Development Kit (ADK), which is part of the SAP system.
Configure the archiving process: The archiving process is configured using the Customizing function in the SAP system. This involves setting up a number of parameters, such as the retention period for the archived data, the storage location, and the archiving mode (such as immediate or scheduled).
Archive the data: Once the archiving objects are defined and the process is configured, you can start the archiving process. This involves running a series of SAP programs that extract the data from the active system, transform it as necessary, and store it in the designated storage location.
Retrieve and use the archived data: After the data is archived, it can be retrieved and used as needed.

Links:
SAP Blog: SAP S/4 HANA Data Management (Aging/Archiving) blog:
A great SAP blog with details steps for data archiving process :
https://blogs.sap.com/2020/10/07/sap-archiving-process-and-configuration-steps/

SAP Content Server: is a lightweight solution for SAP document archiving that originally has been used for SAP environments. There is no licensing cost for the software, you only take into account the cost of infrastructure and OS in your provisioning. At the time of writing this blog SAP Content Server still only support MaxDB.

You can improve the performance of the SAP Content Server by separating database server from HTTP server and cache server. Normally using database instead of file system has more benefit to administrate a large amount of data and searching contents.

Archivelink :can be used from the SAP system to an SAP Content server.
It is worth to mention SAP Content Server is not ILM enabled and cannot use SAP ILM features.
You can deploy SAP Content Server in hyperscaler like Azure cloud in the same VNET your SAP landscape resides with a separate cache server:

SAP Content Server in Azure

Links:
A great Microsoft Article if you are going to deploy Content Server in Azure
For more information on the Content Server and SAP MaxDB, see SAP Notes: 1619726, 2786364
SAP Blog: SAP Content Server 7.53 Installation and Connectivity with ERP using SSL
Installation of SAP Content Server 7.5 and Higher on UNIX
Installation of SAP Content Server 7.5 and Higher on Windows

2-SAP Information Lifecycle Management (ILM)
SAP ILM is a built-in business function in SAP NetWeaver that doesn't require any additional installation. This functionality offers a suite of tools for managing data across its lifecycle, such as archiving, retention, GDPR compliance, and disposal. By integrating with ECM, ILM elevates SAP data archiving to a new level, providing advanced Data Lifecycle Management capabilities. With ILM, historical data can be moved from ECC system to a separate ECM repository, while the retention and disposition of the data can be efficiently managed, ensuring that it is preserved as long as necessary and securely disposed of once it's no longer required.

SAP ILM requires an additional license and is essential for organizations managing high data volumes with regulatory compliance needs.
Three main functions of SAP ILM

One of core roles of SAP ILM is to move the data from the legacy system to archive files and then archive file can be compressed and stay read-only to prevent any change.
SAP ILM is capable to apply data retention rules to archive files that makes it possible to comply to any requirements to destroy data when it is not needed.
SAP ILM support different interfaces to store data like WORM based storage, cloud BLOB, and SAP IQ

The diagram below illustrates the evolution of SAP ILM over the years:

Evolution From Data Archiving to ILM

SAP ILM Retention Warehouse: Is used to support data from SAP and Non-SAP applications after system decommissioned. It runs in its own NetWeaver service to support the data retrieval after live SAP application decommissioned. This system can be run on premises or cloud.

Capability of SAP Information Lifecycle Management

The picture below shows the overview of SAP ILM options:

As you can see, there are verity of options if we use ILM store, specially nowadays you can connect your SAP system to a public cloud and use the advantage of cloud BLOB storage and even integrate cloud storage lifecycle management in your solution.

When it comes to understand the type of data in ERP, we need to consider we have OLAP and OLTP data in ERP systems. SAP BW and BW/4 can support OLAP data archiving and for OLTP data SAP IQ Near-Line Storage Functionality is the best choice.

SAP ILM with SYBASE IQ: can support the general storage of BLOB,ILM-controlled archive files and archived indexes from your OLTP DB system in your ERP. The main advantage of using SAP IQ is to support Near-Line Storage functionality. SAP IQ can compress SAP HANA data up to 80% of its original size since it used column-based compressing.

In the past customers used to use NLS for OLAP and Archivelink or WebDAV storage for OLTP apps. ILM Retention management with Sybase IQ addressed this challenge to avoid having separate storages for this archiving.

Basically, the NetWeaver ILM functionality of Sybase IQ can support operation data (OLTP) along with OLAP data that can be supported by BW NLS interface in Sybase IQ.

ILM Store with SAP IQ

Links:
A great SAP Blogs by @RolandKramer explained a very detail of ILM with SAP IQ:
SAP ILM with SAP IQ Database
SAP (Sybase) IQ – the hidden treasure …

Other references:
Quick Configuration Guide for the ILM Store
SAP NLS Solution for BW(4) - Overview 2021 - 12.2021
Should you archive both BW (or BW/4HANA) and ERP (pre-S/4HANA) on the same instance of SAP IQ?
Note 3085053 – Landscape Considerations for NLS and ILM
OLAP Implementation - SAP-NLS Solution for SAP BW (all Versions)
OLTP Implementation - SAP ILM based on SAP IQ Database

SAP ILM with Azure BLOB: There is a pre-built Azure adapter as a package provided by SAP DMLT team can be leveraged to connect your ERP system to Azure BLOB as an external storage. SAP ILM Store component can enable storing of archive files and attachments from your SAP systems into Azure Blob storage:

SAP ILM with Azure Blob

SAP ILM with Microsoft Azure Blob
For more information about using Azure BLOB for ILM, you can check the blog below:
https://blogs.sap.com/2020/04/24/using-azure-blob-for-storing-files-in-sap/

SAP application: Minimum recommended version using simplified blocking and deletion	SAP Notes
SAP ERP 6.0 EHP7 SP12
SAP Supplier Relationship Management 7.03 EHP3 SP05	2011539
SAP Customer Relationship Management 7.0 EHP3 SP05	2039738
IS-UT EHP7	2591185
SAP ERP Human Capital Management 6.0 EHP6 SP16	1559133

1825544 – Simplified Deletion and Blocking of Personal Data in SAP Business Suite
1825608 – Simplified Blocking and Deletion of Central Business Partner
2122906 – ILM: List of ILM objects with assigned archiving/data destruction objects
2590321 – Upgrade recommendations to support GDPR compliance

3-S/4 HANA integration with External Storage
Integration with SharePoint:
SAP provides mechanisms to configure Microsoft SharePoint as a content repository using the ArchiveLink interface. This setup allows for the storage of archived documents and attachments directly within SharePoint. For instance, CaRD offers an ArchiveLink interface that connects SAP systems with SharePoint, enabling it to function as a content server for SAP. This integration is compatible with S/4HANA and supports various attachment types across different SAP interfaces.

YAC stands for universal Attachments Connector and is an SAP add-on that facilitates the integration of attachments and documents from SAP ERP and S/4HANA systems with various Document Management Systems (DMS) both in the cloud and on-premises

Integration with Dropbox:
Integrating SAP S/4HANA with Dropbox is less straightforward but achievable through third-party integration platforms. For example, MIP offers prebuilt integration packages that connect SAP and Dropbox systems, facilitating uninterrupted data flow and automated workflows between the two platforms.

Alfesco Integration with S/4 HANA:
Alfresco offers a Content Connector (add-on module) specifically designed for seamless integration with SAP applications, including SAP S/4HANA. This connector links structured data within SAP with unstructured content in Alfresco, allowing organizations to manage documents and records effectively.

Integration to external storage types:

4-Third-party archiving and ECM solutions
There are a number of third-party solutions available for SAP archiving and ECM data, including solutions from vendors such as OpenText, Dolphin, and PBS Software. These solutions typically provide advanced archiving capabilities, such as automatic data aging, retention management, and secure disposal.

OpenText:
SAP (Extended) ECM by OpenText

OpenText ECM can be integrated with SAP ILM to provide a comprehensive content management solution for managing data throughout its lifecycle.

SAP ECM by OpenText

Main benefits of SAP ECM by OpenText:

This integration allows businesses to move historical data out of their primary SAP system and store it in an OpenText ECM repository, which can help to reduce data volume in the primary SAP system.
OpenText ECM provides features such as document imaging, workflow automation, and records management to help businesses manage all types of content, including documents, records, images, and multimedia files.
The combination of SAP ILM and OpenText ECM provides a robust solution for managing data volume and ensuring compliance with industry regulations. This can help businesses to reduce storage costs and improve system performance, while also minimizing legal and regulatory risks associated with data management.
SAP Extended ECM allows users of non-SAP applications such as Microsoft Office access to SAP information.
SAP Extended ECM by OpenText includes pre-built integrations with SAP applications, enabling businesses to seamlessly manage content from within their SAP environment. This can improve productivity and reduce errors by eliminating the need for users to switch between systems.
SAP Extended ECM by OpenText provides enhanced records management capabilities, enabling businesses to manage records throughout their lifecycle, from creation to disposition. This can help businesses comply with regulatory requirements and reduce legal risks.
SAP extended ECM tightly integrates unstructured contents with transactional SAP processes. Complete customer records that include unstructured data like emails, contracts and offers made easily accessible across the enterprise to compliance officers, customer supports, and operations teams.
SAP Extended ECM by OpenText includes advanced workflow automation features, enabling businesses to automate document-centric processes such as invoice processing, contract management, and HR onboarding. This can improve efficiency and reduce costs by eliminating manual tasks and reducing errors.

Links:
OpenText Solution for SAP
Demo for SAP Extended ECM

5-SAP Data Services
Data Services formerly known as BODS is a data integration and transformation tool that can be used to extract data from your ECC system and load it into a separate data warehouse or other storage location. Data Services includes a range of data quality and cleansing capabilities that can help to ensure the accuracy and completeness of the data as it is moved.

SAP Data Services and System Decommissioning

6-Export and load
You can also consider a simple export and load approach, where you extract the historical data from your ERP system and load it into a separate database or file system for future reference. This approach is more manual and may require some data cleansing and transformation, but it can be an effective way to quickly move the data without investing in specialized tools.

MuleSoft: is not specifically designed for data archiving, it can be used to extract data from SAP systems and transform it for use in other systems or locations.

Extract data using SAP connectors: MuleSoft provides connectors for SAP systems, which can be used to extract data from SAP tables and objects. By using these connectors, you can pull historical data from your SAP ERP system and transfer it to another location, such as a data warehouse or file system.

Note: MuleSoft is one of popular third-parties software has been used for transforming data within SAP systems I have in this section as an example
Link:
SAP Connector 5.7 - Mule 4

Conclusions

Managing data volume in SAP can be challenging, and it's critical to have a comprehensive understanding of available solutions and techniques. To effectively manage data, it's important to evaluate options and select a solution that aligns with your specific needs and requirements, such as data retention policies, security requirements, and budget constraints. Seeking guidance from experienced SAP data consultants or the SAP DMLT team can also be helpful in planning and executing the data management as part of your project. With the right resources and approach, managing data growth in SAP can be streamlined, ensuring that your organization is equipped to handle increasing data volumes effectively.

We also have to remember how far we want to go in this journey, normally for ERP data management we can start with archiving method first then enable ILM and go to further strategic approach with ECM solutions.

SAP Training and Education:

BIT660 – Data Archiving
BIT665 – SAP Information Lifecycle Management
BIT670 – ILM/data archiving – customer specific development (available only in Germany)
BIT65H – SAP Information Lifecycle Management (ILM) (half-day remote training)

References:
SAP Solution for Information Lifecycle Management presentation
SAP Information Lifecycle Management to Reduce Business Risk and Costs pdf
Information Lifecycle Management Capabilities for Data Protection and Privacy pdf
SAP Information Lifecycle Management
Data Management and Landscape Transformation | Implementation Service for SAP ILM Service
Explore SAP Services
ASUG Asks the Authors: SAP Information Lifecycle Management

Share with others and Connect with us!

Please leave your comment if you have anything to add!

If you would like to ask questions, please use the community Q&A.

Give us a like and share on social media if you feel it was useful
You can follow me in People SAP @Amin_Omidy
Thanks!