Automate Datasphere Objects' transport with datasphere-cli

[amiramarouani]

SAP Datasphere #DevOps #Transport #Automation #Git

Overview:
SAP Datasphere provides built-in transport capabilities for promoting artifacts between tenants along with cTMS which is another tool that can be leverage for objects promotions across a multi-tier landscape. However, for teams already working with Git-based workflows where peer review, branching strategies, version control and deployment pipelines are standard practice, there is a natural desire to bring that same discipline to Datasphere artifact lifecycle management.

This blog describes how SAP Datasphere CLI (@sap/datasphere-cli) can be used to build a lightweight, Git-native transport pipeline that fits into existing developer workflows and automates this process end-to-end.

Followed Approach:
The pipeline is built around a simple principle: Git as the intermediary or staging layer. Artifacts are exported from a source Datasphere tenant, versioned in GitLab, and imported into a target tenant, all via scripted datasphere-cli calls. No manual file handling, no browser exports.

The workflow runs in two directions:

1. DSP → Git (Pull): A dedicated script connects to the source tenant, discovers all artifacts by specified types, and compares the full content hash of each artifact against the locally stored version. Only genuinely changed artifact or new artifacts are written to an export folder. Once the pull completes, the developer reviews the changes in VS Code Source Control, selects which files to stage, and pushes to GitLab giving full control and flexibility over what gets pulled from Datasphere and versioned.
2. Git → DSP (Push): A second script scans Git history for changed/new files and presents an interactive numbered list, allowing the user to select exactly which artifacts to transport. It then imports each artifact into the target tenant using datasphere-cli create command, falling back to update on conflict, and verifies each import with a post-import read. A full audit log captures every outcome.

How does datasphere-cli make it possible?
Beyond artifact transport, datasphere-cli exposes a broad set of capabilities that can be leveraged to automate surrounding workflows:

• Artifact lifecycle: discover, read, create, update across all major object types including local tables, views, replication flows and analytic models
• Package management: list, create, update, read and export package definitions for dependency-aware transport
• CI/CD integration: the pipeline can be fully automated as every transport operation runs without manual intervention, making it straightforward to integrate into existing CI/CD workflows.

Workflow diagram:

Pre-requisites: 

• SAP Datasphere tenant(s)
• OAuth client credentials configured in SAP Datasphere, with Interactive Usage
• SAP Datasphere CLI installed 
• Git repository (GitLab or GitHub)
• Python 3.9 or above

Authentication: 

1. SAP Datasphere Access is maintained using an OAuth client (client ID and secret) created in Datasphere tenant. These credentials can be stored in the config file or as environment variables which is recommended to avoid storing secrets in plain text. datasphere-cli establishes a session on first run and caches it locally, so subsequent runs reuse the existing session without re-authenticating.
2. GitLab Access is authenticated using a Personal Access Token generated in GitLab. The token is stored in Windows Credential Manager and picked up automatically by Git on every push and pull, with no configuration needed in the scripts.

Dependency Ordering
Transporting multiple artifacts in the right order is crucial as a view cannot be imported before the table it references. By leveraging datasphere-cli objects packages list, create, read and export commands, package manifests can be pulled alongside artifacts. These manifests capture which objects belong to each package and their dependencies, enabling the pipeline to resolve the correct import sequence automatically using a topological sort. Teams that organise their artifacts into Datasphere packages benefit from fully automatic dependency ordering with no manual sequencing required.

Change Detection
Two approaches are supported for detecting whether an artifact has changed since the last pull:

1. Hash comparison: computes a SHA-256 hash of the full artifact content and compares it against the locally stored version. Only genuine content changes trigger a write. This is the recommended default.
2. Timestamp comparison: compares Datasphere's internal deployment timestamp against the local file. Less precise as redeploying an object in Datasphere without structural changes still updates its timestamp, triggering an unnecessary write.

Audting:
Every action (pull, push) is written to a structured log file with a timestamp, status, and detail, making it straightforward to trace what happened and when across any transport run.

Important Considerations:

• Authentication Expiry: OAuth client credentials configured for each tenant have a maximum validity period of 180 days. Once expired, all pipeline operations (pull, push) will fail to authenticate. It is recommended to track the expiry date of each client credential and renew them in advance to avoid unexpected transport failures.
• Merge requests/pre-import conflict detection: Before importing, the pipeline checks whether the artifact already exists in the target space. Three outcomes are possible: the artifact is new and gets created, it exists with identical content and is skipped, or it exists with different content and the operator is prompted to resolve the conflict before proceeding.
• Credentials handling (e.g connections, access tokens): Connection definitions can be transported as part of the pipeline, credentials are never stored in the exported artifact but can be retrieved securely at import time from a secrets manager (e.g. AWS Secrets Manager, Azure Key Vault, etc) keeping the transport fully automated without exposing sensitive credentials in Git.
• Transport across spaces: The same pipeline logic applies to transporting artifacts across spaces within the same tenant. Rather than pointing the source and target configurations to different tenants, both can be configured with the same host but different space IDs. Pull, conflict detection, and push all behave identically. Git remains a useful intermediary in this scenario, providing a version-controlled audit trail of what moved between spaces and when.
• Scalability and performance: Initial tests covered local tables, views, and flows transported individually and as part of a package with consistent results. For larger spaces, performance scales linearly as artifacts are processed one at a time, and future iterations could address this through parallelisation and batched API calls. For high-volume migrations, a performance comparison against cTMS would be a worthwhile exercise and the right choice ultimately depends on whether the team favours a Git-native workflow or centrally managed package releases.

Conclusion:
This approach is not a replacement for Datasphere's native transport capabilities or cTMS. It is a solution for teams running Datasphere through automation or DevOps pipelines. Version control, branching, code review, and automated deployment are now part of how Datasphere artifacts can be managed, all made possible through datasphere-cli.

[Martin_Kuma]

Thank you for the step by step, just a quick question. How do you automate the package deployment via CLI please? I mean I can do the export, but how do you automate import via CLI? What command do you use please? 

https://help.sap.com/docs/SAP_DATASPHERE/d0ecd6f297ac40249072a44df0549c1a/9dfeea1833ce4b61a3304e0dbb...

I have 2026.14.0 version installed. No import command for object packages exists, just export. Documentation, nor J4C found any import command. 

Thanks in advance, Martin 

[reyemsaibot]

Hi Martin,

I think he is downloading the JSON Files for each object and uploading them again. Maybe I am wrong?

Tobias

[Martin_Kuma]

Thanks Tobias, got it now. So not entire package, just the objects. Object auto-deploy during create/update. Makes sense. 

[dirkt]

Thank you for the contribution! I would like to build on Martin’s question.

I can still easily understand the automation of deployments, as well as the need to first define an object in the form of a JSON file that contains all required properties and configurations. However, this also seems to imply that an initial development step in Datasphere is always required in order to create such an object, which can then be copied and deployed to other tenants or spaces.

I have a specific use case involving the creation of Replication Flows. The source and target connections are always the same. The only parameter that changes from one Replication Flow to the next is the extractor (CDS View).

Given this scenario, is it really necessary to first create a Replication Flow in Datasphere and then export it as JSON? Or would it be possible to directly read the metadata/schema of a CDS View within Datasphere and automatically generate a fully functional JSON definition that can then be deployed without any prior manual development step?

[ferdswich]

Hello,

As it should not replace CTMS,

could we use it only in DEV for our developers to boost their coding capabilities with AI in VS Code for example ? 

And still handle transport DEV > UAT > PROD with CTMS.

Thank you in advance for your advice,

[JulianJuraske]

Hello @dirkt ,

yes this can be done, but not by standard ;).
Have a look at the SAP Help for the CLI capabilites https://help.sap.com/docs/SAP_DATASPHERE/d0ecd6f297ac40249072a44df0549c1a/e3b2cbef9d1f4b38b51c64fe9e...
and than you have to implement a Solution arround that (e.g. python) that reads the CDS-View informations (it has to connect against your S/4 and read meta data like associations, semantics ect.) and transforms it into DSP-Configurations. As the last step you can build (create/update) in mass in DSP using the DSP CLI to e.g. create the Replication Flows with the local Table as a Target. (in future maybe aswell the DataProducts directly)
Than you don't have to use the SAP DSP UI at all, but can work instead with the SAP DSP CLI, you have a standaradized way (no manual errors), all object have the same naming convention ect.
Maybe a SAP Partner build something like this already? (Yes we have 🙂 )

@Martin_Kuma 
I'm also just aware of the export Function for Packages via the CLI, an Import function is (still) missing and hopefully soon coming.
Otherwise the above workarround (this blog) requires you to individually create/update Objects on your PROD Tenant based on a Script (you have human in the loop verfiying the Objects to be transported).
I think we can all agree that we are missing the abap based Transport from the SAP BW, where you could also lock objects.

[amiramarouani]

Hi @Martin_Kuma, 

Thank you for your comment. As of today, there is no specific command for datasphere-cli to import packages in Datasphere. In this solution, packages serve a different but equally important purpose: dependency resolution. By reading the package manifest and storing it in Git, the pipeline knows exactly which objects belong together and in what order they need to be imported without scanning the entire space. This makes transports faster and more targeted. The pipeline also gives the operator flexibility at export time as you can choose which changed artifacts to commit and transport, though this step can be fully automated as well for CI/CD scenarios.. The result is not a native package import, but it achieves the same outcome: all dependent objects land in the target space in the correct order, with full traceability.

[amiramarouani]

Hi @dirkt, 

Great question. The current approach assumes that an initial modeling/creation step in Datasphere is done: you build the object in the UI, export it, and then automate from there. 

For your use-case, since your replication flows share the same source and target connections and only differ by the CDS view, theoratically, you could template the JSON file. Export one replication flow as a baseline, then programmatically generate new ones by substituting only the extractor reference in the JSON. No UI step needed for each new flow. The structure stays identical, only the CDS view name changes.

As for reading CDS view metadata directly to auto-generate the full JSON definition, this would depend on whether the metadata exposed via the ABAP source system (e.g. through OData or RFC) gives you enough information to construct a valid replication flow payload. It is theoretically possible but would require mapping the CDS view schema to Datasphere replication flow JSON structure. This would require further testing and validation against datasphere-cli's expected format

The bottom line is: one working JSON as a baseline, parameterised by CDS view name, deployed at scale via script.

[Cyril_Metge]

Hi amiramarouani,

Thank you for sharing this very interesting article. I found your Git-based transport approach particularly valuable and inspiring.

I especially like the idea of using Git as an interme
I have one question regarding the Python scripts mentioned in the article (dsp_pull_hash.py and dsp_push.py). Is the purpose of this blog mainly to present the overall concept and architecture, or have you published (or do you plan to publish) sample implementations of these scripts somewhere?

Having access to a basic example would be extremely helpful for quickly testing the approach and adapting it to our own Datasphere transport processes.

Thanks again for sharing your experience and insights with the community.

Best regards,
Cyril

[amiramarouani]

Hi @Cyril_Metge, 

Thank you for your comment. I am planning to follow-up with a second blog with a step-by-step workflow showcasing how this can be implemented, including the scripts (dsp_pull_hash.py and dsp_push.py).