What’s New in SAP Datasphere Version 2024.2

Data Cataloging

1 Data products are data purchased from a third-party provider. Listed data products available in data marketplace are searchable in the catalog. Catalog User Interface Overview

You can perform a free-text search across the entire catalog for data assets, business glossary terms, tags, data products, and key performance indicators (KPIs). Data products must be listed in Data Marketplace before they appear in the catalog. A data product is either free or purchased data from a third-party provider that you can use in this product.

Data Integration

1 When replicating data to Parquet files in a cloud storage provider, you can now have the time data type columns converted into int64 (microseconds after midnight) so that they can be consumed by Apache Spark. Using a Cloud Storage Provider As the Target

When you use a cloud storage provider as the target for your replication flow, you need to consider additional specifics and conditions. One of the target settings is Enable Apache Spark Compatibility that allows to convert and store time data type columns to int64 in the Parquet files. The int64 data type represents microseconds after midnight. This conversion allows the columns to be consumed by Apache Spark.

2 You can now create Microsoft SQL Server connections with the version Microsoft SQL Server 2022 to connect to databases with this version. Microsoft SQL Server Connections

Once you create Microsoft SQL Server connection you need to give details including Server Name, Port, Database Name and Version which Microsoft SQL Server 2012 to 2022 (default is 2022) versions are supported.

3 When replicating data to a cloud storage provider, you now have the option to suppress duplicate records in the target. Using a Cloud Storage Provider As the Target

When you use a cloud storage provider as the target for your replication flow, you need to consider additional specifics and conditions. One of the target settings is Suppress Duplicates that allows to avoid duplicate records in your target file. During initial load, if a data record already exists in the target, the default system behavior with cloud storage provider targets is to write this record to the target once again, which results in duplicate records. If this is not the desired behavior for your use case, you can change it by enabling this option.

4 When you run a transformation flow that loads delta changes to a target table, the system uses a watermark (a timestamp) to track the data that has been transferred. You can now view the watermark for a source table. You can also reset the watermark if required. If you reset a watermark, the system will transfer all data to the target table the next time the transformation flow runs (using the load type Initial and Delta). Watermarks

If you reset the watermark you do not need to redeploy the transformation flow and use the load type Initial Only. You may need this feature for example:

If a table that can capture delta changes is joined with a second table, and columns in the second table have been updated.

If corrupt data is present in the target table.

Data Modeling

1 Identify available table structure updates for all tables sharing the same source connection, and resolve conflicting errors resulting from these updates. Process Source Changes for Several Remote Tables

When changes are made in source models, they might not be reflected immediately in SAP Datasphere. This can result in errors and impact on dependent objects, and runtimes, and you need to do a refresh to get these updates in your remote table definition. So you need to Validate Remote Tables then the remote tables with incompatible changes will get the status Runtime Error. Then you can select the changes you want to apply to your remote table and redeploy the remote table.