Migrate to SAP Commerce Cloud: Migrate Media with AzCopy

[ythomas_schroeder]

 

AzCopy: A fast, simple way for SAP Commerce Media Migration

AzCopy  is a command-line utility that can be used to copy blobs or files to or from an Azure Blob Storage account. This article describes the migration of SAP Commerce media files to SAP Commerce Cloud Media Storage. Before reading this article it is recommended that you review the introduction in Migrating to SAP Commerce Cloud: Data and Media Migration.

Overview

During a media migration your media are copied from the existing on-premise source to the target instance of SAP Commerce Cloud. The migration occurs in a like-for-like environment to ensure data segregation and security, that is, for example, Development to Development, Staging to Staging, and Production to Production.

AzCopy Overview

Migration Phases

The following diagram outlines how you can go about migrating your media. More detailed explanations for each of the phases are covered in individual sections below.

AzCopy Flow

Planning

This phase should define which environments will be involved in the media migration: the typical setup involves three environments (Development, Stage, and Production), however, every customer is different and you may find you need to migrate fewer or additional environments. For example, some development environments might be re-initialized when deployed. Therefore, their data and media could be excluded from a particular migration project. The actual migration of media should be performed environment-by-environment, tweaking and improving along the way.

Setup

SAS Token

A Shared Access Signature (SAS) Token is a single-use identity token URI that grants restricted access rights to Azure Storage resources. A client who is in possession of the SAS token can make a request against Azure Storage with just the SAS URI, and the information contained in the SAS token is used to authorize the request. In order to be able to copy the media files to the targeted SAP Commerce Cloud media blob storage, a temporary SAS Token will need to be created for each environment via a Support Ticket. One Ticket will be needed per environment. SAS Tokens are set to expire after 2 weeks. When the SAS Token expires a new request will be needed. Please allow up to 1 week for ticket to be executed. The following example shows how that request would be created. Make sure that the environment name is mentioned, because multiple environments of the same type can exist.

Please note that ALL the information in the ticket between square brackets are mandatory. The IP to be allowlisted is particularly important as it will be the only IP which is allowed communication to the media blob store in SAP Commerce Cloud.

Media Migration to Commerce Cloud with AZCopy - SAS Tokens

Commerce Subscription ID: [link to your Cloud Portal subscription]

Storage Account Name: [Storage Account Name]

Cloud Environment: [Commerce Cloud Environment Name]

Purpose: On Prem to Commerce Cloud Media Migration

Incoming IP Address: [100.100.100.101]

Please create a SAS Token for the cloud environment mentioned above and send them. Thank you.

AzCopy - Contact Support

Installation

Ideally, the you would want to run AzCopy on a system that has direct access to the source media files, such as an application server with a mounted NFS folder containing the sys_master folder of the SAP Commerce media. The following script shows how you can run AzCopy:

# Log in to the server as a regular user

# Download the tar.gz file to the home folder:
$ wget https://aka.ms/downloadazcopy-v10-linux

# Untar the archived files:
$ tar -xvf downloadazcopy-v10-linux


# Go into to the untar'ed folder to locate AzCopy executable:
$ cd azcopy_linux_amd64_10.0.4/

# Verify that the program can run:
$ ./azcopy

Testing

Once the allowlisting is complete and you're in possession of the needed SAS tokens, a browser can be used to test access using a URL similar to the example below. The Storage Account Name is set in the media.globalSettings.cloudAzureBlobStorageStrategy.connection property that can be read in the hybris Admin Console (hAC) of one of your environment endpoints.

https://[storage account name].blob.core.windows.net/[container name]?[SAS Token]&comp=list&restype=container

Once the AzCopy installation is complete and library is accessible, test connectivity by listing the contents of a container from the SAP Commerce Cloud's media storage.

#list destination to see if connected and accessible
./azcopy list "https://[storage account name].blob.core.windows.net/[container name]?[SAS Token]"

Executing

For media storage SAP Commerce provides different strategies which define how and where medias are stored - see Media Storage Overview for more details. Usually customers are using localFileMediaStorageStrategy which stores media on the file system. For SAP Commerce Cloud the cloudAzureBlobStorageStrategy is used, which stores media in Azure Blob Storage and is using containers to structure the data.

Simple mapping

When a build is deployed to an SAP Commerce Cloud environment with the initialization option, by default, SAP Commerce will create the following containers for the typically used media types.

Media Type	Source Location	Source Folder	Target Container	Target Folder
root	.../sys_master	/	sys-master-root	/
images	.../sys_master	/images	sys-master-images	/images
impex	.../sys_master	/impex	sys-master-impex	/impex
cronjob	.../sys_master	/cronjob	sys-master-cronjob	/cronjob
catalogsync	.../sys_master	/catalogsync	sys-master-catalogsync	/catalogsync
jasperreports	.../sys_master	/jasperreports	sys-master-jasperreports	/jasperreports

For each Media type, an AzCopy command is needed to copy the data to the correct container. In the following script example, it will copy the catalogsync from the source and fully copy that folder and all its content into the target container. With this block of code, scripting the migration to handle all media types should be straight forward.

#catalogsync example
./azcopy copy "../sys_master/catalogsync" "https://[storage account name].blob.core.windows.net/sys-master-catalogsync/catalogsync?[SAS Token]" --recursive=true --log-level ERROR

Some customers might have other media types created from customizations utilizing the Commerce Media Storage Strategy. In general the mapping should be done by analyzing the table "mediafolders"; the "p_qualifier" column is the media type and the "p_path" is the folder name in the old system. Now the mapping should map from .../sys-master/<path> to sys-master-<qualifier>/<path>.

Media Type	Source Location	Source Folder	Target Container	Target Folder
customMediaType	.../sys_master	/customMediaType	sys-master-customMediaType	/customMediaType

The strategy to migrate those custom media types is the same, with the additional operation to create the target container that will ultimately get the migrated files.

#customMediaType example
./azcopy make "https://[storage account name].blob.core.windows.net/sys-master-customMediaType?[SAS Token]

./azcopy copy "../sys_master/customMediaType" "https://[storage account name].blob.core.windows.net/sys-master-customMediaType/customMediaType?[SAS Token]" --recursive=true --log-level ERROR

Complex mapping

The mapping gets a little bit more complicated if subfolders are used. Here is an example: Let's assume a customer has two brands ("fishy" and "tonic") and stores the images for the different brands in different folders. To make the example more interesting the qualifier for the media have the old-brand names in it ("salt" and "pepper"). The mediafolders tables has these entries:

p_qualifier	p_path
images-salt	images/fishy
images-pepper	images/tonic

With the localFileMediaStorageStrategy the different images are stored in

• sys-master/images/fishy
• sys-master/images/tonic

In SAP Commerce Cloud these have to go into the new blob containers

• sys-master-images-salt
• sys-master-images-pepper

The mapping should look like this

Source Location	Source Folder	Target Container	Target Folder
.../sys-master	/images/fishy	sys-master-images-salt	/images/fishy
.../sys-master	/images/tonic	sys-master-images-pepper	/images/tonic

Determine mapping

It is not possible by just looking at the folder structure and folder names in the local filesystem to determine the correct container names and folders in Blob Storage. The correct mapping can only be done by also looking at the mediafolders table.

Mapping for other Media Storage Strategies

The mapping that was described so far is only for localFileMediaStorageStrategy, if another Media Storage Strategy is used in the source system the mapping might be different.

Here is for example the simple mapping for the s3MediaStorageStrategy for Amazon S3 storage:

Folder in AWS S3	Container and Folder in Blob Store
.../sys-master/catalogsync	sys-master-catalogsync/sys-master/catalogsync
.../sys-master/cronjob	sys-master-cronjob/sys-master/cronjob
.../sys-master/images	sys-master-images/sys-master/images
.../sys-master/impex	sys-master-impex/sys-master/impex
.../sys-master/jasperreports	sys-master-jasperreports/sys-master/jasperreports
.../sys-master/root/	sys-master-root/sys-master/root

Large folders

AzCopy creates a lot of log files, which is a nice feature because it allows it to continue working where it left off if the process gets interrupted. In case of very large folders (e.g. the images folder) it might be a problem, because the disk that stores the logs would run out of space. A mitigation could be to perform the copy process not for the complete images folder with one call to AzCopy, but split the task in several chunks. After each successful chunk the logs can be deleted with

./azcopy jobs clean

Renaming

As was explained earlier the container name in Azure Blob Store is derived from the "p_qualifier" column in the table "mediafolders". This can cause some problems, because there are naming restrictions for container names in Azure (see https://docs.microsoft.com/en-us/rest/api/storageservices/Naming-and-Referencing-Containers--Blobs--...). This means that you may have to adjust the container name that can be used and also the content in the "mediafolders" table.

Delta execution

AzCopy can also support delta migration using the sync option to facilitate pre-cutover and cutover activities reducing the migration time further. This approach is meant to be an extension to the current process, which will give customers a different option to use in case a large amount of media is to be migrated.

Conclusion

You should now have an understanding of the fundamental steps of migrating media into your SAP Commerce Cloud in the Public Cloud environment(s). For more information regarding migration please see the rest of the articles in the Migrate to SAP Commerce Cloud series.