S3 Media sync

[former_member1026091]

Hello,

we want to synchronize the s3 media bucket from Production to the Test environments and we are thinking to use S3 Replication. However the DB will be synchronized twice at year though. Would that work?

Should the S3 bucket be in sync with the DB or it's not needed?

Thanks

[christoph_probst]

With your restriction (de-coupled media and db sync) I'd try the following:

0. Configure the S3 replication

Now media binary sync can be ignored for the next steps and only the state of the DB has to be taken in consideration. Problem is that the creation of media files will fail when dataPk is already taken.

1. Restore production DB Dump on test system (with all necessary anonymization)

2. Adapt dataPk counter by adding an offset (e.g. 10 million) to the current counter value.

3. Start sap commerce

Insights on dataPk

For the dataPk the same numberseries is used as for the Media PK itself. (Media Type Code = 30). The value in question can found with

select * from numberseries where serieskey = 'pk_30'

[christoph_probst]

When it comes to SAP commerce it is recommended to have DB and Media storage not diverging and more important anonymize production data.

The problem which may occur is that the mediaPK for a new entry is already taken by a file on the storage. When this happens the creation of new media fails. It is not that problematic when the media storage is behind the DB then only some medias can not be loaded.

Therefore SAP created a snapshot restore feature which takes this in consideration for CCv2. https://help.sap.com/docs/SAP_COMMERCE_CLOUD_PUBLIC_CLOUD/0fa6bcf4736c46f78c248512391eb467/f777fc0a4...

Hint: To avoid the collision during media creation it is possible to increase the counter of the mediaPK (this is not the PK of Media entity) to the current number of production database.

[former_member1026091]

Unfortunately we are no using SAP Cloud. So as for now we have some jobs that copys object from the prod s3 to a folder, named based on a date, in another bucket and then when it comes to sync we copy from that folder to the test bucket. At the same time we take a snapshot of the DB and restore to Test. The problem is that this process takes ages, especially when it comes to copy terabytes of data from one bucket to another.

So I'm looking for a faster way to copy objects between buckets. S3 replication would have been a great option because once has been setup there is nothing left to do.

Any suggestion?

I'm not expert on SAP Hybris, on the contrary I know very little. I'm just a Cloud Architect and trying to improve some of the processes.