- SAP Managed Tags:
This doc is wrote for describing how to solve media file could not be auto removed issue happened on Azure storage account, during Cloud Hot Folder file processing progress.
Impact : Cloud Hot Folder file processing stuck.
Background:
Here is some background about the issue.
Cloud Hot Folder functionality has been enabled on Commerce Cloud via Azure Storage Account.
Here is a picture for you to have an understanding on original Hot Folder.
Mechanism behinds it is Spring integration periodically scans the configured input directory for new files.
If new files are found exists on specific folder, they are moved to the processing subdirectory and then sent to the Batch Import pipeline, which consists of the following tasks:
When it comes to Commerce Cloud, specific folder moved to Azure Storage Account. one blob storage container will be created for file uploading, files can be uploaded and stored quickly and safely.
Issue happened when file processing progress proceeds to Cleanup Task.
From original hot folders (see Data Import) you'll be familiar with the archive and error folders, directories created within your hot folder by the CleanupTask/ErrorHandler.
If the data import completed without exception, The CleanupTask deletes the header and transformed files and the imported file is then moved to the archive subdirectory,
if however an error occurred during any of the tasks, the ErrorHandler will move it to the error subdirectory and take care of any clean up.
In Cloud Hot Folders, we have aimed to replicate this behaviour, by running each polled message in its own transaction, with a defined commit and rollback behaviour.
This is set up against our in-bound channel adapter.
Issue happened during removing file on uploaded folder on Azure storage account, as the screenshot displayed one of media file name Medias_20210901_183029.xml failed to be auto removed as expectation.
when we looked into the detail found the remove activity was blocked by snapshot of file Medias_20210901_183029.xml. file on Azure storage account could be deleted when snapshot file exists.
About the concept of the blob snapshot please refer to : https://docs.microsoft.com/en-us/azure/storage/blobs/snapshots-overview
Cleanup Task blocked snapshots behaviour in screenshot.
Solution:
To solve media file could not be deleted issue, snapshot file should be removed manually first.
After that blocked media file could be deleted by API command or operation on Azure portal manually.
Impact : Cloud Hot Folder file processing stuck.
Background:
Here is some background about the issue.
Cloud Hot Folder functionality has been enabled on Commerce Cloud via Azure Storage Account.
Here is a picture for you to have an understanding on original Hot Folder.
Mechanism behinds it is Spring integration periodically scans the configured input directory for new files.
If new files are found exists on specific folder, they are moved to the processing subdirectory and then sent to the Batch Import pipeline, which consists of the following tasks:
- HeaderSetupTask: Creates a new BatchHeader.
- HeaderInitTask: Retrieves a sequence ID and (optionally) a language from the file name.
- ImpexTransformerTask: Creates one or many ImpEx files from the CSV input and writes error lines to the error subdirectory.
- ImpexRunnerTask: Processes all ImpEx files sequentially with multiple threads.
- CleanupTask: Deletes all transformed files and moves the imported file with an optionally appended timestamp to the archive subdirectory.
- ErrorHandler: Deletes all transformed files and moves the imported file with an optionally appended timestamp to the error subdirectory.
When it comes to Commerce Cloud, specific folder moved to Azure Storage Account. one blob storage container will be created for file uploading, files can be uploaded and stored quickly and safely.
Issue happened when file processing progress proceeds to Cleanup Task.
From original hot folders (see Data Import) you'll be familiar with the archive and error folders, directories created within your hot folder by the CleanupTask/ErrorHandler.
If the data import completed without exception, The CleanupTask deletes the header and transformed files and the imported file is then moved to the archive subdirectory,
if however an error occurred during any of the tasks, the ErrorHandler will move it to the error subdirectory and take care of any clean up.
In Cloud Hot Folders, we have aimed to replicate this behaviour, by running each polled message in its own transaction, with a defined commit and rollback behaviour.
This is set up against our in-bound channel adapter.
Issue happened during removing file on uploaded folder on Azure storage account, as the screenshot displayed one of media file name Medias_20210901_183029.xml failed to be auto removed as expectation.
when we looked into the detail found the remove activity was blocked by snapshot of file Medias_20210901_183029.xml. file on Azure storage account could be deleted when snapshot file exists.
About the concept of the blob snapshot please refer to : https://docs.microsoft.com/en-us/azure/storage/blobs/snapshots-overview
Cleanup Task blocked snapshots behaviour in screenshot.
Solution:
To solve media file could not be deleted issue, snapshot file should be removed manually first.
After that blocked media file could be deleted by API command or operation on Azure portal manually.
