How to Avoid Excessive Storage Load Caused by Using MPL Attachments for Message Logging

[markus_muenkel]

This blog is part of a series that shall assist Integration Developers to properly address non-functional aspects like resource consumption, performance, and reliability. The previous contribution in this series is Avoid Storing Payloads in the Message Processing Log .

In continuation of the previous contribution we want to focus a bit more on message logging via Message Processing Log (MPL) Attachments and its impact on storage resources.

We see many Integration Flows that are writing MPL Attachments to log the message, in particular the message body (aka payload). This is done via Script flow steps, which execute code snippets like the one shown in the blog Avoid Storing Payloads in the Message Processing Log.

This code is using the following API method  (cf. Product Documentation ) to write an MPL Attachment:

void addAttachmentAsString(String name, String text, String mediaType)

Such Script steps are usually placed in multiple locations within the Integration Flow, as illustrated in the following simple example:



This example assumes that there are two Script flow steps labeled "Log Before" and "Log After", and each of them references a script that writes the message body to an MPL Attachment. A user can then display the corresponding Message Processing Log in the Message Monitoring Web UI and inspect these MPL Attachments.

However this technique has to be used with great care, as explained in the following.

Firstly,  it is important to understand that the MPL Attachments are persisted to the tenant storage, and are accumulating therein. Secondly, the monitoring data, including MPL Attachments, is retained for 30 days by default. This means when an MPL Attachment is written, it will be removed from the storage only after 30 days.

As long as this technique is used for a few Integration Flows during the development and testing phases, this might be fine. But as the number of Integration Flows in your tenant is growing, and so is the message throughput, you can easily end up creating Gigabytes of MPL Attachment data on a daily basis. Such enormous amounts of data can quickly fill the tenant storage. In the worst case this could overload the storage and even result in an outage of the tenant.

Another issue with this approach is the following: Assume you used MPL Attachments for tracing purposes during development and testing of an Integration Flow. Then it can happen that Integration Developers forget to disable these MPL Attachments again when they are no longer required. This is in fact something we frequently observe in productive tenant clusters.

For the development or debugging use case please consider using Integration Flow Tracing (cf.  Enabling Trace for Message Processing  for recent enhancements) . Following are some facts about this feature that are relevant in this context:

• The tracing mode expires automatically after 10 min


• The trace data is cleared from the tenant storage after 60 min


• The Integration Flow does not have to be modified and redeployed in order to enable or disable tracing

When you are using MPL Attachments for message logging purposes, please bear the following in mind:

• Use MPL Attachments sparingly


• When writing message payloads to MPL Attachments, consider extracting the critical parts of the message payload and writing an excerpt only. Often the size of message payloads ranges in the tens or even hundreds of MB, and it is anyway very challenging for human users to inspect such huge amounts of data.


• Ensure that message logging via MPL Attachments is disabled when not absolutely required. For example, when the Integration Flow is moved from a test environment to production.

How exactly MPL Attachments can be enabled and disabled varies between Integration Flows. Some pre-packaged Integration Flows support message logging based on an external parameter, which is usually called ENABLE_PAYLOAD_LOGGING or ENABLE_LOGGING. In this case you can enable and disable message logging without modification of the Integration Flow.

For your own Integration Flows you might check if a central script is used, which supports configuring message logging via dedicated variables. In that case please ensure that these variables are adjusted to only log the required details, or to disable message logging if it is not required.

In case no central script is used for message logging, you might inspect the scripts in your Integration Flow and look out for the following API call:

messageLog.addAttachmentAsString("a huge one", body, "application/json"); // Example

If you do not require message logging, please comment this line by prefixing it with a double slash "//".

With this blog we wish to make you aware that the data volume created by MPL Attachments in the tenant storage can grow very quickly, when MPL Attachments are used for message logging. This can eventually overload storage resources, and even result in outages in the worst case. Please consider the recommendations in this blog to ensure economical usage of MPL Attachments and storage resources.