How to improve performance of SAP Commerce Cloud populators

[GuillaumeLasnier]

Abstract

SAP Commerce Cloud Populators and Converters can have a critical impact on your Storefront performance. Having worked as an SAP Commerce consultant for over 10 years, I have frequently encountered this performance issue. This is why correct usage and implementation of Populators and Converters documented in SAP Commerce Cloud Development Best Practises and more advanced techniques such as Configurable Populators and Data Options should be followed.

We have also identified concerns with Composable Storefront with the additional Web Services DTO Data Layer for OCC which need to be taken into both at the Composable Storefront and SAP Commerce sides.

This article aims at describing common pitfalls and how they have been addressed in SAP Commerce Cloud releases.

Problem Statement

Populators divide the conversion process of filling out a Data Object into a series of population tasks or steps. The number and nature of population steps depends heavily on the context or purpose where those Data Object are used and it must be ensured that only the necessary data is populated as you may otherwise end-up with several issues :

1. Overload the database with unnecessary queries
2. Consume CPU and heap to populate unnecessary data
3. Consume bandwidth when transferring data over the network

As mentioned earlier there are 2 layers of Data conversion involved when targeting an OCC endpoint :

1. Facade conversion which is responsible to convert Models to Data objects. This layer controls which data needs to be queried from the database and populate into Data objects. This layer addresses issues 1) and 2) by means of Configurable Populators and Data Options to specify which data must be populated. Populators are implemented by Java classes. 
2. OCC Controller conversion which is responsible to convert Data object and Web Services DTO (aka WsDTO). This layer addresses issues 2) and 3) by specifying which fields to be populated, without a nesting limit, and its default implementation uses Orika - a popular Java Bean mapper framework. It does not require any Java code in most cases. For more details on how fields are defined, refer to Fields Configuration.

Unfortunately these 2 layers work independently and the fields parameter defined in OCC Controller request is only used for OCC mapping, not for Facade mapping.

While solving this conversion issue holistically, without breaking backward-compatibility could be very challenging, SAP Commerce Cloud has been improved in recent releases to provide specific solutions for well-known conversion performance problems.

Depending on context, various approaches have been taken to mitigate those issues and are described as examples in the next sections.

SAP Commerce Cloud Populator Improvements

Basic Populator for BaseSite

One of the first OCC calls in SAP Commerce Cloud composable storefront is to query the BaseSite endpoint to retrieve CMSSites from the backend. By default this call would also populate the CMS Site's BaseStores, PointOfService, OpeningSchedule and OpeningDays as shown in the below class diagram:

This single request could take more than 10 seconds for site having thousands of PointOfService and OpeningDays.

SAP Commerce Cloud 2211.12 introduced an improvement to populate BaseStores only when the SKIP_BASESTORE_POINTSOFSERVICE session attribute is specified and true, as shown in below code sample:

public class BaseStorePopulator implements Populator<BaseStoreModel, BaseStoreData>
{
    public static final String SKIP_BASESTORE_POINTSOFSERVICE = "SKIP_" + BaseStoreModel._TYPECODE + "_"
            + BaseStoreModel.POINTSOFSERVICE;
    [...]
    public void populate(final BaseStoreModel source, final BaseStoreData target) throws ConversionException
    {
        [...]
        final boolean skipPoS = getSessionService() != null
                    && BooleanUtils.toBooleanDefaultIfNull(getSessionService().getAttribute(SKIP_BASESTORE_POINTSOFSERVICE), true);
        if (skipPoS) {
            return;
        }
        target.setPointsOfService(new ArrayList<>());
        source.getPointsOfService().forEach(pos -> target.getPointsOfService().add(getPointOfServiceConverter().convert(pos)));
     
    }
}

As such, PointOfServices will not be populated by default.

To calculate the SKIP_BASESTORE_POINTSOFSERVICE accordingly, the BaseStoresController class simulates the conversion of a dummy BaseStoreData with an single PointOfServiceData into BaseStoreWsDTO using the fields parameter and check if pointOfServices have been effectively populated as illustrated in the code sample below :

public class BaseStoresController extends BaseController
{
    [...]
    public BaseStoreWsDTO getBaseStore(
            @Parameter(description = "Base store identifier.", required = true) @PathVariable final String baseStoreUid,
            @ApiFieldsParam @RequestParam(defaultValue = DEFAULT_FIELD_SET) final String fields)
    {
        // construct temporary BaseStoreData within PointOfService
        // call getDataMapper().map() with "fields" to check whether to skip loading PointOfService
        // add SkipPoS attribute to SessionContext, so as to retrieve in BaseStorePopulator and skip loading PointOfService for performance improvement.
        final BaseStoreData tempBaseStoreData = new BaseStoreData();
        final List<PointOfServiceData> pointOfService = Lists.newArrayList(new PointOfServiceData());
        tempBaseStoreData.setPointsOfService(pointOfService);
 
        final BaseStoreWsDTO baseStoreWsDTO = getDataMapper().map(tempBaseStoreData, BaseStoreWsDTO.class, fields);
        final boolean skipPointOfService = CollectionUtils.isEmpty(baseStoreWsDTO.getPointsOfService());
        sessionService.setAttribute(BaseStorePopulator.SKIP_BASESTORE_POINTSOFSERVICE, skipPointOfService);
 
        final BaseStoreData baseStoreData = baseStoreFacade.getBaseStoreByUid(baseStoreUid);
 
        return getDataMapper().map(baseStoreData, BaseStoreWsDTO.class, fields);
    }
}

Configurable Populators in ProductsController

A similar technique is used for the ProductsController class to remove unnecessary Populator Data Options used by the Configurable Populators.

public class ProductsController extends BaseController
{
    [...]
    public ProductWsDTO getProduct(
            @Parameter(description = "Product identifier.", required = true) @PathVariable final String productCode,
            @ApiFieldsParam @RequestParam(defaultValue = DEFAULT_FIELD_SET) final String fields)
    {
        Collection<ProductOption> options = extractProductOptions(fields);
         
        final ProductData product = productFacade.getProductForCodeAndOptions(productCode, options);
        return getDataMapper().map(product, ProductWsDTO.class, fields);        
    }
 
    protected Collection<ProductOption> extractProductOptions(final String fields)
    {
        final ProductData tempProductData = new ProductData();
        tempProductData.setImages(Lists.newArrayList(new ImageData()));
        // populate all configurable attributes
        [...]
 
        final ProductWsDTO productWsDTO = getDataMapper().map(tempProductData, ProductWsDTO.class, fields);
        final boolean skipImages = CollectionUtils.isEmpty(productWsDTO.getImages());
        final EnumSet<ProductOption> options = EnumSet.allOf(ProductOption.class);
        if (skipImages)
        {
            options.remove(ProductOption.IMAGES);
            options.remove(ProductOption.GALLERY);
        }
        // check all configurable attributes are populates to remove unnecessary options
        (...]
 
        return options;
    }
}

This improvement was introduced in version 2211.20. Before, no matter which fields were provided, all Product populator option were included, leading to unnecessary database queries and populated Data and WsDTO fields.

The table below describes which Product populators option are used depending on the product fields included :

Field Names Included product options :

 Included 

Field Name	Product options
images	IMAGES
images	GALLERY
reviews	REVIEW
numberOfReviews	REVIEW
potentialPromotions	PROMOTIONS
price	PRICE
purchasable	PRICE
priceRange	PRICE_RANGE
stock	STOCK

How to extend Product WsDTO and Data with additional Fields

Let's take the following example to see how to efficiently manage additional fields using configurable populators. In your last workshop with the marketing team, you discussed the opportunity to display the list all product sheets in the storefront's Product Detail Page.  The team informed you there could be up to 50 product sheets for a single product. 

You did your research and found that the data_sheet attribute defined for the Product item type could be used to store product sheets documents in various formats. Unfortunately this attributes is not defined in the Facade nor OCC layer.

As an experienced SAP Commerce developer, you know how to define these attributes in Data and WsDTO and populate them using configurable populators.

Now how do you ensure that product sheets will only be populated when required, i.e. when the custom CMS Component is rendered in the storefront ? For this you can apply the same technique implemented in the OOTB ProductsController  to simulate conversion of a dummy ProductData with an single entry data sheet list at the OCC level and test if the empty product sheets was effectively populated as illustrated in the code sample below:

public class CustomProductsController extends BaseController
{
    public ProductWsDTO getProduct(
            @Parameter(description = "Product identifier.", required = true) @PathVariable final String productCode,
            @ApiFieldsParam @RequestParam(defaultValue = DEFAULT_FIELD_SET) final String fields)
    {
        Collection<ProductOption> options = extractProductOptions(fields);
         
        final ProductData product = productFacade.getProductForCodeAndOptions(productCode, options);
        return getDataMapper().map(product, ProductWsDTO.class, fields);
    }
 
    protected Collection<ProductOption> extractProductOptions(final String fields)
    {
        final ProductData tempProductData = new ProductData();
        // copy logic from OOTB ProductsController to populate dummy ProductData
        [...]
        // Populate data_sheet with single dummy entry
        tempProductData.setData_sheet(Lists.newArrayList(new ImageData()));
         
 
        final ProductWsDTO productWsDTO = getDataMapper().map(tempProductData, ProductWsDTO.class, fields);
        // copy logic from OOTB ProductsController to check attributes are populated
        [...]
        // check data_sheet populated
        final boolean skipData_sheet = CollectionUtils.isEmpty(productWsDTO.getData_sheet());
        if (skipData_sheet)
        {
            options.remove(ProductOption.DATA_SHEET);
        }
 
        return options;
    }
}

SAP Commerce Cloud, composable storefront

Configuring OCC endpoints

When implementing functionality that requires OCC interaction, make sure the required endpoints are configured using only necessary fields. For our last example we have defined an additional datasheet scope for the product endpoint which include the data_sheet field :

@NgModule({
  providers: [
    provideConfig(<OccConfig>{
      backend: {
        occ: {
          endpoints: {
            product: {
              datasheet:
                'products/${productCode}?fields=DEFAULT,averageRating,images(FULL),classifications,manufacturer,numberOfReviews,categories(FULL),baseOptions,baseProduct,variantOptions,variantType,data_sheet',
            }
        },
      },
    }),
  ],
})
export class ProductDatasheetModule {}

Product Carrousel Improvement

SAP Commerce composable storefront Product Carousel component has also been updated to only populate required fields depending on the context :

• In the default case, the number of fields has been reduced to code,name,price(formattedValue),images(DEFAULT),baseProduct
• When Configurable Slide Action , such as ProductAddToCartComponent, additional fields, such as stock, are added

Conclusion

In conclusion, the effective use of SAP Commerce Cloud Populators and Converters is crucial for optimizing storefront performance. By adhering to best practices and advanced techniques such as Configurable Populators and Data Options, businesses can avoid common pitfalls that lead to performance issues. The introduction of improvements in recent SAP Commerce Cloud releases, such as the Basic Populator for BaseSite and Configurable Populators in ProductsController, demonstrates the ongoing efforts to address these challenges.

Understanding the dual-layer conversion process involving Facade and OCC Controller conversions is essential for ensuring that only necessary data is populated, thereby reducing database overload, CPU and heap consumption, and bandwidth usage. The examples provided in this article illustrate how specific solutions have been implemented to mitigate well-known conversion performance problems.

As SAP Commerce Cloud continues to evolve, it is important for developers and consultants to stay informed about the latest enhancements and apply them effectively. By doing so, they can ensure that their storefronts remain efficient, responsive, and capable of delivering a superior user experience.