SAP Hybris Commerce to SAP Hybris Marketing Integration via the Expressway

[former_member280241]

Disclaimer: Expressway is a non-standard integration solution for sending master and transactional data from SAP Hybris Commerce to SAP Hybris Marketing (Cloud and on premise). The solution offers a point-to-point alternative to Data Hub, which is the middleware option that ships with SAP Hybris Commerce. In addition to supporting the same master and transactional data as Data Hub, Expressway can be useful for specific customer situations. For example, Expressway’s simple design can be leveraged when extensive field enhancements are required, or when expert knowledge of Data Hub is missing. Proficiency in SAP Hybris Commerce platform technology is the only prerequisite for understanding how to deploy, run, and enhance Expressway. As it is not officially supported, Expressway should only be considered as a template for building custom integration projects. Please note that Expressway can be co-deployed with the SAP Hybris Commerce Data Hub integration.
 

Expressway is a very lightweight integration between SAP Hybris Commerce and SAP Hybris Marketing. To deploy Expressway on your SAP Hybris Commerce server, all you have to do is add one extension to your localextensions.xml file, rebuild, and update your system. Once Expressway is deployed, you can immediately start sending master data and other marketing-relevant information to SAP Hybris Marketing. With Expressway, you can send customers, products, categories, reviews, abandoned carts, saved carts, and orders, all out-of-the-box. Moreover, Expressway can be deployed on older versions of SAP Hybris Commerce as far back as version 5.7.

Expressway uses the SAP Hybris Commerce job framework to initiate the replication of data to SAP Hybris Marketing. Each object type (e.g. product, customer, order, etc.) gets its own job, and you can run and schedule this job from Backoffice Administration Cockpit. Expressway was designed with maximum reuse in mind, so that any enhancements you need to make, whether to add new fields to an existing object type or add a completely new object type, are straightforward and involve minimal changes. A major feature of Expressway is its end-to-end monitoring capability. Not only does Expressway provide detailed monitoring of events and errors that may occur in the SAP Hybris Commerce system; Expressway can also be configured to monitor errors that occur in the SAP Hybris Marketing system.

 

High-Level Overview (click image to enlarge)

 

How to Install

1. Download the appropriate version of Expressway here.


2. Extract sapymktsync to <your_hybris_folder>\bin\ext-integration\ymkt.


3. Add the following SAP Hybris Marketing server settings to your local.propertiles file:
   
   
   
   • sapymktcommon.odata.url.root=https://<server_name>:<port>
   
   
   • sapymktcommon.odata.url.sap-client=<server_client_number>
   
   
   • sapymktcommon.odata.url.user=<username>
   
   
   • sapymktcommon.odata.url.password=<password>
   
   
   
   


4. Add <extension name='sapymktsync' /> to your localextensions.xml file.
   
   
   
   • If not already present, add <extension name='platformbackoffice' /> to be able to access Backoffice.
   
   
   
   


5. Run the following to rebuild and update your system: ant clean all updatesystem

 

How to Use

To use Expressway, navigate to the Cronjobs menu in Backoffice. There you will find all the cronjobs delivered with Expressway (click to enlarge):



 

Next, we configure a cronjob. Select a cronjob and navigate to the Administration tab. There you will find four configurable properties for use with Expressway (click to enlarge):



The properties are described below:

• Batch Size: objects are sent to SAP Hybris Marketing in batches, one batch per OData request. This property defines the maximum number of objects in a batch. The default is 500.


• Catalog Version: specifies the catalog version to be used when selecting certain objects, like products or categories. For example, the ymktProductCronJob only sends products belonging to the catalog version specified in this property.


• Error Stop Count: objects are sent to SAP Hybris Marketing in batches. The attempt to send a particular batch may fail. The 'Error Stop Count' property defines the maximum number of failures before the job is aborted. Note that the Expressway cronjobs can also be aborted on demand.


• Perform Validation: part of the end-to-end monitoring capability of Expressway. When enabled, this property issues a call to SAP Hybris Marketing after a batch is sent to see if there were any errors importing that batch. The errors are attached to the cronjob log. Enabling this property impacts performance, so use judiciously.

Multi-Language Support

A cronjob can be configured for a particular language, and Expressway sends the data in that language. For example, if the language property of ymktProductCronJob is set to German, Expressway sends the product descriptions in German. To send the products in multiple languages, you can maintain language-specific copies of the ymktProductCronJob (e.g. ymktProductCronJob_EN, ymktProductCronJob_DE, etc.).


 

cronjob Logging

Messages logged by Expressway during the execution of a cronjob can be seen in the cronjob log. Open the log and download it:


Make sure to set the logging to the desired level using the 'Log level file' dropdown as show in the screenshot above.

Note that that logging level chosen in the dropdown does not set the logging level of sapymktsync. That has to be done separately in your local.properties file. The level chosen in the dropdown only determines which messages get written to the cronjob log file. So if the logging level of sapymktsync is set to 'INFO', it does not matter that the level in 'Log level file' is 'DEBUG'. No 'DEBUG' messages from sapymktsync are logged.

Note also that the logging level set in the 'Log level file' dropdown can be no higher than the value specified in the 'cronjob.logtofile.threshold' property. By default this property is set to 'INFO', so if you want to set 'Log level file' to 'DEBUG', as in the screenshot, you would need to set 'cronjob.logtofile.threshold' to 'DEBUG' also.

Expressway provides a lot of useful logging information. In particular, it tells you the status and execution time of the cronjob, the number and type of the changes that were detected along with the time taken to detect those changes, the number and type of the changes that were marked as 'processed' along with the time to process those changes, and the number of objects that were sent along with the time to send them:

17.05.17 18:48:13:393 INFO SUCCESS/FINISHED=2528ms, CollectChanges(NEW=1)=47ms, ConsumeChanges(NEW=1)=106ms, SendModels(1)=1525ms, ValidateImportHeader(1)=840ms

 

Abandoned Cart

Expressway defines a property that specifies the length of time in minutes after which a cart is considered abandoned. The property name is 'sapymktsync.syncjob.abandonedcart.interval' and the default value is 120 minutes. You can specify a custom value for this property in your local.properties file.

 

How to Enhance

Adding a Field to an Existing Object Type

To add a field to an object type, or to remove a field, or to change the way a field is mapped to its OData fieldname, find the sync service (e.g. CustomerSyncService) for the object type that you're interested in and make your changes there. For example, to add the customer postal code, which isn't currently mapped, you would locate the CustomerSyncService class and add the following to the convertAddressModelToMap(...) method:

//<Property Name="PostalCode" Type="Edm.String" MaxLength="10" sap:unicode="false" sap:label="Postal Code"/>

this.optionalPut(map, "PostalCode", address, AddressModel::getPostalcode);

Adding a New Object Type

To add a new object type, create a new sync job and a new sync service.

Adding a New Sync Job

For the new sync job, extend either AbstractSyncJob or AbstractChildSyncJob. The latter is used for object types that are associated with a child object type. For example, OrderSyncJob extends AbstractChildSyncJob because an order has items.

If you extend AbstractSyncJob, you may need to override some or all of the following methods:

• getModelClass() is where you provide the model class for the item type that you wish to send to SAP Hybris Marketing. This method is mandatory. For customers, the method is as follows:
  

  @Override
  
  protected Class<CustomerModel> getModelClass()
  
  {
  
      return CustomerModel.class;
  
  }

  
  

• getStreamConfigurationItemSelector(...) is where you provide search criteria for your object type. The value provided in this method is used as part of the 'WHERE' clause of the Flexible Search query that is constructed by deltadetection. Consider saved carts, which are simply carts whose 'saveTime' isn't null. To retrieve saved carts, SavedCartSyncJob provides the following search criteria:
  

  @Override
  
  protected Optional<String> getStreamConfigurationItemSelector(SyncCronJobModel cronJob)
  
  {
  
      return Optional.of("{item.saveTime} IS NOT NULL");
  
  }

  
  


• You may need to communicate object deletions to SAP Hybris Marketing for your new object type. Although deltadetection can detect that an object was deleted, the only information that remains about the object is its automatically-generated SAP Hybris Commerce-specific primary key (after all, a deleted object no longer exists in the database!). Typically, this primary key is not something your SAP Hybris Marketing system will know about. So how do you inform SAP Hybris Marketing that the object was deleted? Expressway provides two methods for this very purpose:
  
  
  
  • getItemStoreFunction() is where you provide the function that gets the information to store for your new object type. The stored information is used later to restore the object after it's been deleted (as a technical aside, the information is stored in the 'info' field of a deltadetection item type called 'ItemVersionMarker'; the 'ItemVersionMarker' is used to mark changes to items). SavedCartSyncJob implements getItemStoreFunction() for saved carts. All that is needed to communicate the deletion of a saved cart to SAP Hybris Marketing is the cart's ID. Hence, getItemStoreFunction() does the following:
    

    @Override
    
    protected Optional<Function<CartModel, String>> getItemStoreFunction()
    
    {
    
        return Optional.of(CartModel::getCode);
    
    }

    
    
  
  
  • getItemRestoreFunction() is where you provide the function that initializes the restored object. The restored object's fields are mapped to OData field names in the sync service method convertModelDeletedToMap(...). Please note that the restored object is only needed for the mapping to OData; it is not persisted in SAP Hybris Commerce. For saved carts, getItemRestoreFunction() does the following:
    

    @Override
    
    protected Optional<BiConsumer<CartModel, String>> getItemRestoreFunction()
    
    {
    
      return Optional.of(CartModel::setCode);
    
    }

    
    
  
  
  
  

If you extend AbstractChildSyncJob, you may need to override some or all of the following methods in addition to the methods described above.

• getChildClass() is where you provide the model class for the child item type. This method is mandatory. For orders, the method is as follows:
  

  @Override
  
  protected Class<OrderEntryModel> getChildClass()
  
  {
  
      return OrderEntryModel.class;
  
  }

  
  


• getParentFunction() is where you provide the function that returns the parent of the child item. This method is mandatory. For orders, the method is as follows:
  

  @Override
  
  protected Function<OrderEntryModel, OrderModel> getParentFunction()
  
  {
  
      return OrderEntryModel::getOrder;
  
  }

  
  

• getChildStreamConfigurationItemSelector(...) is where you provide the search criteria for the child. For saved carts, the method is as follows:
  

  @Override
  
  protected Optional<String> getChildStreamConfigurationItemSelector(SyncCronJobModel cronJob)
  
  {
  
      return Optional.of("{item.order} IN ({{ SELECT {c.pk} FROM {Cart AS c} WHERE {c.saveTime} IS NOT NULL }})");
  
  }

  
  

Adding a New Sync Service
The sync service is configured for a specific SAP Hybris Marketing OData service. This is specified in the bean definition of the sync service in sapymktsync-spring.xml:

<bean id="ymktCustomerSyncService" class="com.hybris.ymkt.sapymktsync.services.CustomerSyncService" parent="ymktAbstractSyncService">

    <property name="ODataService" ref="ODataService_CUAN_IMPORT_SRV" />

</bean>

...

<bean id="ymktOrderSyncService" class="com.hybris.ymkt.sapymktsync.services.OrderSyncService" parent="ymktAbstractSyncService">

    <property name="ODataService" ref="ODataService_CUAN_BUSINESS_DOCUMENT_IMP_SRV" />

</bean>

Notice that the customer sync service uses a different OData service than the order sync service.

If you're using OData service CUAN_BUSINESS_DOCUMENT_IMP_SRV, extend AbstractBusinessDocSyncService.

If you're using OData service CUAN_IMPORT_SRV, or if you're using an OData service whose root entity is like the 'ImportHeader' in CUAN_IMPORT_SRV, extend AbstractImportHeaderSyncService.

Otherwise, you can extend AbstractSyncService.

In all cases, the sync service is responsible for mapping the fields of SAP Hybris Commerce items (instances of ItemModel) to OData field names. This is done in method convertModelToMap(...). The sync service is also responsible for sending the maps returned by convertModelToMap(...) to the SAP Hybris Marketing system. This is done in method sendModels(...).

After you're done creating your new sync job and new sync service, you must declare the corresponding beans in sapymktsync-spring.xml. For example:

<!-- Category -->

<bean id="ymktCategorySyncService" class="com.hybris.ymkt.sapymktsync.services.CategorySyncService" parent="ymktAbstractSyncService">

    <property name="ODataService" ref="ODataService_CUAN_IMPORT_SRV" />

</bean>



<bean id="ymktCategorySyncJob" class="com.hybris.ymkt.sapymktsync.jobs.CategorySyncJob" parent="ymktAbstractSyncJob">

    <property name="syncService" ref="ymktCategorySyncService" />

</bean>

You must also specify the cronjob that runs your sync job in projectdataConfig.impex. For example:

INSERT_UPDATE SyncCronJob ; code[unique=true]        ; job(code)                 ; catalogVersion(catalog(id), version) ; sessionLanguage(isocode)

                          ; ymktCategoryCronJob      ; ymktCategorySyncJob       ; Default:Staged                       ; en

The ImpEx file will be imported when you update your SAP Hybris Commerce server.

 

And that's it! You're now ready to hop on the Expressway!

 

Product Owner: Emmanuel Piochon (emmanuel.piochon@sap.com)

Main Developer: Robin Vezina (robin.vezina@sap.com)

Developer: Christos Nikolakakis (christos.nikolakakis@sap.com)