SuccessFactors Learning Data Load Strategy - Full or Delta Load?

[karenperez]

INTRODUCTION

When implementing SuccessFactors Learning is very common to receive some questions concerning the Data Load Strategy from customers, such as: What is the recommended way to schedule the User, Org, Job data load, always as a full or incremental (delta) load? Because this is a recurring topic, this blog tries to give some guidance based on the product recommendation concerning the connector background jobs, our primary method for data loads. The alternative options besides connector background jobs, such as API, will not be mentioned in this blog as it is not recommended/approved for such use.

OVERVIEW

Before we start to talk about what should be the recommended data load strategy for SuccessFactors LMS data, let’s first understand some concepts.

Connector Background Jobs

SuccessFactors Learning Connectors allow Administrators to automate importing data into the SF Learning application. Data can be provided directly by the customer, come from an integrated source e.g. SuccessFactors HXM Suite, or a 3rd party provider e.g. Skillsoft connector. Structured  TXT and CSV input files, uploaded to a dedicated SFTP location, are used as data vehicle.  The SuccessFactors Learning application processes the input files based on the connector configuration and updates the Database (Please refer to the Learning Connector Workbook for configuration and technical information).

Connector Detailed Processing Overview

Full Load

Is a process of an entire data dump that takes place the first time a data source is loaded into the destination.

Incremental (Delta) Load

Incremental load is a process of loading data incrementally. Only new and changed data is loaded to the destination. Data that didn’t change will be left untouched.

Now that we have the concepts clearly, the next chapter will show us when should we use each (full or delta) and what are the pros/cons of it.

RECOMMENDATION

Below we describe in a table structure, the type of the data load strategy and when to use each:

Data Load Strategy	Type	Description	When to use this strategy
Full Load	System Data Load	A full initial load is a simple load of data into an empty data instance.	Use this strategy the first time you load data into a new instance.
Incremental (Delta) Update	Incremental Data Update	An incremental update adds new records to a data instance, and update existing records.	Use this strategy when you want to add new records to the instance, or to update existing records.

It gets clear that the full load is recommended in an initial load phase and the incremental should be used after, but what is behind this recommendation?

First of all, ‘Time’ is the primary reason.

Opting to do a full load normally takes a great amount of time and because of this, it is recommended to run overnight with the expectation of completing it before users start their working day. The overnight window may not be enough time for the full load to complete (depending on the customer’s data size).

Secondly, scheduling full load in a recurring way can generate a non-desired workload to the datacenter/ background job, especially considering that you’re most likely reloading some data that hasn’t changed. So let’s try to only handle the data we need to – the new and updated rows.

LMS Recommended Schedule:

• Run an initial full load;


• Schedule a daily Delta feed that imports only the changed users from the last 24 hours.

Most customers run the daily Delta at an overnight time for their environments, when the amount of users on the system is low.

It is recommended to use Delta feeds as they reduce the load on the connector background job.

CONCLUSION

This blog post described what are the data load strategies and when to apply them. Also, it highlights the pros/cons of each and proposes a recommended connector schedule.

Additionally, it is very recommended to read the SAP SuccessFactors Learning: Impact of implementing SAP SuccessFactors Employee Central IDP (Implementation Design Principle) to get more information on how to set up the Integration Center Jobs, including schedule and filters.

Looking forward to your comments, additions, and questions.