Question Re: Cloud Integration: How to access nextLink using message flow in Technology Q&A

Cloud Integration: How to access nextLink using message flow

THEN — Thu, 08 Jun 2023 14:27:00 GMT

Hello all,

I've developed an integration flow which fetches the data from the CRM system and writes it to a query log. However, the source system only provides 5000 records and a nextLink that points to the next x-thousand records. Unfortunately, I have no idea how to access the nextLink from the XML to fetch the next x-thousand records until there are none left. Can anyone tell me how to proceed here?

Cloud integration - message flow:

ODATA.NextLink

Re: Cloud Integration: How to access nextLink using message flow

katherresh_shanmugasundar — Thu, 08 Jun 2023 22:04:02 GMT

Hi Torben,

This kind of paginated extract is quite common now a days. One way of resolving this would be

Have 2 Main integration processes
First process should execute the initial query, fetch first 5000 records, process and post to the target. In addition this process should extract the next link URL & update it to a Data Store
The second process should have a Data Store sender adapter which would extract data using this next link & fetch next 5000 records, process and post to the target. And also extract the next link URL & update it to the Data Store
Step 3 will be recursive and it will write to the data store until next link has a value
The common steps should be in a local integration process which should be called in both Main processes

And the length of next link should be less than 255

Best Regards,

Katherresh

Re: Cloud Integration: How to access nextLink using message flow

THEN — Fri, 09 Jun 2023 08:32:54 GMT

Dear Katherresh,

thanks for the information. This helped me a lot - I was able to write a script which passes identifies the nextLink. However, if I try to copy that Link into the ODATA query, I can't execute the process (I think the issue ist the $skiptoken) anymore. My code looks like this:

Everything before the "$skiptoken" is the same and is executable. Am I doing something wrong here or does the cloud integration service doesn't support the $skiptoken-parameter?

Re: Cloud Integration: How to access nextLink using message flow

katherresh_shanmugasundar — Thu, 15 Jun 2023 08:05:42 GMT

Hi Torben,

Hopefully you were able to resolve this already. Skip Token is basically a pointer for the next set of records to be selected & it is supported by SAP Cloud Integration as it is just a parameter.

If the issue still persists then it might be because of session/ cookie handling and re use.

Best Regards,

Katherresh