CPI: API has restriction to pull only 100 records in each call

[naveenkk]

Hi Experts,

There is a source system and source system is exposing an API and CPI needs to get the data from source system and send it to target system(some third party). But they are some restrictions defined in the source system API level to pick the data.
1st restriction is it can give only 100 records in a call not more than that.
2nd restriction is sender API itself can be called only once per minute.
but i have close to 10k records from source application which needs to transfered to target application.

Please explain how to achieve this in CPI level? it would great helpful if you answer.

[bdBais]

So it's only 2 hours of run, questions are:

1) this API give you possibility to ask $top and $skip (paging) so you can work with sequentials runs?

2) How much time to finish your processing with 100 records?

3) In this "absurd" case 🙂I could create a little flow that take 100 and finish, local variable with $skip.

4) Do you know how many records are recorded when you start to pick them?

5) These records are contínuos changing/updating/growing?

example steps:

After your answers this flow could be different.

[martinfriebe]

Maybe you could also validate if you really need all the 10k data, maybe you could redesign your integration scenario to only updated specific ID`s?

[kiran0009 · Accepted solution]

@naveenkk 

Please use a looping process to call the respective service, retrieve the data, and append it to a variable using the content modifier until the loop exits. Once the loop is complete, return the appended data into the main flow body using the content modifier.

[naveenkk · Accepted solution]

Hi Kiran,

Thanks for your response.
I have used the looping process call with condition ${property.Receiver.OData.hasMoreRecords} = 'true'  and to append the data i have used expression ${property,propertyname}${in.body} in content modifier exchange properties.

Thanks everyone for your response now i have implemented the iflow and its working fine.

[naveenkk · Accepted solution]

Hi Kiran,