Import data into model on event

[#1]

Hello everyone,

I would like to ask you, if is there a some way to import data (from CSV file server) into model on event. Not in exact time. Maybe some way throught REST API?

Thank you a lot,

Ondrej

[DebjitSingha · Accepted solution]

Hello horynao,

We had similar required where we wanted to utilize event based triggering based on when our 3rd party vendor team copy a file in shared network drive. Unfortunately there is not available API that can help triggering SAC job.

Instead we end up scheduling jobs every 15 mins. Setup four jobs with 15 mins gap and frequency every 1 hour. This setup let us check and process (when available) every 15 mins. Our source CSV file have a created on date columns. We use the same for incremental load. So when the filed is already processed, entire file is skipped for next time onwards.
Now when vendor creates a new file under shared folder with Created On date, SAC pickup with a worst case latency of 15mins.

Note: Setting dependency with skip query option can come handy if you want to map all 4 jobs together.

Hope this helps. Let us know if you have any query around this topic.

Viewers/ readers - If you find above information helpful, feel free to up-vote (arrows on left side).

- Debjit

[horynao]

Thanks a lot Debjit, I think I understand what you mean. But could you more describe this part?

Our source CSV file have a created on date columns. We use the same for incremental load. So when the filed is already processed, entire file is skipped for next time onwards.

Thank you

Ondrej

[DebjitSingha]

Hello horynao,

We instructed our vendor to update "Created On" field with file creation date. This is a manual step vendor perform once they have final file ready.
This field - "Created On" date is considered as "Incremental Date" field in our SAC Query. Acting as delta relevant field. While extracting CSV data, SAC checks content under Created On column. If the date exceeds previous load, then system consider entries as new if the date is same then they gets ignored.

This step ensure, we do not process same file multiple time and saved resources on SAC side. SAC just checks CSV file and in nothing new found then skips the entire file and confirms no data available message.

Thanks,

Debjit

[horynao]

Ok and thank you very much.

Best regards

Ondrej