Sleep, wait for file task into SAP DataSphere ?

[#1]

Hello experts,

is it possible to add a sleep task into DWC?

I need to "wait for a file" before start to read it.

Is it there something similar to BODS "Wait_For_File()" function in SAP Datasphere ?

Regards,

Alberto

[XaviPolo · Accepted solution]

Hello,

No, there is nothing similar.

I don't think you can implement that logic with what is actually available in DSP, because although in a Data Flow you have python and there you could "wait", the transformation ONLY read the input data once, and not multiple times which would be necessary to be able to do the check while waiting.

Probably until a Data Flow can be launched externally, or they add that function to the SFTP connector, the only option I think is to schedule the DF to run regularly.

Regards,

[albertosimeoni · Accepted solution]

Hello Xavier,

Probably a task chain with a dummy step:

step 1) dataflow with wait for file.

step 2) dataflow with read from csv/xlsx/txt.

Step 1) DUMMY (table) -> WAIT (Script) -> DUMMY (target).

Other problems are:

- can you set a "wait"? ( is it available as function in the script block ?)

- can you read the file from a connection? (or better can you check if the file is available).

[XaviPolo · Accepted solution]

the problem is that you can "wait" in python (script) but nothing will change in the input.

In the DF:

1. Source (File from SFTP Connection or hyperscaler)
2. Pass the content of the file (0 or more rows) to the Python transform
3. Inside Python you can use sleep() or something like this to "wait", but data from source is already sent to python, and the file will no longer be read in that execution. So is not a "wait 1 second and check the file" loop, is just "wait 1 second to do nothing"
4. Data from python is sent to output.

You get the data present in the file at the time of execution.

It may even happen (depending on the software used and the configuration of the system) that if the file is very large, you read it when it has not yet been completely uploaded to SFTP.

An option is to use an external job scheduller solution, which sends the file to the SFTP with a temporary name FILE1.XLS.TEMP, checks that the file size is the same (so you know if it has been fully loaded into the SFTP) and then renames it to the original name FILE1.XLS.

Then the tool calls DSP and executes the DF. But this last step is not yet available (in the DSP roadmap it is planned for Q2-2023).

In other words, doing a process as was done in Data Services, where you leave a JOB running and wait for a file to be present to load it, cannot be done.

[albertosimeoni · Accepted solution]

Sorry we misunderstood each other,

I mean a 2 step procedure,

1) a dataflow that read from a "Dummy" table (1 row 1 column DUMMY), then in python it uses some function to verify the file exists in a connection with a wait in a loop. The purpose of this dataflow is only acting as semaphore for another dataflow that will read the file.

2) a Second dataflow that uses the file as source and do what it needs to do.

Your reply was helpful as I didn't know there will be the possibility to trigger task chains from external tools in the near future.

Best regards,

Alberto