cancel
Showing results for 
Search instead for 
Did you mean: 

process multiple files

former_member203645
Active Participant
0 Kudos

Hi all,

I am trying to implement the SAP Note below,

Question :  In the Step 3 , it was suggested to add a condition in while loop $counter < $list_size.But where was the $counter value defined ??


After adding the below logic, The job is not running into the loop. Please advice.

Note 1825911:How to use wait_for_file function in a data services job to access bunch of files in a folder and process it one file at a time?

Resolution

1. Create a batch job in data services designer and declare the following global variables:

$file_list - varchar(1000), $list_size - int, $counter - int, $file_name - varchar(100), $file_list_temp - varchar(1000)

$file_list and $file_list_temp hold comma separated list of absolute paths to the files so the size should be big enough to hold the list. $file_name holds an absolute path to single file.

2. Create a startup script as follows:

wait_for_file('C:\Program Files (x86)\SAP BusinessObjects\Data Services\dsdata\*.txt', 1, 1, -1,$file_list, $list_size,',');
print('Number of file found = ' || $list_size);
print('Comma separated file list = ' || $file_list);
$file_list_temp = $file_list;

The script shows an example of where to look for the files. You can change it according to your system. In this case it looks for all txt files in the folder: C:\Program Files (x86)\SAP BusinessObjects\Data Services\dsdata. If the script runs properly, it will print number of files found and the comma separated list of files with their absolute path. You can tweak polling inteval and other options as desired.

3. Add a while loop after the script in the batch job. the while loop condition is: while $counter < $list_size

4. Add a script in the while loop as follows. It extracts a single file each time from $file_list using simple string functions. It prints the current file in process and remaining file list at the end.

$counter = $counter + 1;
print('While Loop Counter = ' || $counter );
#print(' comma index = ' || index($file_list , ',', 1));

if($counter = $list_size)
$file_name= $file_list_temp;
else
$file_name = substr($file_list_temp , 1, index($file_list, ',',1)-1);
$file_list_temp = substr($file_list_temp, index($file_list,',',1)+1, length($file_list_temp)-index($file_list,',',1));

print('Current file in process is ' || $file_name);
print(' Remaining file list is = '|| $file_list_temp);

5. Now you can add a data flow, use the current file for processing. You can also use $file_name to crate a flat file format to be used in data flow.

Accepted Solutions (0)

Answers (3)

Answers (3)

Former Member
0 Kudos

Probably not an issue for you now but I was using the same script. The issue is that you need to set the $counter variable to 0 before you step into the while loop

akhileshkiran
Contributor
0 Kudos

Hi

Hope the below link will help you out.

Regards,

Akhilesh Kiran.

Former Member
0 Kudos

See this solution to my problem, also with that:

former_member203645
Active Participant
0 Kudos

I got bit confused with the solution, Please explain the solution again

I have to process 3 + files.

Do we need to the add script ( highlighted) with in the while loop.

$file_list = exec('cmd','dir/b C:\Program Files (x86)\SAP BusinessObjects\Data Services\dsdata\MYNAME.*.txt');

$ctr = 1;

$file = word($file_list, $ctr);

while $file is not null

     ***process the file***

     $ctr = $ctr + 1;

     $file = word($file_list, $ctr);

former_member203645
Active Participant
0 Kudos

Can someone explain me the logic on the step 4 ( in step 3 it is mentioned as $counter < $list_size)  in the SAP Note : 1825911

I tried implementing this SAP Note and seems it is not entering into While loop.

I am bit confused with this logic

$counter = $counter + 1;

print('While Loop Counter = ' || $counter );

#print(' comma index = ' || index($file_list , ',', 1));

4. Add a script in the while loop as follows. It extracts a single file each time from $file_list using simple string functions. It prints the current file in process and remaining file list at the end.

$counter = $counter + 1;
print('While Loop Counter = ' || $counter );
#print(' comma index = ' || index($file_list , ',', 1));

if($counter = $list_size)
$file_name= $file_list_temp;
else
$file_name = substr($file_list_temp , 1, index($file_list, ',',1)-1);
$file_list_temp = substr($file_list_temp, index($file_list,',',1)+1, length($file_list_temp)-index($file_list,',',1));

print('Current file in process is ' || $file_name);
print(' Remaining file list is = '|| $file_list_temp);