SFTP duplicate file checking

[#1]

Dear Experts,

         I have done the development  SFTP->PI->File(FTP) and its working fine, But I don't want to process the duplicate file.

Issue in Detail:

   Every day our client will placing the file in SFTP, one file per day and no need to process the duplicate file. So that iam checking the duplicate file in channel configuration. So everyday we need to pick the latest file from SFTP using PI.

  PI is picking the latest file and updating in FTP and after 2hrs PI is again picking some of the random duplicate file from SFTP folder and updating in FTP and I increased the modification check in SFTP channel configuration, but the problem is still persists.

Kindly give some solution to resolve this issue if any body face this.

Best Regards,

Monikandan.

[Former Member]

Hi Monikandan,

Based on your requirement , the client is placing only one file a day and you have to process that .

The best way to handle this and avoid duplicate processing will be to create a schedule for your sender channel and make it run for 10 - 15 min once a day after the file got created in the folder. For example if the file is placed at 12:15am every day , schedule the channel from 12:20am - 12:30 am . In PI the file processing time will be fast.

Also make sure the polling interval in the channel is more than the scheduled run time so that it doesn't pick the same file again.

Hope this helps you to resolve the issue.

- Muru

[monikandan_p]

Dear Murugavel,

    Thanks for your reply

        I have already scheduled the channels from 2pm to 3pm IST,But once PI picked the latest file from SFTP it again picking some random duplicate file within the 1hour schedule.

Previously i scheduled it for 30mins what happen in the case,here we have some 100 duplicate files exist in SFTP folder,So PI need to confirm each and every file for duplicate,For this PI is taking more than 30mins to validate the duplicate file check.

Kindly suggest some solution.

Best Regards,

Monikandan

[dipen_pandya]

Hi,

Better to use archive method as Azhar said.

Try to convince your client.

Regards,

Dipen.

[Former Member]

Hi Monikandan,

            As your schedule is between 2 - 3 PM , setthe polling interval in the channel to 7200 sec which is 2hrs.

            It will avoid channel running again within the scheduled time frame , also please let us know

          Are you picking all the file from the source folder or do you have any naming convention defined?

          The files placed by your client daily , are they overwriting the existing one or creating a new one?

Regards,

Muru