Hot folder processing queue

[Former Member]

Hi experts

In our project its very important to serialize data import and load data only in appropriate sequence. Of cause we must correctly assign csv filenames and we are ready to provide unique sequenceIDs for these filenames.

It is expected that sometimes csv files will be mass generated, e.g. 50-100 files per second.

And there are some doubts about... How HotFolder tool will process these files: in serial mode (based on sequenceID in filename) or in parallel multithreading mode?

[former_member483526]

By default, files are sorted in Alphabetic order but it is not always guaranteed. In this case you may think of writing your own custom directory scanner by extending DefaultDirectoryScanner and overriding listEligibleFiles methods.

You can configure your custom directory scanner as below-

 <int-file:inbound-channel-adapter id="filesIn" directory="file:${input.directory}"
          scanner="customDirectoryScanner"/>
 

[Former Member]

Hmm. In our case each of these files contain data about current stock level of small piece of product catalog. The first file will contain stock level data for product "A" at 10:00 AM The second file - for product "B" at 10:01 AM The fird - for products "A" and "B" at 10:02 AM and so on.

And we need that all of these files will load in appropriate order based on some sequence. How can we implement it with HotFolder? Is it possible?

[former_member638343]

Picking up on an old thread. Were you able to get this sequencing implemented?

[Former Member]

The sequence id is used to ascertain whether a record for a price, sku etc is a newer version or not. If they arrive out of sequence, then early version will just get discarded during the impex part.