Bulkloader Loading Performance

[mrstifler777]

Hi Experts,

We noticed that in the processing folders of the bulkloader (e.g. /workingdirectory/SAPCC_CIT/PREPAID/Instance#1/CC1) the processing seems to be slow:

1. The number of files are already increasing on this directory (reaching 800+) which normally only averages about 200 files.

2. When we re-process files from INVALID folders, it's not getting processed immediately. Takes hours for example where the file is still in .arc.csv. It gets processed after so we know that the original issue while it failed was already corrected.

Question: For these type of cases, can these be monitored for example in an application like introscope? And is the solution here restarting the bulkloader?

Thanks,

Steve

[francois_thimon]

Hi Steve,

One way to address this is to look out for the ".bxx" file, in the folder where the files are picked up for processing (SAPCC_CIT/PREPAID/Instance#1/CC1).
Each time a bulkloader chooses a file and starts loading its contents, that file is renamed, in order to indicate that it's already being taken care of (and which bulkloader is processing it).
For example, if your bulkloader is called "bulkLoader#3", it'll rename files with ".b03" (at the very end of the file name).
You should therefore find files named roughly as follows:
SID_CIClass_001_min_03_20230315T082537899_0000000025.arc.csv.b03

You may check how long the same file keeps that ".bxx" extension and, more importantly, you should generate thread dumps of the bulkloader while the file is locked.
This will tell you what the bulkloader's doing (and why this prevents it from completing its current task on that file).
The issue might be related with your JCo connections (the resources allowed in your JCo configuration might be insufficient).

Even before this, you may also want to check if the bulkloader has generated automatic thread dump.
This will tell you if CC has detected issues (such as timeouts or other errors).

Please let us know if you need further guidance.

Best regards.

François
SAP Convergent Charging Support

[mrstifler777]

Thanks François, how do we check the following?

1. generate thread dumps of the bulkloader?

2. available resource for Jco (resources allowed in your JCo configuration might be insufficient)

Thanks,