Bulkloader Loading Performance

[mrstifler777]

Hi Experts,

We noticed that in the processing folders of the bulkloader (e.g. /workingdirectory/SAPCC_CIT/PREPAID/Instance#1/CC1) the processing seems to be slow:

1. The number of files are already increasing on this directory (reaching 800+) which normally only averages about 200 files.

2. When we re-process files from INVALID folders, it's not getting processed immediately. Takes hours for example where the file is still in .arc.csv. It gets processed after so we know that the original issue while it failed was already corrected.

Question: For these type of cases, can these be monitored for example in an application like introscope? And is the solution here restarting the bulkloader?

Thanks,

Steve

[former_member428395]

Hello Steve,

unfortunately, we can only say that SAP CC is waiting for an answer from SAP CI. The pending request is a BIT loading. We don't have more details to share on SAP CC side.

The next step is to investigate on SAP CI side in order to understand why there is no response provided to SAP CC.

Best regards,

Eric F.

[former_member428395]

Hello Steve,

If the thread is waiting from SAP CI, it means that SAP CI didn't provide any answer and the slowness or the lock has to be investigated on SAP CI side.

Please check this topic with the support team of SAP CI.

Best regards,

Eric

[mrstifler777]

Thanks Eric,

Yes, we have been working with CI on this but there are several background processes and programs running and we are trying to figure out if there's a way in CC to identify what exactly it is waiting for? maybe a program or table lock in CI?

This is an example of the log, maybe there's any information here we can use to check in CI? (note omitted some info here)

[francois_thimon]

Hi Steve,

Please find below the requested details.

1. to find back the thread dumps, or generate new ones
By design, the CC instances automatically generate thread dumps when they run into timeouts or severe exceptions.
If that's your case, you'll find these files in the work/dump directory. Since you're trying to diagnose a past occurrence of an issue, we recommend you to search these files first. If they're present, they'll give you context information about what happened.
The name of each file is suffixed with the type of even that prompted its creation: automatic files are consequently suffixed with either "timeout" or "exception", this is how you can recognise them in the directory.
Also, please note that CC automatically deletes these files after 14 days by default (this can be changed by modifying the THREAD_DUMP_RETENTION_PERIOD parameter).

If you still need to create files manually, please log into admin+, and use the "dump" command, to generate thread dumps for the targeted instance(s).
For example:

# if you're interested only in bulkLoader#3
dump bulkLoader#3

# if you want a thread dump for each running bulkloader
dump bulkLoader

# with no argument, you'll generate a dump for each running instance
dump

Whatever you choose, the generated files will also be in the work/dump directory, but with the suffix "ondemand", this time.

Numerous settings are available for the tread dumps (in particular if you want CC to generate several dumps in a row when an error occurs, with a defined time interval, etc.).
Here's the corresponding documentation, from our Parameter reference:
https://help.sap.com/docs/Convergent_Charging/9fe6117842b24350b6eb2fe619418b44/15ac157efb1c42caa49c8...

2. to check the allocated JCo resources
The JCo settings are in your jco.destination file. When you set up CI manually, you have to import that file using setup.sh (the name is usually "jco.destination", but you may choose a different one).
If you still have that file around, you can reopen it, and see the settings there. Otherwise, you can export it using setup.sh as follows:

/usr/sap/SID/CCDxx/script/setup.sh jcodestination export /tmp/jco.destination.export CI_JCO_DEST_NAME -login=adminuser

This'll at least give the values for "peak_limit" and "pool_capacity" (defining how many JCo connections may be kept, and used at the same time). The parameters that aren't explicitly set in your file keep their default value, and you may need the assistance of a JCo expert to tune them.
Before even checking the settings, it's best to go to the work/ directory, and see if you have JCo- or RFC-related logs. They may contain detailed information about what happened (for example, if the issue was due to an insufficient number of available connections, you'll see it there). If you don't find any such files, please let us know, we can help you activate the JCo traces.
Here's already the reference of the JCo logging parameters:
https://help.sap.com/docs/Convergent_Charging/9fe6117842b24350b6eb2fe619418b44/e2a5e04837694f0094bc3...

Feel free to ask for more details about the procedures, or to open a customer ticket if that's preferable (in particular if you want us to analyse your files).

Best regards.

François
SAP Convergent Charging Support

[mrstifler777]

Thank you François for the detailed input,

From the work dump folder mentioned we actually saw some files there generated on the date when the issue was encountered. However, they are suffixed with "shutdown" and "periodical" only. Do we have any references we can check to better understand the suffixes here and the content of those files?

So far we have identified that threads were blocked on that one bulkloader server causing the slow performance. In the work dump file we can see some info like threads e.g. for PREPAID CIT READER in RUNNABLE state and some are in WAITING state. The waiting state seem to refer to locks? We are trying to look for info on what might be the cause for example if it's waiting, what is it waiting for and if in CI can we see trace the program?

Steve

[francois_thimon]

Hi Steve,

One way to address this is to look out for the ".bxx" file, in the folder where the files are picked up for processing (SAPCC_CIT/PREPAID/Instance#1/CC1).
Each time a bulkloader chooses a file and starts loading its contents, that file is renamed, in order to indicate that it's already being taken care of (and which bulkloader is processing it).
For example, if your bulkloader is called "bulkLoader#3", it'll rename files with ".b03" (at the very end of the file name).
You should therefore find files named roughly as follows:
SID_CIClass_001_min_03_20230315T082537899_0000000025.arc.csv.b03

You may check how long the same file keeps that ".bxx" extension and, more importantly, you should generate thread dumps of the bulkloader while the file is locked.
This will tell you what the bulkloader's doing (and why this prevents it from completing its current task on that file).
The issue might be related with your JCo connections (the resources allowed in your JCo configuration might be insufficient).

Even before this, you may also want to check if the bulkloader has generated automatic thread dump.
This will tell you if CC has detected issues (such as timeouts or other errors).

Please let us know if you need further guidance.

Best regards.

François
SAP Convergent Charging Support

[mrstifler777]

Thanks François, how do we check the following?

1. generate thread dumps of the bulkloader?

2. available resource for Jco (resources allowed in your JCo configuration might be insufficient)

Thanks,