Skip to Content

Parallel processing skipping records

Aug 07, 2017 at 02:35 PM


avatar image


I am implementing parallel processing for creating supplier master in S4H system.

I have created a custom BAPI that is a replica of standard BAPI RFC_CVI_EI_INBOUND_MAIN with changes meeting customer requirements.

I have implemented the logic as below.

DO (free work process) times.

1.) Divide total records by number of work processes(lets say = 100).

2.) use append.... from 1 to 100, 101 to 200 etc in each loop.

3.) CALL FUNCTION "Function module to perform "in parallel

STARTING NEW TASK w_taskname "Name for identifying this RFC call
DESTINATION IN GROUP w_group "Name of group of servers to use for parallel processing.


if sy-subrc = 0.

w_total_jobs = w_total_jobs + 1.


WAIT UNTIL gw_rcv_jobs >= w_total_jobs

here gw_rcv_job is incrementing in the callback perform.

and w_total_jobs is total jobs that are called in parallel processes..

Sometimes I get all records like if I load 10k ... resultant spool output is 10k

Usually for small number range like 1k to 5k all records are returned by this code but when range goes over 10k then there is inconsistency in the returned data to spool.

Kindly, help in identifying the issue. Whether it is related to my code or some scenario that I am not handling.

<removed by moderator>

Help is appreciated.

Thanks in advance.

10 |10000 characters needed characters left characters exceeded
* Please Login or Register to Answer, Follow or Comment.

2 Answers

Matthew Billingham
Aug 08, 2017 at 08:29 AM

Your logic is wrong. The basic issue is that you launch a parallel thread and then increment the counter. If a thread finishes before all the FMs have been launched, you'll leave the loop early.

When I use parallelisation, I take the number of available threads, and (typically), divide by 3, to give the maximum number of threads I'll use. E.g. I might decide I'll have a maximum of 3 threads.

I hardcode (or allow setting by parameter) a minimum number of records to process at once. E.g. 101000 records - I set my minimum to 20000. In this way, I prevent a thread processing just a few records, which would be unperformant. I then process my data in chunks, bearing in mind, I have only 3 threads and 5 chunks, so I have to reuse a thread when one becomes available. One approach would be this:

  • I take records 1 to 20000, and CALL FM with them
  • I take records 20001 to 40000 and CALL FM with them
  • I take records 40001 to 60000 and CALL FM with them

I wait until all three threads are complete.

  • I take records 60001 to 80000 and CALL FM with them
  • I take records 80001 to 101000 and CALL FM with them - I have to put the extra 1000 in, otherwise my next FM call will have fewere records than my minimum.

I wait until these two threads are complete.

This makes for simple thread handling. A more complicated approach, to use perhaps when the runtimes of each thread can vary quite a bit, is to keep track of active threads, and as soon as one becomes free, if there's still data left to process, then start a new thread.

Other issues.

  • You check sy-subrc after the CALL FM. This is wrong. You should check return codes (and act on them) in the callback form
  • You've written it using performs instead of methods. Forms are obsolete nowadays.
10 |10000 characters needed characters left characters exceeded
Mike Pokraka Aug 07, 2017 at 03:21 PM

First off, I would call this "over-parallelization". Beyond a certain number, parallel processes don't provide any significant performance benefit because the management takes up more and more resources and you start to run into all sorts of issues. Usually this number is in the range of 5-20 processes, so at 100 I do wonder...

Next, you did a good job of describing your setup, but fell short in describing the problem. For "inconsistency in the returned data", I can at best answer: something must be going wrong.

Suggest reducing your number of processes to 5, 10, and 20; and doing some real real measurement to find what works. See if you still have issues at those numbers.

Show 3 Share
10 |10000 characters needed characters left characters exceeded

"you start to run into all sorts of issues"


I wrote a parallel processing system that spawned jobs to handle an incoming sales order - creating production orders, purchase orders, configurabale materials etc etc etc which worked wonderfully in development and QA.

In production the first big order to come in (which was to clad a warehouse nearly a kilometre long) hung the pc on which it was processing.

Then...slowly but surely (in the words of 'War of the Worlds'....) it made it's plans against the Sales department and each session on each pc ground to a halt....

The only way out of it was to shut down SAP......


I did decrease the number of work processes for parallel processing.

The issue is that when dialog process are created for processing I usually get a dump for some processes: 'DBIF_SETG_SQL_ERROR' 'DBSQL_SQL_ERROR'. ‘SQL error "-10807" while accessing a table’

I have traced the dump to SM21 to find that the connectivity between the AS and the database is breaking hence the skipping of records.

Now is this happening because the dialog process is running for long or is this some issue with the client version or maybe the disk space.

The BAPI that I am calling does take 1.5 -1.7 seconds to process data due to BAPI_TRANSACTION_COMMIT being called inside the loop.



Are you trying to perform parallel processing in foreground ? That's not designed for that purpose.

Moreover, is your calling machine external to your processing group?

Best regards