Skip to Content
avatar image
Former Member

Parallel processing skipping records


I am implementing parallel processing for creating supplier master in S4H system.

I have created a custom BAPI that is a replica of standard BAPI RFC_CVI_EI_INBOUND_MAIN with changes meeting customer requirements.

I have implemented the logic as below.

DO (free work process) times.

1.) Divide total records by number of work processes(lets say = 100).

2.) use append.... from 1 to 100, 101 to 200 etc in each loop.

3.) CALL FUNCTION "Function module to perform "in parallel

STARTING NEW TASK w_taskname "Name for identifying this RFC call
DESTINATION IN GROUP w_group "Name of group of servers to use for parallel processing.


if sy-subrc = 0.

w_total_jobs = w_total_jobs + 1.


WAIT UNTIL gw_rcv_jobs >= w_total_jobs

here gw_rcv_job is incrementing in the callback perform.

and w_total_jobs is total jobs that are called in parallel processes..

Sometimes I get all records like if I load 10k ... resultant spool output is 10k

Usually for small number range like 1k to 5k all records are returned by this code but when range goes over 10k then there is inconsistency in the returned data to spool.

Kindly, help in identifying the issue. Whether it is related to my code or some scenario that I am not handling.

<removed by moderator>

Help is appreciated.

Thanks in advance.

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

2 Answers

  • Aug 08, 2017 at 08:29 AM

    Your logic is wrong. The basic issue is that you launch a parallel thread and then increment the counter. If a thread finishes before all the FMs have been launched, you'll leave the loop early.

    When I use parallelisation, I take the number of available threads, and (typically), divide by 3, to give the maximum number of threads I'll use. E.g. I might decide I'll have a maximum of 3 threads.

    I hardcode (or allow setting by parameter) a minimum number of records to process at once. E.g. 101000 records - I set my minimum to 20000. In this way, I prevent a thread processing just a few records, which would be unperformant. I then process my data in chunks, bearing in mind, I have only 3 threads and 5 chunks, so I have to reuse a thread when one becomes available. One approach would be this:

    • I take records 1 to 20000, and CALL FM with them
    • I take records 20001 to 40000 and CALL FM with them
    • I take records 40001 to 60000 and CALL FM with them

    I wait until all three threads are complete.

    • I take records 60001 to 80000 and CALL FM with them
    • I take records 80001 to 101000 and CALL FM with them - I have to put the extra 1000 in, otherwise my next FM call will have fewere records than my minimum.

    I wait until these two threads are complete.

    This makes for simple thread handling. A more complicated approach, to use perhaps when the runtimes of each thread can vary quite a bit, is to keep track of active threads, and as soon as one becomes free, if there's still data left to process, then start a new thread.

    Other issues.

    • You check sy-subrc after the CALL FM. This is wrong. You should check return codes (and act on them) in the callback form
    • You've written it using performs instead of methods. Forms are obsolete nowadays.
    Add comment
    10|10000 characters needed characters exceeded

  • Aug 07, 2017 at 03:21 PM

    First off, I would call this "over-parallelization". Beyond a certain number, parallel processes don't provide any significant performance benefit because the management takes up more and more resources and you start to run into all sorts of issues. Usually this number is in the range of 5-20 processes, so at 100 I do wonder...

    Next, you did a good job of describing your setup, but fell short in describing the problem. For "inconsistency in the returned data", I can at best answer: something must be going wrong.

    Suggest reducing your number of processes to 5, 10, and 20; and doing some real real measurement to find what works. See if you still have issues at those numbers.

    Add comment
    10|10000 characters needed characters exceeded