500.000-1000000 IDOCs per day via XI?

[Former Member]

We are looking to implement a project that could use XI as an interface to R/3 and BW. Initially the system would receive about 5000 files daily and convert these into 500,000 - 1,000,000 IDOCs for posting into R/3 and BW. Is anyone doing anything around this size and can you provide any advice?

Thanks in advance, Hermann

[thorsten_nordholmsbirk · Accepted solution]

I have been involved with a scenario that transforms a handful of files from a legacy system into about 350.000 IDocs in R/3. This volume is processed in XI in about 15 minutes, but we learned some lessons before getting this to work:

We started out with a BPM performing a split mapping to go from batch files to single IDocs. This performed horribly in general and for the large volumes invariably crashed the XI server. The problem seemed to be that the workflow engine (which executes the BPM) cannot handle the large number of container elements. The way to avoid this problem is to use a simple, straight-through mapping (i.e. no BPM) and use IDoc bundling to send many IDocs as a single XI message. There is a note describing how this is done.

Based on the numbers you gave, you are looking (on average) at a single file being mapped to a bundle of 1000-2000 IDocs. I don't think this will be a problem for XI. However, we found that while XI handles these loads quite well, the R/3 system receiving the IDocs required significant tuning in order to keep up. (I don't know the details of this, so I can't elaborate.)

Best regards,

Thorsten

[Former Member · Accepted solution]

Hi Thorsten,

Are you able to post how you got the IDOC Bundling to work without using a BPM?

[Former Member · Accepted solution]

@Thorsten

Could you please tell us some details about the XI hardware specification used for that scenario in your project(especially processor and memory)?

Thanks in advance

Christian

[Former Member · Accepted solution]

Hi Jason,

Refer to SAP Notes 814393 to post IDoc Packaging using Message mapping. As per this note, change your idoc occurence to 1 to unbound and do a message map to collect all idocs under one root like, for example,

<ORDERS05>

<IDoc>...</IDoc>

<IDoc>...</IDoc>

<IDoc>...</IDoc>

<IDoc>...</IDoc>

</ORDERS05>

You need not build a BPM to post idoc package. With XSLT mapping it will be more easier to process IDoc package and send to R/3.

rgds,

Sasi

[Former Member · Accepted solution]

Can you explain how BPM is avoided.

I think BPM is required, otherwise how you do the Interface determination since output is your IDOC interface and since you have modified the XSD and store it as External Definition. You Interface name is not the IDoc Interface.

SO BPM is required and in the second Interface determination you keep IDOC package as source interface and IDoc as target interface . No mapping is required.

Although there will be only Recieve and Send step in Integration process.

We have completed couple of scenario with this approach.

Regards,

Satish

[Former Member · Accepted solution]

We had an interface where a 20 record text file was sent to XI (3.0 sp11) for splitting in a BPM and sent as 20 IDOC's to an SAP system and it took over 15 minutes to run. I've changed it to use IDOC packaging as described above and now it runs in less than 20 seconds...!

What is wrong with BPM???

[thorsten_nordholmsbirk · Accepted solution]

Its an IBM Regata system with 6 CPUs and at least 12GB memory allocated to the XI system. In other words, quite large

- Thorsten

[thorsten_nordholmsbirk · Accepted solution]

Avoiding the BPM is simple:

Create an Interface Mapping with your source message interface and the original (unmodified) IDoc as the target. Create a Message Mapping (for use in the Interface Mapping above) with your source message type as input and with the modified IDoc definition as the target structure.

Your receiver and interface determinations work because as far as they are concerned, they are dealing with an ordinary IDoc definition as per your Interface Mapping. The message mapping also works because XI does not validate the structure produced by the mapping program against the schema specifies in the Interface Mapping.

- Thorsten

[Former Member · Accepted solution]

Hi Jason -

Well, we had a similar issue. The difference between an interface with BPM and without BPM has to do with the way XI is persisting. When using BPM, every step, every state, every variable is logged as it passes thru. This is so that in case of failure, the process can be restarted exactly where it left off. This causes a bit of overhead, and slows the process down.

We have asked for benchmarks on XI 3.0 in this arena, but haven't recieved any. All the info we've been given has to do with 2.0. (not too helpful, as it's apples & oranges) In general, our consulting help has advises us to avoid BPM in high-volume, high-performance scenarios. Helpful, eh? Good luck...

regards

amy

[Former Member · Accepted solution]

Hi all,

We worked on a similar scenario with 200,000 IDocs's with BPM and its a very expensive process. Especially if XSLT mapping is involved forget about it. The 2 step persisting for asynchronous messages is a main culprit..But again BPM is a dream if you can accomadate a BAPI with GRAPHICALorJAVA mapping...Just a suggestion..Worked for us..

-Teresa

[Former Member · Accepted solution]

Hi All.

I am following Thorsten's approach of using IDOC packaging WITHOUT using a BPM. It is stated in his post that the "The message mapping also works because XI does not validate the structure produced by the mapping program against the schema specified in the Interface Mapping"...

I have tested this and it works great, however my concern is that upon applying further patches or upgrades to XI this "loophole" may be closed in that the Interface Mapping will check the Structure of the messages in the Message Mapping and this technique will have to be changed to use a BPM.

Any thoughts?

[thorsten_nordholmsbirk · Accepted solution]

Hi Jason

This approach was suggested to us by SAP precisely because we were using a BPM with a split mapping to produce single IDocs and it was failing to perform. After this, the note quoted by someone else in this thread was released describing this solution. So I firmly expect the current "loophole" to remain open for the foreseeable future. But I also would expect to see a more direct and obvious solution in the not-to-distant future.

Regards,

Thorsten

[Former Member · Accepted solution]

I notice that the OSS note in question (listed above) explains how to modify the idoc structure and load it into a mapping but then just ends... Leaving it open to interpratation whether its a "loophole" or bug or whatever...

[Former Member · Accepted solution]

Since posting this message OSS have come back to me and stated that this IDOC packaging capability will be kept in future releases even though validation will be done between structures used in the interface mapping and message mapping (you will have the option of de-activating the validation when using idoc packaging). Therefore making this a viable method of interfacing...

[thorsten_nordholmsbirk · Accepted solution]

Thanks for sharing that!