Handling a large message(>100MB) in a Proxy to File scenario in PI 7.31

[Former Member]

Hi,

We have a proxy to SFTP interface where PI is receiving an XML proxy message with message size between 100 MB - 150 MB.

The message has reached the maximum size limit and is unable to get out of the ECC system; it is throwing the below error:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>

- <SAP:Error SOAP:mustUnderstand="1" xmlns:SAP="http://sap.com/xi/XI/Message/30" xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">

<SAP:Category>XIServer</SAP:Category>

<SAP:Code area="INTERNAL">CLIENT_SEND_FAILED</SAP:Code>

<SAP:P1>400</SAP:P1>

<SAP:P2>Bad Request</SAP:P2>

<SAP:P3>(See attachment HTMLError for details)</SAP:P3>

<SAP:P4 />

<SAP:AdditionalText />

<SAP:Stack>Error while sending by HTTP (error code: 400 , error text: Bad Request) (See attachment HTMLError for details)</SAP:Stack>

<SAP:Retry>A</SAP:Retry>

</SAP:Error>

To add a little more context to the issue, we are on PI 7.31 (single stack). We do not want to dump the PI system with such huge size of data, rather thinking to split it into smaller chunks in ECC and stitching them back in PI into one single file. This is because, the target legacy system can not process multiple files; it can process one full load file per day.

I have come across the below blog where splitting and stitching methods are explained:

http://scn.sap.com/people/smasanta/blog/2012/12/21/how-to-handle-large-volume-of-data-for-proxy-to-f...

But, the blog involves OS level commands and writing the file to a temporary folder and I assume this blog was written keeping PI 7.1 in mind.

Is there any other optimum solution using standard methods to handle the large messages in PI 7.31, bearing the below points in mind:

1. We do not want to change the system level parameters(this might impact the overall performance of the system)

2. BPM is not a preference(can go for it if there is no other option). This interface runs just once in a day, hence just thinking if developing a BPM interface is really worth it.

Awaiting for a solution to this problem.

Thank you.

Regards,

Subbalaxmi Musunuri

[engswee · Accepted solution]

Hi Subbalaxmi

I think there are very limited standard methods to deal with your scenario. If you really need to split it on the sender side due to size limitation, you need a "staging" area to hold the intermediate messages. Temporary files approach is a common method to stage these intermediate messages. However, looking at the design mentioned in the blog, it does seem quite elaborate.

I'd personally go for BPM (especially if you have NW BPM and not ccBPM) because it's not too difficult to design an aggregator pattern (I just so happen to complete one recently!) You can check out the following blog that list a series of post related to aggregator patterns.

Rgds

Eng Swee

[Former Member · Accepted solution]

Hi Eng,

I agree with your approach for combining the messages but is it recommended to process messages around 150MB through PI ?

BR,

Harish

[engswee · Accepted solution]

Harish, you are definitely right and I totally missed it. My bad! Thanks for pointing it out. If it can't handle the initial full 150MB load, it wouldn't be able to handle it when it eventually combine them back.

[Former Member]

Hello All,

Thank you for your valuable responses.

For the problem I mentioned above, as an immediate solution, we have reduced the length of the XML tag names to a maximum of 2 - 3 characters for each field(We have more than 100 fields in the structure). Since, the XML tag names occupy more message size than the actual data, this helped in reducing the XML message size to 1/3 rd of the actual message size. This change to the message structure resolved the issue for now and the messages are getting processed successfully in both ECC and PI systems.

Further to this, we are looking at implementing BPM(splitting and stitching approach) as a long term/ permanent solution to these kind of issues.

Thanks once again for your inputs.

Regards,

Subbalaxmi Musunuri

[former_member182412]

Hi Subbalaxmi,

• Instead of sending the data via proxy from ECC you can create the final text file using ABAP program and create the single file in ECC application server.
• You can send this file using chunk mode option in File adapter, you can refer below blog.

Regards,

Praveen.

[siddhesh_pathak4]

Hello Subbalaxmi,

If your file is text file you can split the data coming from ECC in chunks and process that in single file using APPEND in file communication. If you want to process the records in sequence use EOIO.

[RaghuVamseedhar]

Subbalaxi,

If target file is text file (not XML), try this: -

First interface: - Small messages from SAP ECC -> SAP PI EOIO -> Temp folder (append mode).

Second interface: - Pick above file -> SAP PI -> SFTP. If header and trailer are required add in mapping.

If target file is XML: - Schedule the interface (>150 MB XML) when there is less load on PI system. There is no official size limit, it depends on hardware, available java heap memory and complexity of mapping. Try to reduce complexity of mapping, PI will be able to handle huge file.

[iaki_vila]

Hi Subbalaxmi,

The issue doesn't  seem to be a memory problem, that it is the usual in these cases, you can change the sxmb_adm ECC parameter EO_MSG_SIZE_LIMIT. Also, i know you concern about to overload the ECC system, in this possibility you can develop a own module, or anotther scenario which responsibility was to split the files in n parts and a second scenario to take only this n parts.

Regards.