Out of Memory Exception when processing SAP messages in xMII

[Former Member]

Hi All,

I am using xmii 12. I have configured a message listener to listen for OPERA3 messages from SAP system and have configured a transaction to get executed when such a message comes.

The transaction will write the received XML into a file in local system and will execute a custom action which will parse the file and insert its contents to DB.

But twice when an initial download for OPERA3 has been triggered, the j2ee server has restarted with an exit code of 666(out of memory exception).

The file size of the OPERA3 xml message is 57MB and it has 60000 records in it.

I have created a custom action since standard XML Loader action will slow down handling files with larger size

1) Is the problem due to large amount of records and file size?

2) Should i have to increase the jvm parameters?

3) How should i approach messages with large number of records?

Thanks

[Former Member · Accepted solution]

Hi,

mass data and XML is a known issue. To process such data the complete DOM is built in memory by MII. If you want to fight this issue with

resources the only chance you have is that you use x64 Windows Server für MII with several GB for the JVM. Typically you could try to give

the JVM a heap size of 3GB/4GB (keep increasing till it works :-). Though this is architectural not a nice solution, so you should send, for

example, the message zipped and base64 encoded. This message could then be received by MII and saved to a file. Decoding and dezipping

this file should be done on file system such that a decoded and dezipped file is created. Then you can process this file with a custom action.

Remember that you will face the same resource problems in your custom action. The solution is to use an event driven parser like SAX to parse

the file and do the inserts in JDBC batch mode. With this procedure you can guarantee that a fixed amount of memory is used when parsing

a big XML file.

Good luck,

Matthias

[agentry_src]

You should start with your logs and looking at the jvm settings. I would also do a search on this forum for "Heap" or "Heap size". You are not the first person to run into jvm limitations while processing huge files. I would also suggest looking on the SAP ERP side of things to see if you A) really need 57M of data and B) 60000 records. If you are doing something along the lines of downloading all the material masters, then I would suggest looking into Change Pointers in the ERP system and only download those which need updating.

Please give a bit more information on your custom action block and also what OPERA3 messages are.

Thanks,

Mike