on 10-08-2012 5:07 PM
Hello,
I am working on a File to SOAP Integration Scenario, with a very huge file processing (>500Mb).
I have read about the Chunk capabilities on File Sender Adapter, but as far as i know, it chunks the file once it's loaded by the adapter, so the Adapter Engine keeps having to process the whole file and it seems it could easily cause an OutOfMemoryError in J2EE Server.
Has anybody ever worked with so huge files in 7.3 version? Is there another workaround?
Thank you very much.
Regards,
Fran
is it Flat file or XML file? if it is flat file try using Record set per message option, defintly it helps to split the file.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Fran,
If you read carefully the Michal Krawczyk l's blog http://scn.sap.com/community/pi-and-soa-middleware/blog/2010/10/18/pixi-pi-73-processing-of-large-fi...
you can observe that from SAP PI 7.3 the chunk file division tries to avoid the memory problem.
Also there are interesting these two blogs:
Niki Scaglione's blog http://scn.sap.com/people/niki.scaglione2/blog/2009/10/31/chunkmode-for-binary-file-transfer-within-...where is including an interesting comparative
Former Member's blog http://scn.sap.com/community/pi-and-soa-middleware/blog/2011/12/26/fileftp-adapter--large-file-trans... where you can obtain a valuable information how the chunk mode works
Regards.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Thank you very much Iñaki,
But unfortunately, i need to extract the information included in the file and apply a simple mapping to call with it to a Web Service in the receiver side.
So it seems the chunkmode will not work in my case, as it only applies to File Transfer scenarios (without any mapping).
Thanks and Regards,
Fran
Hi Fran Pelayo,
I did work with with a file of size ~500MB to test the chunk mode capabilities in PI 7.3. I did not encounter out of memory error. This was plain file transfer without any mapping. Just remember that mapping is not possible in chunk mode. Given this fact i am not sure it makes sense for your scenario(File->SOAP).
I have read about the Chunk capabilities on File Sender Adapter, but as far as i know, it chunks the file once it's loaded by the adapter, so the Adapter Engine keeps having to process the whole file and it seems it could easily cause an OutOfMemoryError in J2EE Server.
Has anybody ever worked with so huge files in 7.3 version? Is there another workaround?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Fran,
i think you should test it. The file adapter can handle up to 2 Gig - of course dependend from server RAM. Just create a chunk scenario and try it, i guess it will work. Consider that test environments often have more RAM than development environment.
Regards,
Udo
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
84 | |
23 | |
11 | |
9 | |
8 | |
5 | |
5 | |
5 | |
5 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.