Skip to Content

PI 7.3 - Huge File Processing

Hello,

I am working on a File to SOAP Integration Scenario, with a very huge file processing (>500Mb).

I have read about the Chunk capabilities on File Sender Adapter, but as far as i know, it chunks the file once it's loaded by the adapter, so the Adapter Engine keeps having to process the whole file and it seems it could easily cause an OutOfMemoryError in J2EE Server.

Has anybody ever worked with so huge files in 7.3 version? Is there another workaround?

Thank you very much.

Regards,

Fran

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

4 Answers

  • Best Answer
    Oct 08, 2012 at 04:59 PM

    is it Flat file or XML file? if it is flat file try using Record set per message option, defintly it helps to split the file.

    Add comment
    10|10000 characters needed characters exceeded

  • Oct 08, 2012 at 04:39 PM

    Hi Fran,

    i think you should test it. The file adapter can handle up to 2 Gig - of course dependend from server RAM. Just create a chunk scenario and try it, i guess it will work. Consider that test environments often have more RAM than development environment.

    Regards,

    Udo

    Add comment
    10|10000 characters needed characters exceeded

  • Oct 09, 2012 at 12:07 AM

    Hi Fran Pelayo,

    I did work with with a file of size ~500MB to test the chunk mode capabilities in PI 7.3. I did not encounter out of memory error. This was plain file transfer without any mapping. Just remember that mapping is not possible in chunk mode. Given this fact i am not sure it makes sense for your scenario(File->SOAP).

    I have read about the Chunk capabilities on File Sender Adapter, but as far as i know, it chunks the file once it's loaded by the adapter, so the Adapter Engine keeps having to process the whole file and it seems it could easily cause an OutOfMemoryError in J2EE Server.

    Has anybody ever worked with so huge files in 7.3 version? Is there another workaround?

    Add comment
    10|10000 characters needed characters exceeded

  • Oct 10, 2012 at 11:47 AM

    Hi Fran,

    If you read carefully the Michal Krawczyk l's blog http://scn.sap.com/community/pi-and-soa-middleware/blog/2010/10/18/pixi-pi-73-processing-of-large-files--teaser

    you can observe that from SAP PI 7.3 the chunk file division tries to avoid the memory problem.

    Also there are interesting these two blogs:

    Niki Scaglione's blog http://scn.sap.com/people/niki.scaglione2/blog/2009/10/31/chunkmode-for-binary-file-transfer-within-pi-71-ehp1where is including an interesting comparative

    Former Member's blog http://scn.sap.com/community/pi-and-soa-middleware/blog/2011/12/26/fileftp-adapter--large-file-transfer-chunk-mode where you can obtain a valuable information how the chunk mode works

    Regards.

    Add comment
    10|10000 characters needed characters exceeded

    • Thank you very much Iñaki,

      But unfortunately, i need to extract the information included in the file and apply a simple mapping to call with it to a Web Service in the receiver side.

      So it seems the chunkmode will not work in my case, as it only applies to File Transfer scenarios (without any mapping).

      Thanks and Regards,

      Fran