cancel
Showing results for 
Search instead for 
Did you mean: 

PI 7.3 - Huge File Processing

0 Kudos

Hello,

I am working on a File to SOAP Integration Scenario, with a very huge file processing (>500Mb).

I have read about the Chunk capabilities on File Sender Adapter, but as far as i know, it chunks the file once it's loaded by the adapter, so the Adapter Engine keeps having to process the whole file and it seems it could easily cause an OutOfMemoryError in J2EE Server.

Has anybody ever worked with so huge files in 7.3 version? Is there another workaround?

Thank you very much.

Regards,

Fran

Accepted Solutions (1)

Accepted Solutions (1)

rajasekhar_reddy14
Active Contributor
0 Kudos

is it Flat file or XML file? if it is flat file try using Record set per message option, defintly it helps to split the file.

0 Kudos

Hello Raja,

We have both cases (one scenario with Flat file and one other with XML). Correct me if i am wrong, but the problem will still being at Adapter Engine side, isn't it?

I will try the Recordset per message option and confirm you.

Thanks a lot!

Fran

rajasekhar_reddy14
Active Contributor
0 Kudos

if you are doing mapping then i dont see an y better alternatuve than record set per message.

Answers (3)

Answers (3)

iaki_vila
Active Contributor
0 Kudos

Hi Fran,

If you read carefully the Michal Krawczyk l's blog http://scn.sap.com/community/pi-and-soa-middleware/blog/2010/10/18/pixi-pi-73-processing-of-large-fi...

you can observe that from SAP PI 7.3 the chunk file division tries to avoid the memory problem.

Also there are interesting these two blogs:

Niki Scaglione's blog http://scn.sap.com/people/niki.scaglione2/blog/2009/10/31/chunkmode-for-binary-file-transfer-within-...where is including an interesting comparative

Former Member's blog http://scn.sap.com/community/pi-and-soa-middleware/blog/2011/12/26/fileftp-adapter--large-file-trans... where you can obtain a valuable information how the chunk mode works

Regards.

0 Kudos

Thank you very much Iñaki,

But unfortunately, i need to extract the information included in the file and apply a simple mapping to call with it to a Web Service in the receiver side.

So it seems the chunkmode will not work in my case, as it only applies to File Transfer scenarios (without any mapping).

Thanks and Regards,

Fran

nageshwar_reddy
Contributor
0 Kudos

Hi ,

I did work with with a file of size ~500MB to test the chunk mode capabilities in PI 7.3. I did not encounter out of memory error. This was plain file transfer without any mapping. Just remember that mapping is not possible in chunk mode. Given this fact i am not sure it makes sense for your scenario(File->SOAP).

I have read about the Chunk capabilities on File Sender Adapter, but as far as i know, it chunks the file once it's loaded by the adapter, so the Adapter Engine keeps having to process the whole file and it seems it could easily cause an OutOfMemoryError in J2EE Server.

Has anybody ever worked with so huge files in 7.3 version? Is there another workaround?

udo_martens
Active Contributor
0 Kudos

Hi Fran,

i think you should test it. The file adapter can handle up to 2 Gig - of course dependend from server RAM. Just create a chunk scenario and try it, i guess it will work. Consider that test environments often have more RAM than development environment.

Regards,

Udo