cancel
Showing results for 
Search instead for 
Did you mean: 

Ways to handle large volume data (file size = 60MB) in PI 7.0 file to file

madanmohan_agrawal
Contributor
0 Kudos

Hi,

In a file to file scenario (flat file to xml file), the flat file is getting picked up by FCC and then send to XI. In xi its performing message mapping and then XSL transformation in a sequence.

The scenario is working fine for small files (size upto 5MB) but when the input flat file size is more then 60 MB, then XI is showing lots of problem like (1) JCo call error or (2) some times even XI is stoped and we have to strat it manually again to function properly.

Please suggest some way to handle large volume (file size upto 60MB) in PI 7.0 file to file scenario.

Best Regards,

Madan Agrawal.

Accepted Solutions (1)

Accepted Solutions (1)

Former Member
0 Kudos

Hi Madan,

If every record of your source file was processed in a target system, maybe you could split your source file into several messages by setting up this in Recordset Per Messages parameter.

However, you just want to convert you .txt file into a .xml file. So, try firstly to setting up

EO_MSG_SIZE_LIMIT parameter in SXMB_ADM.

However this could solve the problem in Inegration Engine, but the problem will persit in Adapter Engine, I mean, JCo call error ...

Take into account that file is first proccessed in Adapter Engine, File Content Conversion and so on...

and then it is sent to the pipeline in Integration Engine.

Carlos

madanmohan_agrawal
Contributor
0 Kudos

Hmmm.....I think I am now getting the picture clearly. Could you please confirm the below.

To put less stress in adapter engine, we are configuring the EO_MSG_SIZE_LIMIT in SXMB_ADM and

to put less stress in integration engine, we have to use recordset per message in sender FCC.

Thanks for your information!

Br,

Madan Agrawal.

Former Member
0 Kudos

Hi,

No, its exactly the opposite.

SXMB_ADM --> Integration Engine.

FCC --> Adapter Engine.

Carlos

madanmohan_agrawal
Contributor
0 Kudos

Thanks Carlos !

Answers (1)

Answers (1)

prateek
Active Contributor
0 Kudos

You may split or zip the incoming file as per the requirement.

Regards,

Prateek

madanmohan_agrawal
Contributor
0 Kudos

Thanks Prateek for quick reply,

In my scenario I have to split the messages and I have also tried that using FCC "recordset per messages " but that will split my output to multiple files but my client dont want to generate multiple files for one file. they want all data in one file. so please suggest some way!

Br,

Madan Agrawal

former_member181985
Active Contributor
0 Kudos

Hi,

In the receiver file adapter CC use Append as construct mode.

Also use Adapter specific parameters if you want to use the same file name in the target side.

Thanks,

Gujjeti.

Former Member
madanmohan_agrawal
Contributor
0 Kudos

Thanks Carlos,

I think this is a good solution.

But do we need to use both (1) tuning of EO_MSG_SIZE_LIMIT in SXMB_ADM and (2) recordset per message in FCC.

Br,

Madan Agrawal