How to handle large data in file adapter

[vkaushik82]

We have a scenario Proxy -> PI -> File Sever using File adapter.

File adapter is using FCC for conversion.

recently we had wave 2 products live and suddenly for this interface we have increase in volume of messages, due to which File adapter is not performing well, PI goes slow or frequent disconnect from file server problem. Due to which either we will have duplicate records in file or file format created is wrong.

File size is somewhere around 4.07 GB which I also think quite high for PI to handle.

Can anybody suggest how we can handle such large data.

Regards,

Vikrant

[Former Member]

Hi Vikrant ,

We had the similar situation in our Previous Project ..

Only thing you can do that split the message in PI at the mapping level ..(Multi mapping - 0..unbounded )

Write an UDF which will split the message into multiple buckets ..Lets say ,each bucket contains 5000 records and use Appending mode in Receiver Adapter to append the records into the same file..

let me know if you need any other info.

Regards

Anandan

[rajasekhar_reddy14]

We have a scenario Proxy -> PI -> File Sever using File adapter.

Record set per message option only works with sender File adapter so in your case it is not going to help at all.

When you say File adapter not performing well means what is happening adapter getting locked?

If possible try to split records in proxy and process it and try to minimize FTP calls as much you can.

Regards,

Raj

[anupam_ghosh2]

Hi Vikrant,

You can use java mapping to read (from proxy) and write the file contents in chunks instead of using FCC.

Thus the problem might be avoided.

Regards

Anupam

Edited by: anupamsap on Mar 8, 2012 6:28 PM

[Former Member]

Vikrant,

>>>File size is somewhere around 4.07 GB,due to which File adapter is not performing well, PI goes slow or frequent disconnect from file server problem

As Baskar & Maciej mentioned chunk the file & consolidate is a better option

Cheers

Agasthuri

[maciej_jarecki]

Hi,

If you use PI 7.11 first of all use Ico. Then write custom adapter module that will split files into smaller part and attache it to first message.

BR

Maciej

[Former Member]

Check this Blog for Huge File Processing:

However, you can take a look also to this Blog, about High Volume Messages:

PI Performance Tuning Best Practice:

http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/2016a0b1-1780-2b10-97bd-be3ac6221...

[baskar_gopalakrishnan2]

>File adapter is using FCC for conversion

You can specify the following in the conversion to split the message in to chunk

Recordsets per Message: 500 (just desired number) to split the message in to multiple messages.

Another tip is

use tuning parameter in SXMB_ADM

EO_MSG_SIZE_LIMIT (specify message size here to handle these kind of message in a special queue)