Processing huge file loads through XI

[Former Member]

In File To SAP integration scenario, XI File Adapter loads and processes whole file in memory. For huge files volumes such as 1GB, it doesn't seem worth doing it through XI.

Could XI File Adapter split the incoming file into multiple chunks or messages?

If not, is there any better way in XI for processing huge loads, or should I go to ETL tool for this?

In typical EAI middleware tools, we use Java IO to read the file, compose them into chunks, and put them in the JMS Queue. The next set of components can pick the data from Queue and process in parallel. This is how we solve the integration problem of processing huge volume.

Please provide your suggestions on this.

[Former Member]

And another thing,

it is also posibble to split large files,

as shown in:

/people/sravya.talanki2/blog/2005/11/29/night-mare-processing-huge-files-in-sap-xi

[Former Member]

The following blog would solve your problems:

/people/michal.krawczyk2/blog/2005/11/10/xi-the-same-filename-from-a-sender-to-a-receiver-file-adapter--sp14

in addition, if you want to save the original "File name" at the destination

read this:

/people/alessandro.guarneri/blog/2007/02/21/sap-xi-acting-as-a-huge-file-mover

[Former Member]

I was just wondering.

if my file is in a local path (no need for file transfer)

and I have a reasonable monitoring tool (direct input\batch input), why bother loading a huge file (>1G)to XI and relay on lean resources of memory in XI when I can process the file locally using an R\3 program with a path to the file and a job scheduled to run at specified time.

My guess it will take 1\4 of the time then processing the file using split and idoc\rfc processing.

what do you recon??

[Former Member]

This is an interesting discussion. Why bother? Maybe because we want to achieve 'automation' and most of the time the large file comes from somewhere else (put in an FTP server, etc).

On the other hand, it's been a pain to use EAI tool to do this thing, as I experienced the same. Because what is needed there is really a 'file mover'. The EAI tool does nothing but delivering the file from one location to another. It does not really care about the content of file, so no parsing is required. There are some EAI tools that provide this simple functionality in the market.

So I would be interested if XI can do the same -

---

without hampering system's resources.

Any more comments?

Tq.

[Former Member]

Well Ferry,

In my scenario i needed to use a file convertion inorder

to break a huge file to Financial IDocs.

This action "for my opinion" has huge memory consumption

(i didn't even botherd to test it) and when the file is allready located in the LAN the use of xi is not reccomended.

scenario Process steps:

1.take file from folder with file convertion

2.break file to groups acording to a specfific "buisness-logical" key field using java or XSLT mapping.

3. fill IDocs with the groups using BPM and mapping one-to-many.

4.dealing with errors and monitoring,on the message level,Idoc level....

So in this scenario unless we make a change in the Sender buisness system(web services,software change) the use of XI can be a dissaster.

nimrod

[Former Member]

Dear Sorna,

as you mention already it does make sense to process such a huge file with the file adapter. Another aspect is that with only one file adapter it will take very long.

We have face this problem also and we separate the file with an extra tool in small chunk and let the file process paralell with 5 file adapter.

So here we use file adapter in paralell and it works fine and the response time is better also in XI-Server.

regards,

Ly-Na Phu

[Former Member]

Hi -

I did read where the file adapter will parse for you... Look into the Recordsets per Message section on help.sap.com - when you read about converting file content in the sender. It does say you can create more than one message into XI from a single file - I'd be interested to see if anyone has tried this...

[Former Member]

Hi amanda

Well ,file adapter will generate multiple message for u if you are giving recorset per message option.( even u can create new message for each recordset) But this will ok for small file . But id the file size is huge then u need to split it before giving to adapter due to performance problem other wise u will have nightmare. Even in many cases it stops J2EE engine ..

thanx

Vinod

[Former Member]

Hello,

We are currently using the split function of a file adapter. I suggest you use EOIO processing. It will collect a certain number of messages in each chunk and send it over to XI Integration Engine. The performance of the system is fine. We were able to process files as large as 1.2 gigs.

Few notes though.

1. If you are trying to push data from a file to SAP system, be sure you have added additional connections to your RFC adapter.

2. Plus, EOIO processing together with RFC adapter only works starting with SP11.

3. It will also be interesting to see SP12, since it contains an additional queue for processing VERY large messages, as SAP mentioned in their documentation.

Will see...

Dmitriy Mindich

[Former Member]

Hi Dmitri

Can you brief us on how and where you have configured the SPLIT function in file adapter..we would appreciate if you could share your experience on this config.

regards

Sriram V.

[Former Member]

Hi guys,

Talk abt large files and nightmares....We have been seeing some of those lately although we are running only 300 mb file....We set the recordset per message to 1000 , though it improve the performance, it brought down the running time from month to days...:)...

I see the interesting post on SPLIT function..Dmitri could you explain more on that?

-Teresa

[Former Member]

Hi,

As far as I am aware, the file adapter cannot split the incoming file into multiple chunks. We are using an OS level script to pick up large files and split them into smaller chunks, which are then processed by the file adapter.

Regards

Manish