Skip to Content
avatar image
Former Member

File with huge data

Hi Experts,

I am going to send a file which will be a huge file with 1GB size. What precautions should I take to process the huge file using XI? Scenario is File to IDoc. Suggest me with different options to send a huge file thourgh XI without performance issue.

Regards,

Ramesh.

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

6 Answers

  • Best Answer
    avatar image
    Former Member
    Sep 13, 2009 at 03:16 AM

    Hi Ramesh,

    First suggestion to you would be to try using RMFT.(RepliWeb Managed File Transfer). this is a tool which can be used to do massive data transfer with many file related operations simplified.Try exploring this,it will prove a good option now and in future too for your organization and save XI too!!!....[more info|http://www.repliweb.com/products/rmft/index.php]

    The other options would be as suggested in the below mentioned blogs or forums ,have a look

    1. [link|http://www.sdn.sap.com/irj/scn/weblogs;jsessionid=(J2EE3414800)ID2129061550DB01842675584212478734End?blog=/pub/wlg/2737]

    2. Massive file

    3.Processing huge file loads through XI

    4.Processing of huge no of Inbound xml files

    5.Bottleneck in Large file processing

    6.How to process large data files in XI ? 100 MB files ?

    7. search sdn forums ,there are lots of discussions going on this!

    Thanks and regards,

    Srini

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    Sep 11, 2009 at 10:28 AM

    Hi,

    >>I am going to send a file which will be a huge file with 1GB size. What precautions should I take to process the huge file using XI? Scenario is File to IDoc. Suggest me with different options to send a huge file thourgh XI without performance issue.

    Don't use file adapter with GB files.. instead try

    1. breaking the file (if input file is a txt file then recordset per message can be used)

    2. java proxy in sender side

    3. Try http post method, where http adapter will be in the sender. (I am not sure whether this will be feasible for you or not)

    Regards

    Suraj

    Add comment
    10|10000 characters needed characters exceeded

  • Sep 13, 2009 at 11:05 AM

    Hi Ramesh,

    Use record set per message option in File sender channel, and give value aproximately 250 like that, check the performnce,if its good, then use 250,if you are facing problems then reduce the value.

    Cheers,

    Raj

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    Sep 11, 2009 at 05:54 PM

    Hi Ramesh,

    You can use File for this, but you might like to do chunking.

    Few days back I was working on this huge volume of data going to BW and then being used by Business Objects. I was picking up 16 megs at a time, with read time of 20-25 secs and processing time of 37-42 secs.

    Mind you, I was not using any BPM.

    So, if you using BPM, you should better chunk it further.

    Also, you should check with the Business user, how much time window they can provide PI before they expect the message to be processed at their end.

    Hope this helps.

    Regards,

    Neetesh

    Edited by: Neetesh Raj on Sep 11, 2009 11:57 AM

    Add comment
    10|10000 characters needed characters exceeded

  • Sep 11, 2009 at 06:29 PM

    Ramesh,

    >Scenario is File to IDoc

    Hence Chunk the file and send it.

    Cheers

    Agasthuri

    Add comment
    10|10000 characters needed characters exceeded

    • Hey,

      >file which will be a huge file with 1GB size. How do I chunk the file? Could you please elaborate?

      Sender side can chunk the file and send or in File adapter use record set where the file will be broken and read automatically

      Cheers

      Agasthuri

  • avatar image
    Former Member
    Sep 12, 2009 at 10:56 PM

    Edited by: on Sep 13, 2009 12:57 AM

    Edited by:p 13 on Se, 2009 12:57 AM

    Add comment
    10|10000 characters needed characters exceeded