Skip to Content

How to split a XML structured file into chunks?

Hello Experts,

We have a Synchronous scenario File(sftp Adapter) to Web service to File(sftp adapter) with structured XML data. The file size is around 200MB with 200000 records. How can I split the 200MB files into 5MB of size each? or is there any way that I can process only N(Let's say first 1000, and then next 1000...until EOF ) number of records out of unknown (Let's say 2 million) number of records?

Please suggest

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

4 Answers

  • Jun 15, 2017 at 09:26 AM

    Hi Easy Jasunai!

    You can use custom adapter module for splitting source XML and send it one by one to target system.

    Another option (without java development) might be splitting your scenario: at first step you take source XML and put it to some folder using FCC to convert it to a flat file. At second step you take that file and convert it back to XML structure with FCC using parameter "Recordset Per Message" in FCC settings.

    Also you could try to split your source XML using message mapping.

    Regards, Evgeniy.

    Add comment
    10|10000 characters needed characters exceeded

  • Jun 15, 2017 at 09:59 AM
    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    Jun 15, 2017 at 11:22 AM

    You can achieve it by changing the occurance of message type from 0...1 to 0...unbound and then apply some logic in message mapping to split the records in chunks of 1000 .As in the picture :

    Add comment
    10|10000 characters needed characters exceeded

    • Hi Inaki!

      I guess the possible solution might be using flat file at sender's side. Just hope that reading the file using FCC and composing output messages according to "Recordset Per Message" parameter setting consumes less resources than processing XML structure with message mapping.

      Regards, Evgeniy.

  • Jun 15, 2017 at 02:46 PM

    I have a 198094 Record Nodes in the Soruce XML file with 10 elements in each Record node. Would anyone please advise how can I handle this logic in MM such as create context where it can make 20 web service calls of requests and get 20 responses back and write it to 20 files. In my scenario I'm using Module bean which is working fine as expected for less than 100 record nodes. The same scenario needs to work with heavy load in reality. Record(Source XML parent Node) - Count(Statistic function) - Buffer(Target Web service structure Element). I'm getting 198094 in this Buffer Element at target side.

    Add comment
    10|10000 characters needed characters exceeded