Skip to Content
0

How to split a XML structured file into chunks?

Jun 15, 2017 at 03:12 AM

124

avatar image

Hello Experts,

We have a Synchronous scenario File(sftp Adapter) to Web service to File(sftp adapter) with structured XML data. The file size is around 200MB with 200000 records. How can I split the 200MB files into 5MB of size each? or is there any way that I can process only N(Let's say first 1000, and then next 1000...until EOF ) number of records out of unknown (Let's say 2 million) number of records?

Please suggest

10 |10000 characters needed characters left characters exceeded
* Please Login or Register to Answer, Follow or Comment.

4 Answers

Evgeniy Kolmakov Jun 15, 2017 at 09:26 AM
1

Hi Easy Jasunai!

You can use custom adapter module for splitting source XML and send it one by one to target system.

Another option (without java development) might be splitting your scenario: at first step you take source XML and put it to some folder using FCC to convert it to a flat file. At second step you take that file and convert it back to XML structure with FCC using parameter "Recordset Per Message" in FCC settings.

Also you could try to split your source XML using message mapping.

Regards, Evgeniy.

Show 3 Share
10 |10000 characters needed characters left characters exceeded

Hi Eve,

In this Case Normal choosing Chucnk Mode does not help??

0

Chunk mode is applicable only for Binary files but not on Structured files

1

Hi Gurudatta!

Please read my answer to Raghu's similar question. I guess it helps.

Regards, Evgeniy.

0
Raghuraman S Jun 15, 2017 at 09:59 AM
0
Show 2 Share
10 |10000 characters needed characters left characters exceeded

Hi Raghu!

Actually, chunk mode is used for binary file transfer without any access to payload (simple pass-through scenario).

As per the above screenshots, the split never cosiders the payload. It’s just a binary split. So the following limitations would apply

  • Only for File Sender to File Receiver
  • No Mapping
  • No Content Based Routing
  • No Content Conversion
  • No Custom Modules

And if I understand the current requirement right, splitted xml should be sent to web service and responses from web service should be saved as files.

Regards, Evgeniy.

1

Right catch.

0
PIYUSH SHARMA Jun 15, 2017 at 11:22 AM
0

You can achieve it by changing the occurance of message type from 0...1 to 0...unbound and then apply some logic in message mapping to split the records in chunks of 1000 .As in the picture :


capture.jpg (23.7 kB)
Show 3 Share
10 |10000 characters needed characters left characters exceeded

Hi PIYUSH!

The main question is if the PI/PO java server has enough resources to perform message mapping on 200 MB XML: file.

You can read about issues while processing ~20 MB files using graphical mapping:

https://answers.sap.com/questions/217145/queue-stopped-message-timeout-file-greater-than-50.html?childToView=217609#answer-217609

Regards, Evgeniy.

1

Hi,

I totally agree with Evgeniy, i had this problem in the past and PI memory could end overload, even if you increase the memory size in the hardware, there can appear other bottlenecks. No PI solution is applicable here, because you will need the chunks will be XMLs well formed, this means a XML parsing and treatment. The sender system must send less large XMLs

Regards.

1

Hi Inaki!

I guess the possible solution might be using flat file at sender's side. Just hope that reading the file using FCC and composing output messages according to "Recordset Per Message" parameter setting consumes less resources than processing XML structure with message mapping.

Regards, Evgeniy.

0
Easy Jasunai Jun 15, 2017 at 02:46 PM
0

I have a 198094 Record Nodes in the Soruce XML file with 10 elements in each Record node. Would anyone please advise how can I handle this logic in MM such as create context where it can make 20 web service calls of requests and get 20 responses back and write it to 20 files. In my scenario I'm using Module bean which is working fine as expected for less than 100 record nodes. The same scenario needs to work with heavy load in reality. Record(Source XML parent Node) - Count(Statistic function) - Buffer(Target Web service structure Element). I'm getting 198094 in this Buffer Element at target side.

Show 1 Share
10 |10000 characters needed characters left characters exceeded

The most suitable approach is to request sender party to split the file on their side.

Regards, Evgeniy.

1