cancel
Showing results for 
Search instead for 
Did you mean: 

Reducing PI Map Memory Requiement via Premap

HarshC
Active Participant
0 Kudos

Hi All,

We are trying to map large 200+ MB XML files using PI 7.11.

The target is a flat CSV. The maps are quite complex with multiple lookups.

Since we only use a small subset of the xml fields in the source file, I'm wondering if the following "PreMap" would help reduce the memory requirements.

Source XML(200MB+) -> PreMap-> Source XML*(100MB+)

Source XML*(100MB+) -> MainMap ->Target CSV(10MB)

How does this compare with:

Source XML(200MB+) -> MainMap ->Target CSV(10MB)


Note:

1) Both maps are MessageMappings.

2) PreMap is a simple map that discards all XML fields that MAP2 doesn't need, but keeps the same XML structure.

3) MainMap is a complex map that will have mulitple lookups.

Is there any benefit to a PreMap that filters out fields that are not required by the main map? Or maybe it makes things worse?

I'm hoping to optimize memory consumption.

Thanks,

Harsh

Accepted Solutions (0)

Answers (3)

Answers (3)

iaki_vila
Active Contributor
0 Kudos

Hi Harsh,

I think two maps is a worse option.. We have a scenario how worked with files around 100Mb, i've notice that they converted themselves on 200Mb PI payloads, more or less, and if we had two files processing at the same time we used to have memory problems, stuck queues, a total hell. I think the best solution is to talk with your sender partner and to try to reduce the file size. All depends about your current PI memory and if you can have a de-central adapter engine, but files with 50Mb or more can be problematic on PI with huge load time.

Regards.

former_member186851
Active Contributor
0 Kudos

Hello Harsh,

Use chunk mode to reduce the file size and try reducing complex mapping functionalities to reduce time consumption.

Former Member
0 Kudos

Hi Harsh,

First of all I don't think PI would be able to process 200 MB files, we have seen issues at various stages like in the messaging system, pipeline steps etc when the file is around 100 MB itself without any mapping or lookups in between .

I would suggest you to split the source files and then think of any mapping.Also since you say this involves multiple lookups, it is not at all a good idea to even go with two 100MB files.As per the tuning guide from SAP, the ideal size of a message should be between 1MB and 5MB.

In versions starting from PI 7.31, there is a way to handle large files but it will be only a pass through, so even SAP though they say they support large files they also say routing, mapping etc will not be supported for such large files.

BR,

Harish