Skip to Content

Reducing PI Map Memory Requiement via Premap

Hi All,

We are trying to map large 200+ MB XML files using PI 7.11.

The target is a flat CSV. The maps are quite complex with multiple lookups.

Since we only use a small subset of the xml fields in the source file, I'm wondering if the following "PreMap" would help reduce the memory requirements.

Source XML(200MB+) -> PreMap-> Source XML*(100MB+)

Source XML*(100MB+) -> MainMap ->Target CSV(10MB)

How does this compare with:

Source XML(200MB+) -> MainMap ->Target CSV(10MB)


Note:

1) Both maps are MessageMappings.

2) PreMap is a simple map that discards all XML fields that MAP2 doesn't need, but keeps the same XML structure.

3) MainMap is a complex map that will have mulitple lookups.

Is there any benefit to a PreMap that filters out fields that are not required by the main map? Or maybe it makes things worse?

I'm hoping to optimize memory consumption.

Thanks,

Harsh

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

3 Answers

  • avatar image
    Former Member
    Oct 27, 2015 at 03:03 AM

    Hi Harsh,

    First of all I don't think PI would be able to process 200 MB files, we have seen issues at various stages like in the messaging system, pipeline steps etc when the file is around 100 MB itself without any mapping or lookups in between .

    I would suggest you to split the source files and then think of any mapping.Also since you say this involves multiple lookups, it is not at all a good idea to even go with two 100MB files.As per the tuning guide from SAP, the ideal size of a message should be between 1MB and 5MB.

    In versions starting from PI 7.31, there is a way to handle large files but it will be only a pass through, so even SAP though they say they support large files they also say routing, mapping etc will not be supported for such large files.

    BR,

    Harish

    Add comment
    10|10000 characters needed characters exceeded

  • Oct 27, 2015 at 04:14 AM

    Hello Harsh,

    Use chunk mode to reduce the file size and try reducing complex mapping functionalities to reduce time consumption.

    Add comment
    10|10000 characters needed characters exceeded

  • Oct 27, 2015 at 07:05 AM

    Hi Harsh,

    I think two maps is a worse option.. We have a scenario how worked with files around 100Mb, i've notice that they converted themselves on 200Mb PI payloads, more or less, and if we had two files processing at the same time we used to have memory problems, stuck queues, a total hell. I think the best solution is to talk with your sender partner and to try to reduce the file size. All depends about your current PI memory and if you can have a de-central adapter engine, but files with 50Mb or more can be problematic on PI with huge load time.

    Regards.

    Add comment
    10|10000 characters needed characters exceeded