Skip to Content

Processing Large file SFTP to Proxy (mapping involved)

Dear All.

I've a scenario where i need to process 6 to 8 GB file from SFTP to Proxy scenario and the mapping is involved in this scenario.

can you please suggest the best way to process the file.

i can see https://blogs.sap.com/2015/04/03/sftp-adapter-handling-large-file/ the chunk mode processing is the way to handle the large file process, but here i do have mapping as well.

please suggest the best way to handle the large file for this scenario .

Regards

Ramesh.

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

5 Answers

  • Jun 05, 2017 at 11:42 AM

    Hi Ramesh,

    6 to 8 Gb is impossible to be treated in any programming language that i can think. I would talk with the sender system developer about the possibility of generating n files with less size.

    Regards.

    Add comment
    10|10000 characters needed characters exceeded

    • I would agree with Inaki on this - 6-8GB is way too big to process via mapping. Chunking is based purely on size in MB and what you will get is messages with truncated or partial records. The sender should provide files in reasonable processing sizes.

      Regards,

      Ryan Crosby

  • Jun 05, 2017 at 10:26 AM

    Ramesh,

    Is the mapping related to complete records in the file or is it individual to each record and not dependent .

    If the mapping is not dependent then you may use chunk mode and then do mapping.

    What is the source file if its xml file then chunk mode is not helpful. if its text/csv then u may use recordset per message and split it accordingly

    Br,

    Manoj

    Add comment
    10|10000 characters needed characters exceeded

    • Ramesh,

      As mentioned earlier Chunk mode breaks the file on size u have mentioned and not on number of records so there is always a risk of reading a partial record . Chunk mode is useful when yo have pick and drop mechanism only so i would suggest you not to go to chunk mode option.

      Did u try the recordset per message option ?

      Br,

      Manoj

  • Jun 06, 2017 at 07:05 AM

    Hi Ramesh,

    As it is file to proxy scenario , I will suggest to pass through scenario from PI to ECC and ask your Abap team to handle the PI mapping logic in proxy coding.

    Regards,

    Rudra

    Add comment
    10|10000 characters needed characters exceeded

  • Jun 06, 2017 at 07:05 AM

    Hi Ramesh,

    As it is file to proxy scenario , I will suggest to pass through scenario from PI to ECC and ask your Abap team to handle the PI mapping logic in proxy coding.

    Regards,

    Rudra

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    Jun 06, 2017 at 08:15 AM

    To be honest, I would drop the idea of handling the mapping in PI altogether unless you can align with the sender to split the data into manageable packages. Could it be an option to simply use PI as a transport technology, drop the file on your backend's application server and have you ABAP team write a report that will read the file n lines at a time and post the corresponding documents?

    Add comment
    10|10000 characters needed characters exceeded