Skip to Content
avatar image
Former Member

Queue stopped/ message timeout file greater than 50MB

Hi gurus

I have a scenario from SFTP - IDOC(ECC)

where every line of the file is an idoc data.... i am reading large files and breaking it in smaller files using multimapping and then mapping it to IDOC.

But this is creating issue in inbound queue of ECC saying queue stopped and sometimes message timeout.

Is there any setting in ICM i am missing or larger files in this is not possible.

really appreciate your help

Neha

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

9 Answers

  • Best Answer
    Jun 08, 2017 at 07:06 AM

    Hi Neha!

    Actually, design of this scenario depends on the business process taking place at the sender's side.

    Let's imagine that your source file is the some kind of cummulative file which is appended with data during any period of time and after the period ends PI should take that file and process it.

    In this case I would ask the sender party to modify the method of appending the source file so that instead of adding the signal line in the same file the separate file with the same name and different extension would be created. And in sender SFTP adapter I would make settings to check for additional file with given extension for source file and make that file required. If it exists - process starts with FCC splitting and further processing. If not - adapter generates exception which can be caught using CBMA, for example, and alert message is sent to recipients. When the signal file appears - processing starts.

    Regards, Evgeniy.

    Add comment
    10|10000 characters needed characters exceeded

  • Jun 07, 2017 at 07:51 AM

    Neha,

    Why read the complete file and break it in mapping ? You can use recordset permessage =1(or any n number) and read each single line as a separate message.

    Br,

    Manoj

    Add comment
    10|10000 characters needed characters exceeded

    • Yes Eve you are right didn't observe the below comment on EOD.

  • avatar image
    Former Member
    Jun 07, 2017 at 07:36 AM

    The file is .csv and we cant do pass thorugh as if there is a line ***EOD*** in the file then we need to break the file else we need to put the file in error and send an email. and if the EOD is there then only we break the file. I am thinking it like this... SFTP - SFTP (file breaking) - IDOC is this possible? Neha

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    Jun 07, 2017 at 07:55 AM

    HiManoj

    but i the receiver end it will be one big message right?

    error is XI error CLIENT_SEND_FAILED.Internal: Queue Stopped

    Add comment
    10|10000 characters needed characters exceeded

    • Former Member

      When you have properly defined your record structure and set the "Recordsets per message" parameter to an integer number n, a new message should be created after n occurrences of your record structure. In your case, you should see as many messages flowing to the receiver as your file has lines if the parameter is set to 1 and not one big message.

  • Jun 07, 2017 at 10:21 AM

    Hi Neha!

    First of all, I'd suggest to contact the sender party and ask if there is any possibility to decrease file size. They could perform splitting while creating that file.

    By the way, is the position of EOD line is fixed (first, last) or not?

    Regards, Evgeniy.

    Add comment
    10|10000 characters needed characters exceeded

    • Former Member

      I have a scenario where a 20mb file is read by sftp adapter and in the end of file there is a parameter that decides whether to split the file in small parts and put in process folder or put the file in error folder.

      file is csv aand i need to create a java mapping program

      is this possible

  • avatar image
    Former Member
    Jun 07, 2017 at 01:28 PM

    I have a scenario where a 20mb file is read by sftp adapter and in the end of file there is a parameter that decides whether to split the file in small parts and put in process folder or put the file in error folder.

    file is csv aand i need to create a java mapping program

    is this possible

    Add comment
    10|10000 characters needed characters exceeded

  • Jun 07, 2017 at 01:54 PM

    Hi Neha!

    In this case I would use java mapping just to read the last line of file and set the destination folder name in Dynamic configuration for receiver adapter. And in second separate scenario I would read the file from that folder, split it using recordsetPerMessage parameter in FCC and send result messages to target system performing transformation to IDoc structure in message mapping.

    Regards, Evgeniy.

    Add comment
    10|10000 characters needed characters exceeded

    • Former Member

      Mapping "http://SFTP_TO_SFTP/OM_SFTP_TO_SFTP" failed to execute: MappingException: Mapping failed in runtimeRuntime Exception when executing application mapping program com/*/CSVFileSplitUtil; Details: java.lang.NullPointerException; while trying to invoke the method com.sap.aii.mapping.api.AbstractTrace.addInfo(java.lang.String) of a null object loaded from field com.arteriatech.CSVFileSplitUtil.trace of an object loaded from local variable this, ApplicationRuntimeException: Runtime Exception when executing application mapping program com/*/CSVFileSplitUtil; Details: java.lang.NullPointerException; while trying to invoke the method com.sap.aii.mapping.api.AbstractTrace.addInfo(java.lang.String) of a null object loaded from field com.*.CSVFileSplitUtil.trace of an object loaded from local variable this, NullPointerException: while trying to invoke the method com.sap.aii.mapping.api.AbstractTrace.addInfo(java.lang.String) of a null object loaded from field com.arteriatech.CSVFileSplitUtil.trace of an object loaded from local variable this

      Transmitting the message to endpoint using connection File_http:// sap.com/xi/XI/System failed, due to: com.sap.aii.af.service.mapping.MappingException: Mapping failed in runtimeRuntime Exception when executing application mapping program com/*/CSVFileSplitUtil; Details: java.lang.NullPointerException; while trying to invoke the method com.sap.aii.mapping.api.AbstractTrace.addInfo(java.lang.String) of a null object loaded from field com.*.CSVFileSplitUtil.trace of an object loaded from local variable this

  • avatar image
    Former Member
    Jun 08, 2017 at 06:12 PM

    thnk you so much evegniy

    I will implement and let you know :)

    Add comment
    10|10000 characters needed characters exceeded

  • avatar image
    Former Member
    Jun 13, 2017 at 06:30 AM

    I wrote this code to read from directory split it and write it dynamically

    import java.io.BufferedReader;
    import java.io.File;
    import java.io.FileInputStream;
    import java.io.FileOutputStream;
    import java.io.IOException;
    import java.io.InputStream;
    import java.io.InputStreamReader;
    import java.io.OutputStream;
    import java.text.SimpleDateFormat;
    import java.util.Map;
    import com.sap.aii.mapping.api.AbstractTrace;
    import com.sap.aii.mapping.api.AbstractTransformation;
    import com.sap.aii.mapping.api.DynamicConfiguration;
    import com.sap.aii.mapping.api.DynamicConfigurationKey;
    import com.sap.aii.mapping.api.StreamTransformation;
    import com.sap.aii.mapping.api.StreamTransformationConstants;
    import com.sap.aii.mapping.api.StreamTransformationException;
    import com.sap.aii.mapping.api.TransformationInput;
    import com.sap.aii.mapping.api.TransformationOutput;
    //public class CSVFileSplitUtil implements StreamTransformation
    public class CSVFileSplitUtil extends AbstractTransformation
    {
    public void transform(TransformationInput transformationInput, TransformationOutput transformationOutput) throws StreamTransformationException
    {
    //trace.addInfo("Inside transform function");
    getTrace().addInfo("Inside transform function");
    String processedFilePath = "//*/PI share/E10587/processed";
    String errorFilePath = "//*/PI share/E10587/error";
    try
    {
    InputStream inputstream = transformationInput.getInputPayload().getInputStream();
    OutputStream outputstream = transformationOutput.getOutputPayload().getOutputStream();
    
    Map mapParameters = (Map) transformationInput.getInputHeader().getAll();
    mapParameters.put(DynamicConfigurationKey.create("http://sap.com/xi/XI/Dynamic",StreamTransformationConstants.DYNAMIC_CONFIGURATION), "");
    
    DynamicConfiguration conf = (DynamicConfiguration) mapParameters.get(StreamTransformationConstants.DYNAMIC_CONFIGURATION);
    DynamicConfigurationKey key = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/File", "FileName");
    DynamicConfiguration conf1 = (DynamicConfiguration) mapParameters.get(StreamTransformationConstants.DYNAMIC_CONFIGURATION);
    DynamicConfigurationKey key1 = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/File", "Directory");
    
    String filename=conf.get(key); 
    //trace.addInfo("filename: "+filename);
    getTrace().addInfo("filename: "+filename);
    String directoryName = conf.get(key1);
    getTrace().addInfo("directoryName: "+directoryName);
    
    BufferedReader br1 = new BufferedReader(new InputStreamReader(inputstream));
    String sLastLine="", lastLineTxt="";
    while ((sLastLine = br1.readLine()) != null)
    {
    lastLineTxt = sLastLine;
    }
    getTrace().addInfo("lastLineTxt: "+lastLineTxt);
    if(lastLineTxt.contains("Extraction_completed"))
    {
        String strFileName = filename+"_"+"Error"+".csv";
    conf.put(key,strFileName);
    conf1.put(key1,errorFilePath);
    
    byte[] b = new byte[inputstream.available()];
    inputstream.read(b);
    outputstream.write(b);
    }
    else
    {
    BufferedReader br = new BufferedReader(new InputStreamReader(inputstream));
    String sCurrentLine, lineData = "", payLoadData="", headerLine="";
    int i=0, lineCount=0;
    while ((sCurrentLine = br.readLine()) != null)
    {
    if(i == 0)
    {
    headerLine = sCurrentLine;
    }
    else
    {
    lineData = sCurrentLine;
    }
    lineCount = lineCount+1;
    if(lineCount <= 10000)
    {
    payLoadData = headerLine+"\n"+payLoadData+"\n"+lineData+"\n"+lastLineTxt+"\n";
    }
    if(lineCount == 10000)
    {
    i=i+1;
    }
        String strFileName = filename+"_"+"Part_"+i+".csv";
    conf.put(key,strFileName);
    conf1.put(key1,processedFilePath);
    
    byte[] b = payLoadData.getBytes();
    inputstream.read(b);
    outputstream.write(b);
    }
    }
    }
    catch(Exception e)
    {
    getTrace().addInfo("Exception: "+e.getLocalizedMessage());
    }
    }
    
    but i am getting following error
    
    Filesize is 0 Bytes 13.06.2017 
    
    MP: processing local module localejbs/ModuleProcessorExitBean13.06.2017 
    
    Message entered AF MP exit bean and will now be passed to the JCA adapter
    
    Trying to connect to host *.06.2017 
    
    Filename is *.*
    
    Server is *
    
    nHome directory is /
    
    Path is /*/PI share/E10587/
    
    Writing to file: *.*
    
    ErrorMP: exception caught with cause javax.resource.ResourceException: 4: Putting /*/PI share/E10587/*.* failed.
    
    
    i am putting *.* in filename and  ~/*/PI share/E10587/ in filepath
    anything i am missing
    Add comment
    10|10000 characters needed characters exceeded

    • Former Member Manoj K

      folder path is to be decided on runtime in dynamic configuration right

      so in the receiver channel what should i mention....

      filename *.*

      filepath */

      Filename is 1FULL_BTS_REPORT_DUMP_2017-05-17.csv_Error.csv Server is APRINS07 Home directory is / Path is //******/PI share/E10587/error/ Writing to file: 1FULL_BTS_REPORT_DUMP_2017-05-17.csv_Error.csv

      it is giving this in log but it is not creating the directory error physically here //******/PI share/E10587/error/