Queue stopped/ message timeout file greater than 50MB

[Former Member]

Hi gurus

I have a scenario from SFTP - IDOC(ECC)

where every line of the file is an idoc data.... i am reading large files and breaking it in smaller files using multimapping and then mapping it to IDOC.

But this is creating issue in inbound queue of ECC saying queue stopped and sometimes message timeout.

Is there any setting in ICM i am missing or larger files in this is not possible.

really appreciate your help

Neha

[former_member190293 · Accepted solution]

Hi Neha!

Actually, design of this scenario depends on the business process taking place at the sender's side.

Let's imagine that your source file is the some kind of cummulative file which is appended with data during any period of time and after the period ends PI should take that file and process it.

In this case I would ask the sender party to modify the method of appending the source file so that instead of adding the signal line in the same file the separate file with the same name and different extension would be created. And in sender SFTP adapter I would make settings to check for additional file with given extension for source file and make that file required. If it exists - process starts with FCC splitting and further processing. If not - adapter generates exception which can be caught using CBMA, for example, and alert message is sent to recipients. When the signal file appears - processing starts.

Regards, Evgeniy.

[Former Member · Accepted solution]

what we are trying to do is

make a java mapping read last line of file and then using dynamic configuration put the files in two folders based on last line of csv

dont know how dynamic configuration will work here

can you please tell or give me some links

many thanks

[former_member190293 · Accepted solution]

Hi Neha!

https://wiki.scn.sap.com/wiki/display/XI/Dynamic+file+name+and+directory+in+Receiver+File+Adapter+-+...

For setting target folder using ASMA you use parameter "Directory" with namespace "http://sap.com/xi/XI/System/File".

Regards, Evgeniy.

[manoj_khavatkopp]

Neha,

Why read the complete file and break it in mapping ? You can use recordset permessage =1(or any n number) and read each single line as a separate message.

Br,

Manoj

[former_member190293]

Hi Manoj!

I guess it's because of "EOD" line that should be checked for existance before processing.

Regards, Evgeniy.

[manoj_khavatkopp]

Yes Eve you are right didn't observe the below comment on EOD.

[Former Member]

I wrote this code to read from directory split it and write it dynamically

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.text.SimpleDateFormat;
import java.util.Map;
import com.sap.aii.mapping.api.AbstractTrace;
import com.sap.aii.mapping.api.AbstractTransformation;
import com.sap.aii.mapping.api.DynamicConfiguration;
import com.sap.aii.mapping.api.DynamicConfigurationKey;
import com.sap.aii.mapping.api.StreamTransformation;
import com.sap.aii.mapping.api.StreamTransformationConstants;
import com.sap.aii.mapping.api.StreamTransformationException;
import com.sap.aii.mapping.api.TransformationInput;
import com.sap.aii.mapping.api.TransformationOutput;
//public class CSVFileSplitUtil implements StreamTransformation
public class CSVFileSplitUtil extends AbstractTransformation
{

public void transform(TransformationInput transformationInput, TransformationOutput transformationOutput) throws StreamTransformationException
{
//trace.addInfo("Inside transform function");
getTrace().addInfo("Inside transform function");
String processedFilePath = "//*/PI share/E10587/processed";
String errorFilePath = "//*/PI share/E10587/error";
try
{
InputStream inputstream = transformationInput.getInputPayload().getInputStream();
OutputStream outputstream = transformationOutput.getOutputPayload().getOutputStream();

Map mapParameters = (Map) transformationInput.getInputHeader().getAll();
mapParameters.put(DynamicConfigurationKey.create("http://sap.com/xi/XI/Dynamic",StreamTransformationConstants.DYNAMIC_CONFIGURATION), "");

DynamicConfiguration conf = (DynamicConfiguration) mapParameters.get(StreamTransformationConstants.DYNAMIC_CONFIGURATION);
DynamicConfigurationKey key = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/File", "FileName");
DynamicConfiguration conf1 = (DynamicConfiguration) mapParameters.get(StreamTransformationConstants.DYNAMIC_CONFIGURATION);
DynamicConfigurationKey key1 = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/File", "Directory");

String filename=conf.get(key); 
//trace.addInfo("filename: "+filename);
getTrace().addInfo("filename: "+filename);
String directoryName = conf.get(key1);
getTrace().addInfo("directoryName: "+directoryName);

BufferedReader br1 = new BufferedReader(new InputStreamReader(inputstream));
String sLastLine="", lastLineTxt="";
while ((sLastLine = br1.readLine()) != null)
{
lastLineTxt = sLastLine;
}
getTrace().addInfo("lastLineTxt: "+lastLineTxt);
if(lastLineTxt.contains("Extraction_completed"))
{
    String strFileName = filename+"_"+"Error"+".csv";
conf.put(key,strFileName);
conf1.put(key1,errorFilePath);

byte[] b = new byte[inputstream.available()];
inputstream.read(b);
outputstream.write(b);
}
else
{
BufferedReader br = new BufferedReader(new InputStreamReader(inputstream));
String sCurrentLine, lineData = "", payLoadData="", headerLine="";
int i=0, lineCount=0;
while ((sCurrentLine = br.readLine()) != null)
{
if(i == 0)
{
headerLine = sCurrentLine;
}
else
{
lineData = sCurrentLine;
}
lineCount = lineCount+1;
if(lineCount <= 10000)
{
payLoadData = headerLine+"\n"+payLoadData+"\n"+lineData+"\n"+lastLineTxt+"\n";
}
if(lineCount == 10000)
{
i=i+1;
}
    String strFileName = filename+"_"+"Part_"+i+".csv";
conf.put(key,strFileName);
conf1.put(key1,processedFilePath);

byte[] b = payLoadData.getBytes();
inputstream.read(b);
outputstream.write(b);
}
}
}
catch(Exception e)
{
getTrace().addInfo("Exception: "+e.getLocalizedMessage());
}
}

but i am getting following error

Filesize is 0 Bytes 13.06.2017 

MP: processing local module localejbs/ModuleProcessorExitBean13.06.2017 

Message entered AF MP exit bean and will now be passed to the JCA adapter

Trying to connect to host *.06.2017 

Filename is *.*

Server is *

nHome directory is /

Path is /*/PI share/E10587/

Writing to file: *.*

ErrorMP: exception caught with cause javax.resource.ResourceException: 4: Putting /*/PI share/E10587/*.* failed.


i am putting *.* in filename and  ~/*/PI share/E10587/ in filepath
anything i am missing

[manoj_khavatkopp]

Have u enabled ASMA in your reciever channel for Filename and directory?

You can't have * in Filepath or Filename which u have defined in Code.

[Former Member]

i have enabled both

where cant i have *

in CC configuration??

then what exactly i mention there

[Former Member]

got the issue manoj

i was not putting file in http://sap.com/xi/XI/System/File for SFTP

but another thing is it is creating a folder and writing the file in message log but i cannot see the folder physically

i.e. folder name //*/PI share/E10587/error or processed not getting created on the specified location //*/PI share/E10587/

[manoj_khavatkopp]

Neha,

As mentioned earlier don't mention * in your filepath , just mention your actual folder path .

Br,

Manoj

[Former Member]

folder path is to be decided on runtime in dynamic configuration right

so in the receiver channel what should i mention....

filename *.*

filepath */

Filename is 1FULL_BTS_REPORT_DUMP_2017-05-17.csv_Error.csv Server is APRINS07 Home directory is / Path is //******/PI share/E10587/error/ Writing to file: 1FULL_BTS_REPORT_DUMP_2017-05-17.csv_Error.csv

it is giving this in log but it is not creating the directory error physically here //******/PI share/E10587/error/

[Former Member]

thnk you so much evegniy

I will implement and let you know 🙂

[former_member190293]

Hi Neha!

In this case I would use java mapping just to read the last line of file and set the destination folder name in Dynamic configuration for receiver adapter. And in second separate scenario I would read the file from that folder, split it using recordsetPerMessage parameter in FCC and send result messages to target system performing transformation to IDoc structure in message mapping.

Regards, Evgeniy.

[Former Member]

I am new to java mapping

a few things that I wanted to ask is it is a sftp to idoc csv file scenario so I will pick file without fcc and then will use java mapping to read the file and based on last line I will either split the file and put in one folder or put it in error folder

am I right with my approach

where is this java mapping put...inside OM or someplace else... I know its a silly question but I don't understand where to put java mapping that reads this csv and put it in folders

[Former Member]

also if sftp doesn't use FCC and send as is then it will be a csv file that java mapping will split and not xml file right ??

and the output after the java mapping will again be multiple csv files or xml files??

[former_member190293]

Hi Neha!

You can use java mapping only within operation mapping and nowhere else.

In first ICo you read the last line of your file and set target folder for your receiver adapter. Source file is transferred unchanged.

Further you create two more ICos: one for correct files and another - for incorrect. The first one takes the file that was put to "correct" folder by previous ICo, converts it to set of XML messages using FCC and recordsetPerMessage parameter and sends the result to target system with xml to IDocs transformation. Second ICo is used for sending mail alerts about incorrect files, that were placed to "incorrect" folder.

Regards, Evgeniy.

[Former Member]

and what if in one OM i read end of file..

if correct then split and put in one folder

if incorrect then put in another and email

is this not feasible

also if you have some code snippet for file splitting/ writing to a directory and emailing using java...that will be very helpful

[former_member190293]

The main idea is to split the source file into smaller parts before sending it to receiver to avoid big resources consumption during mapping step. Since you perform splitting using FCC, you need to know if your file is correct before doing that. And using the first ICo you check if the file needs to be splitted or not and put it to one or another folder depending on check result.

And in second ICo you can use FCC in sender channel to split the file into parts of appropriate size and send it to receiver.

In described scenario you need rather simple java code just to read the last line of your input payload and write value for ASMA parameter to Dynamic Configuration. Further transformations like XML to IDocs could be done in following ICos using graphical mapping.

Regards, Evgeniy.

[Former Member]

Evgeniy

the code mentioned below is not producing correct output

i wanted to read "Extraction_completed" and split the file else put the file as it is in error folder

but it is not happening

can you please see the code and tell me what i am doing wrong

[Former Member]

Mapping "http://SFTP_TO_SFTP/OM_SFTP_TO_SFTP" failed to execute: MappingException: Mapping failed in runtimeRuntime Exception when executing application mapping program com/*/CSVFileSplitUtil; Details: java.lang.NullPointerException; while trying to invoke the method com.sap.aii.mapping.api.AbstractTrace.addInfo(java.lang.String) of a null object loaded from field com.arteriatech.CSVFileSplitUtil.trace of an object loaded from local variable this, ApplicationRuntimeException: Runtime Exception when executing application mapping program com/*/CSVFileSplitUtil; Details: java.lang.NullPointerException; while trying to invoke the method com.sap.aii.mapping.api.AbstractTrace.addInfo(java.lang.String) of a null object loaded from field com.*.CSVFileSplitUtil.trace of an object loaded from local variable this, NullPointerException: while trying to invoke the method com.sap.aii.mapping.api.AbstractTrace.addInfo(java.lang.String) of a null object loaded from field com.arteriatech.CSVFileSplitUtil.trace of an object loaded from local variable this

Transmitting the message to endpoint <local> using connection File_http://sap.com/xi/XI/System failed, due to: com.sap.aii.af.service.mapping.MappingException: Mapping failed in runtimeRuntime Exception when executing application mapping program com/*/CSVFileSplitUtil; Details: java.lang.NullPointerException; while trying to invoke the method com.sap.aii.mapping.api.AbstractTrace.addInfo(java.lang.String) of a null object loaded from field com.*.CSVFileSplitUtil.trace of an object loaded from local variable this

[Former Member]

I have a scenario where a 20mb file is read by sftp adapter and in the end of file there is a parameter that decides whether to split the file in small parts and put in process folder or put the file in error folder.

file is csv aand i need to create a java mapping program

is this possible

[former_member190293]

Hi Neha!

First of all, I'd suggest to contact the sender party and ask if there is any possibility to decrease file size. They could perform splitting while creating that file.

By the way, is the position of EOD line is fixed (first, last) or not?

Regards, Evgeniy.

[Former Member]

Evgeniy

its the end line of file separated by a comma

[Former Member]

I have a scenario where a 20mb file is read by sftp adapter and in the end of file there is a parameter that decides whether to split the file in small parts and put in process folder or put the file in error folder.

file is csv aand i need to create a java mapping program

is this possible

[Former Member]

HiManoj

but i the receiver end it will be one big message right?

error is XI error CLIENT_SEND_FAILED.Internal: Queue Stopped

[weberpat]

When you have properly defined your record structure and set the "Recordsets per message" parameter to an integer number n, a new message should be created after n occurrences of your record structure. In your case, you should see as many messages flowing to the receiver as your file has lines if the parameter is set to 1 and not one big message.

[Former Member]

The file is .csv and we cant do pass thorugh as if there is a line ***EOD*** in the file then we need to break the file else we need to put the file in error and send an email. and if the EOD is there then only we break the file. I am thinking it like this... SFTP - SFTP (file breaking) - IDOC is this possible? Neha