on 06-07-2017 8:23 AM
Hi gurus
I have a scenario from SFTP - IDOC(ECC)
where every line of the file is an idoc data.... i am reading large files and breaking it in smaller files using multimapping and then mapping it to IDOC.
But this is creating issue in inbound queue of ECC saying queue stopped and sometimes message timeout.
Is there any setting in ICM i am missing or larger files in this is not possible.
really appreciate your help
Neha
Hi Neha!
Actually, design of this scenario depends on the business process taking place at the sender's side.
Let's imagine that your source file is the some kind of cummulative file which is appended with data during any period of time and after the period ends PI should take that file and process it.
In this case I would ask the sender party to modify the method of appending the source file so that instead of adding the signal line in the same file the separate file with the same name and different extension would be created. And in sender SFTP adapter I would make settings to check for additional file with given extension for source file and make that file required. If it exists - process starts with FCC splitting and further processing. If not - adapter generates exception which can be caught using CBMA, for example, and alert message is sent to recipients. When the signal file appears - processing starts.
Regards, Evgeniy.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Neha!
For setting target folder using ASMA you use parameter "Directory" with namespace "http://sap.com/xi/XI/System/File".
Regards, Evgeniy.
Neha,
Why read the complete file and break it in mapping ? You can use recordset permessage =1(or any n number) and read each single line as a separate message.
Br,
Manoj
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Manoj!
I guess it's because of "EOD" line that should be checked for existance before processing.
Regards, Evgeniy.
Yes Eve you are right didn't observe the below comment on EOD.
I wrote this code to read from directory split it and write it dynamically
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.text.SimpleDateFormat;
import java.util.Map;
import com.sap.aii.mapping.api.AbstractTrace;
import com.sap.aii.mapping.api.AbstractTransformation;
import com.sap.aii.mapping.api.DynamicConfiguration;
import com.sap.aii.mapping.api.DynamicConfigurationKey;
import com.sap.aii.mapping.api.StreamTransformation;
import com.sap.aii.mapping.api.StreamTransformationConstants;
import com.sap.aii.mapping.api.StreamTransformationException;
import com.sap.aii.mapping.api.TransformationInput;
import com.sap.aii.mapping.api.TransformationOutput;
//public class CSVFileSplitUtil implements StreamTransformation
public class CSVFileSplitUtil extends AbstractTransformation
{
public void transform(TransformationInput transformationInput, TransformationOutput transformationOutput) throws StreamTransformationException
{
//trace.addInfo("Inside transform function");
getTrace().addInfo("Inside transform function");
String processedFilePath = "//*/PI share/E10587/processed";
String errorFilePath = "//*/PI share/E10587/error";
try
{
InputStream inputstream = transformationInput.getInputPayload().getInputStream();
OutputStream outputstream = transformationOutput.getOutputPayload().getOutputStream();
Map mapParameters = (Map) transformationInput.getInputHeader().getAll();
mapParameters.put(DynamicConfigurationKey.create("http://sap.com/xi/XI/Dynamic",StreamTransformationConstants.DYNAMIC_CONFIGURATION), "");
DynamicConfiguration conf = (DynamicConfiguration) mapParameters.get(StreamTransformationConstants.DYNAMIC_CONFIGURATION);
DynamicConfigurationKey key = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/File", "FileName");
DynamicConfiguration conf1 = (DynamicConfiguration) mapParameters.get(StreamTransformationConstants.DYNAMIC_CONFIGURATION);
DynamicConfigurationKey key1 = DynamicConfigurationKey.create("http://sap.com/xi/XI/System/File", "Directory");
String filename=conf.get(key);
//trace.addInfo("filename: "+filename);
getTrace().addInfo("filename: "+filename);
String directoryName = conf.get(key1);
getTrace().addInfo("directoryName: "+directoryName);
BufferedReader br1 = new BufferedReader(new InputStreamReader(inputstream));
String sLastLine="", lastLineTxt="";
while ((sLastLine = br1.readLine()) != null)
{
lastLineTxt = sLastLine;
}
getTrace().addInfo("lastLineTxt: "+lastLineTxt);
if(lastLineTxt.contains("Extraction_completed"))
{
String strFileName = filename+"_"+"Error"+".csv";
conf.put(key,strFileName);
conf1.put(key1,errorFilePath);
byte[] b = new byte[inputstream.available()];
inputstream.read(b);
outputstream.write(b);
}
else
{
BufferedReader br = new BufferedReader(new InputStreamReader(inputstream));
String sCurrentLine, lineData = "", payLoadData="", headerLine="";
int i=0, lineCount=0;
while ((sCurrentLine = br.readLine()) != null)
{
if(i == 0)
{
headerLine = sCurrentLine;
}
else
{
lineData = sCurrentLine;
}
lineCount = lineCount+1;
if(lineCount <= 10000)
{
payLoadData = headerLine+"\n"+payLoadData+"\n"+lineData+"\n"+lastLineTxt+"\n";
}
if(lineCount == 10000)
{
i=i+1;
}
String strFileName = filename+"_"+"Part_"+i+".csv";
conf.put(key,strFileName);
conf1.put(key1,processedFilePath);
byte[] b = payLoadData.getBytes();
inputstream.read(b);
outputstream.write(b);
}
}
}
catch(Exception e)
{
getTrace().addInfo("Exception: "+e.getLocalizedMessage());
}
}
but i am getting following error
Filesize is 0 Bytes 13.06.2017
MP: processing local module localejbs/ModuleProcessorExitBean13.06.2017
Message entered AF MP exit bean and will now be passed to the JCA adapter
Trying to connect to host *.06.2017
Filename is *.*
Server is *
nHome directory is /
Path is /*/PI share/E10587/
Writing to file: *.*
ErrorMP: exception caught with cause javax.resource.ResourceException: 4: Putting /*/PI share/E10587/*.* failed.
i am putting *.* in filename and ~/*/PI share/E10587/ in filepath
anything i am missing
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
got the issue manoj
i was not putting file in http://sap.com/xi/XI/System/File for SFTP
but another thing is it is creating a folder and writing the file in message log but i cannot see the folder physically
i.e. folder name //*/PI share/E10587/error or processed not getting created on the specified location //*/PI share/E10587/
folder path is to be decided on runtime in dynamic configuration right
so in the receiver channel what should i mention....
filename *.*
filepath */
Filename is 1FULL_BTS_REPORT_DUMP_2017-05-17.csv_Error.csv Server is APRINS07 Home directory is / Path is //******/PI share/E10587/error/ Writing to file: 1FULL_BTS_REPORT_DUMP_2017-05-17.csv_Error.csvit is giving this in log but it is not creating the directory error physically here //******/PI share/E10587/error/
thnk you so much evegniy
I will implement and let you know 🙂
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Neha!
In this case I would use java mapping just to read the last line of file and set the destination folder name in Dynamic configuration for receiver adapter. And in second separate scenario I would read the file from that folder, split it using recordsetPerMessage parameter in FCC and send result messages to target system performing transformation to IDoc structure in message mapping.
Regards, Evgeniy.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I am new to java mapping
a few things that I wanted to ask is it is a sftp to idoc csv file scenario so I will pick file without fcc and then will use java mapping to read the file and based on last line I will either split the file and put in one folder or put it in error folder
am I right with my approach
where is this java mapping put...inside OM or someplace else... I know its a silly question but I don't understand where to put java mapping that reads this csv and put it in folders
Hi Neha!
You can use java mapping only within operation mapping and nowhere else.
In first ICo you read the last line of your file and set target folder for your receiver adapter. Source file is transferred unchanged.
Further you create two more ICos: one for correct files and another - for incorrect. The first one takes the file that was put to "correct" folder by previous ICo, converts it to set of XML messages using FCC and recordsetPerMessage parameter and sends the result to target system with xml to IDocs transformation. Second ICo is used for sending mail alerts about incorrect files, that were placed to "incorrect" folder.
Regards, Evgeniy.
The main idea is to split the source file into smaller parts before sending it to receiver to avoid big resources consumption during mapping step. Since you perform splitting using FCC, you need to know if your file is correct before doing that. And using the first ICo you check if the file needs to be splitted or not and put it to one or another folder depending on check result.
And in second ICo you can use FCC in sender channel to split the file into parts of appropriate size and send it to receiver.
In described scenario you need rather simple java code just to read the last line of your input payload and write value for ASMA parameter to Dynamic Configuration. Further transformations like XML to IDocs could be done in following ICos using graphical mapping.
Regards, Evgeniy.
the code mentioned below is not producing correct output
i wanted to read "Extraction_completed" and split the file else put the file as it is in error folder
but it is not happening
can you please see the code and tell me what i am doing wrong
Mapping "http://SFTP_TO_SFTP/OM_SFTP_TO_SFTP" failed to execute: MappingException: Mapping failed in runtimeRuntime Exception when executing application mapping program com/*/CSVFileSplitUtil; Details: java.lang.NullPointerException; while trying to invoke the method com.sap.aii.mapping.api.AbstractTrace.addInfo(java.lang.String) of a null object loaded from field com.arteriatech.CSVFileSplitUtil.trace of an object loaded from local variable this, ApplicationRuntimeException: Runtime Exception when executing application mapping program com/*/CSVFileSplitUtil; Details: java.lang.NullPointerException; while trying to invoke the method com.sap.aii.mapping.api.AbstractTrace.addInfo(java.lang.String) of a null object loaded from field com.*.CSVFileSplitUtil.trace of an object loaded from local variable this, NullPointerException: while trying to invoke the method com.sap.aii.mapping.api.AbstractTrace.addInfo(java.lang.String) of a null object loaded from field com.arteriatech.CSVFileSplitUtil.trace of an object loaded from local variable this
Transmitting the message to endpoint <local> using connection File_http://sap.com/xi/XI/System failed, due to: com.sap.aii.af.service.mapping.MappingException: Mapping failed in runtimeRuntime Exception when executing application mapping program com/*/CSVFileSplitUtil; Details: java.lang.NullPointerException; while trying to invoke the method com.sap.aii.mapping.api.AbstractTrace.addInfo(java.lang.String) of a null object loaded from field com.*.CSVFileSplitUtil.trace of an object loaded from local variable this
I have a scenario where a 20mb file is read by sftp adapter and in the end of file there is a parameter that decides whether to split the file in small parts and put in process folder or put the file in error folder.
file is csv aand i need to create a java mapping program
is this possible
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Neha!
First of all, I'd suggest to contact the sender party and ask if there is any possibility to decrease file size. They could perform splitting while creating that file.
By the way, is the position of EOD line is fixed (first, last) or not?
Regards, Evgeniy.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
its the end line of file separated by a comma
HiManoj
but i the receiver end it will be one big message right?
error is XI error CLIENT_SEND_FAILED.Internal: Queue Stopped
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
When you have properly defined your record structure and set the "Recordsets per message" parameter to an integer number n, a new message should be created after n occurrences of your record structure. In your case, you should see as many messages flowing to the receiver as your file has lines if the parameter is set to 1 and not one big message.
The file is .csv and we cant do pass thorugh as if there is a line ***EOD*** in the file then we need to break the file else we need to put the file in error and send an email. and if the EOD is there then only we break the file. I am thinking it like this... SFTP - SFTP (file breaking) - IDOC is this possible? Neha
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
68 | |
26 | |
9 | |
9 | |
7 | |
6 | |
4 | |
4 | |
4 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.