Incomplete File Processing

[Former Member]

Hi

I have a File 200MB. I can neither break it into chunks nor increase the poll interval, which has to be 60 seconds. How do I ensure that file is processed completely ? Msecs to wait would not work here as my scenario is FTP and that parameter works only for NFS.

What are the settings that I have to ensure so that the file would not be processed incompletely ?

[former_member187339 · Accepted solution]

Hi Midhun,

Since this is a FTP, the best solution will be to let the process which create the file to rename it to any other extension in the end or move to some other directory.

If this is not possible you may need to write an adapter module which will check the file size at every poll interval and if there is no increase then pick it else need to wait for next polling cycle

Regards

Suraj

[Former Member · Accepted solution]

Suraj,

I don't think I need to rename it. All I need is to make sure that the file would be processed completely. Do I have to make any specific setting in the FTP connection parameters.

[former_member187339 · Accepted solution]

Hi Midhun,

>>I don't think I need to rename it.

No it is not you who need to rename it. It is the process which creates this file. Suppose the format picked by PI is .dat then the process should create a file with extension .tmp and when the file is completely created it should rename it to .dat

>>Do I have to make any specific setting in the FTP connection parameters.

We don't have any such parameters for FTP

Regards

Suraj

[Former Member · Accepted solution]

Suppose the format picked by PI is .dat then the process should create a file with extension .tmp 

If the process create a file with the same extension, then you don't need to rename it, right ? I still don't get why the extension would be different when the size of file is huge

[former_member187339 · Accepted solution]

Hi Midhun,

Ok let me explain like this:

Supposing the file that PI picks currently is having extension as .dat. Now whenever PI sees a new file with extension .dat in the directory it will pick it up (no matter whether it is a completed file or a file which is just getting updated with the source process). So if the source creates a file with extension .tmp (and then keep updating it) and later when the file content is completely transferred (ie the complete file is created) then rename it from .tmp to .dat.

Like this PI will not be able to pick in the incomplete file. Another approach will be to keep creating the file in some other directory and when it is completely created then move it to the polling directory of PI. (for doing this a schedule batch program can be used)

Regards

Suraj

[Former Member · Accepted solution]

Thanks a lot, Suraj for your time

[Former Member]

Hi Midhun,

 I have a File 200MB. I can neither break it into chunks nor increase the poll interval

200 MB is quite big for PI,and processing it every 60 secs ... its will be a great performance hit for PI server.I guess even if you are allowed to increase the plloing time , this may lead to timeout again due to this huge size.

If there are no mappings required, Why not give a try to Managed file transfer(RMFT etc), this will save PI from the burden of huge files to be processed every 1 min as you mentioned.

Please refer this for analysing the performance which PI can handle.

/people/sravya.talanki2/blog/2005/11/29/night-mare-processing-huge-files-in-sap-xi

Regards,

srinivas

[prasanna_shanmugasundaram]

Hi,

You can try this one. See if your file adapter is configured to pick up a file with the name say abc.txt then what you can do is that you can put the file with the name Zabc.txt. Then you can rename the file back to abc.txt using unix shell script and oss command.

Thanks

Prasanna