Avoiding duplicate files

[Former Member]

We need to restrict the duplicates from flowing into SAP for  the MICROS interface.

Currently if duplicate files flow into the folder  they are named with a counter in bracket 20150424-411000 (2) as per OS behavior.

Can we activate the duplicate check in PI so that duplicate files are not imported and also to restrict PI from performing the check only on the first 15 characters ?

[former_member182550 · Accepted solution]

When I have to restrict duplicate import files into SAP I always use one of the MD5_CALCULATE_HASH_FOR_* function modules which calculates a hash code for the first 64k (If I remember correctly) of data.

When the file is uploaded successfully I then store that hash value in a database table along with the file name, date, time etc etc.

So the flow is get the file data into an ITAB,  calculate the MD5 hash code,  read the database table of hash codes to see if there is a match,  if there is then issue a message and do not process the file,  otherwise process the file and add an entry to the database table of hash codes.

In that way you are not reliant on the filename being the same,  but rather the file content.

Regards

Rich

[Former Member · Accepted solution]

Duplicate files can be restricted by the Tick provided. but It stops the processing completely and doesnt process non-duplicate files too.Is there any solution to this?

[pvishnuvardan_reddy · Accepted solution]

Hi Vrushali,

I assume you are talking about the feature present in the file adapter when you say by the Tick Provided ..

Then can you try the below features in the channel for your requirement:

When ever there is any duplicate file has met, the faulty file will not get pick up by PI, and because of the archive configuration done for faulty file, PI archives it into the faulty archive directory, hence making way for other files to get process.

Regards

[Former Member · Accepted solution]

THank you so much.

That was really helpful

[maheswarareddykonda · Accepted solution]

Hi Vishnu,

thanks for explanation..

how its possible to store duplicate files in directory. so why we will get that chance to avoid duplicate files?

could you please explain.

BR,

Maheswarareddy

[pvishnuvardan_reddy · Accepted solution]

Hi Mahesh,

When we enable the duplicate file validation in the channel configuration, then during runtime, channel will validate the incoming file. If it meets the validation, then it becomes faulty which PI system doesn't process messages further which will affect the remaining files.

So, when you activate the archiving these type of faulty messages, it will make way for other messages to process smoothly.

Regards

[maheswarareddykonda · Accepted solution]

Thanks for reply Vishnu,

validating file means, name of the file or data which is there in file?

if name of the file: how same name files will store at same directory?

if data in file: can u pls elaborate?

[vijender_p]

Hi Vrushali ,

Yes !!! we can restrict the duplicate files, but can you elaborate the requirement .

For better requirement can you attach the screen shots.

Regards,

Vijender

[Former Member]

Duplicate files can be restricted by the Tick provided. but It stops the processing completely and doesnt process non-duplicate files too.Is there any solution to this?

[iaki_vila]

Hi Vrushali,

AFAIK there is not a standard way to introduce patterns in duplicate file handling.

If you have ESR development you can do file lookup to do this check, on a pass-trough scenario you would need a own module to do it.

Regards.

[Former Member]

Duplicate files can be restricted by the Tick provided. but It stops the processing completely and doesnt process non-duplicate files too.Is there any solution to this?