Data Hub removes items, if target system publication completes with errors

[Former Member]

Hello everyone,

in our development for a customer, we use the Data Hub for importing product data from an SAP ERP system to hybris (the version is 6.1.0.3 RC1). During the import process, an exception occurred in an special value translator, we've configured in Data Hub target item transformation. The result was, that the target item publication completes with errors (this is OK for us). Unfortunately, as a resuit of that, we found impexes in the /hybris/data/media/sys_master/impex/ folder which removes products from hybris! Is there an appropriate way to prevent the Data Hub from removing data in case of exceptions during the import?

Thanks in advance!

Greets, Robert

[former_member333910]

There was a bug, fixed in 6.1.0.5-RC2, where an item could be republished as DELETE in certain concurrent conditions.

https://jira.hybris.com/browse/ECP-1408

I can't say for sure if this is the exact same issue you are experiencing. However, it sounds similar and upgrading to the latest patch is always a good idea. Give it a try and let us know the result.

[Former Member]

Thanks for your reply. We'll try to update the data hub as soon as possible and see, if the problem is still there. But it will take about a week or two.

Our current approach is to override the DefaultPublicationActionHandler and drop these kind of deletion requests.

Greets, Robert

[Former Member]

We found out that there are TargetSystemPublications with publicationType == DELETE, but where did they come from?

[Slava]

That is result of deleting items either through the REST API or in a data loading (a set of data can be loaded with delete flag set, so datahub will delete those items from the target system). DataHub first pushes all items to be created and updated to the target system, then it replicates all deletions.

As Robin said the process is not atomic, so even if updates/creations fail, the deletions still may succeed.

[former_member224482]

There is no product removal process with the SAP ERP Integration.

Is this new code added ?

Keep in mind that Data Hub is not designed for transaction/process orchestration.

Each entities are processed in batch, parallel and independently. Therefore there the composition nor the publication is not an atomic operation.

[Former Member]

Hello Robin,

thanks for your reply.

We've managed to reproduce "the bug" locally, but we don't know the reason anyway. Isn't there a kind of retry-mechanism introduced with the akka-Framework?

Greets, Robert