Skip to Content

Selective deletion of data from HANA table SDI

Hi,

I'm using HANA SDI flowgraphs to load data from 3rd party DB's to HANA. In the 1st data replication, I will do the full load and from the next load onwards, I need to fetch only the changed records in the last 10 days, delete corresponding records from the HANA table first and then load these 10 days changes again to the same HANA table. What is the right way of handling this situation?

Thanks.

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

2 Answers

  • May 08 at 02:53 PM

    Hi Hari,

    What you are asking is probably possible, you would have a full initial load job and then a separate '10 day delta' job that does not truncate the target table, and uses an UPSERT writer type. This would require the source table to have a column that records its last changed timestamp.

    Is there some reason you don't just implement a real-time replication? This would continuously update the target table whenever changes are made in the source db, and is quite easy to implement. Less manual SQL, simple to maintain and monitor.

    Depending on the source database types, there should be an adapter that can accomplish the real-time loads.

    Regards,

    Jeff K

    Add comment
    10|10000 characters needed characters exceeded

  • May 08 at 03:14 PM

    Jeff,

    Appreciate your quick reply. We are using SQL Adapter and we cant use Realtime replication due to some known reasons. If I use UPSERT by filtering the flow graph to pull only last 10 days, it works fine with updates and inserts. But I don't think it takes care of DELETES in the source DB.If a record gets deleted in the source DB in the last 10 days, I want this records to be deleted in my HANA table if it has been loaded already.

    Thanks.

    Add comment
    10|10000 characters needed characters exceeded