* Stage the output to a table and use key generation transform. Taking this stage table as a source generate file.
* Lookup the previous flat file and get the latest row_id.
* Store the max(row_id) during each load to a temp table and refer it during next loads.
May be some other options, but these are the ones which came to my mind now.
If you are loading file to a permanent table then like you mentioned, use this table in key generation transform. Refer this link for more details.
Add comment