Skip to Content
author's profile photo Former Member
Former Member

Table comparison

Hi,

I have a full refresh file with 1 million records. I should load it into a table initially and extract it in a file. From the second day I should get only the inserts and deletes. I used table comparison stage to achieve this. I tested my job with 3 lakh records and the table comparison takes approximately 10 minutes. Is there any way to improve the performance of this stage.

Note: I checked there is an option where we can set the comparison method as "sorted input" instead of row by row select. For this option, is it enough if my source file alone is sorted.

Add a comment
10|10000 characters needed characters exceeded

Assigned Tags

Related questions

2 Answers

  • author's profile photo Former Member
    Former Member
    Posted on Jan 13, 2016 at 05:37 AM

    The sorted input option will always give you the best performance and should always be used with the table comparison transform, where possible. The input query needs to be sorted on the primary key of the comparison table.

    Add a comment
    10|10000 characters needed characters exceeded

  • Posted on Jan 12, 2016 at 06:36 PM

    If only inserted/deleted needed , use joins rather than table comparison transform for better performance.

    Add a comment
    10|10000 characters needed characters exceeded

Before answering

You should only submit an answer when you are proposing a solution to the poster's problem. If you want the poster to clarify the question or provide more information, please leave a comment instead, requesting additional details. When answering, please include specifics, such as step-by-step instructions, context for the solution, and links to useful resources. Also, please make sure that you answer complies with our Rules of Engagement.
You must be Logged in to submit an answer.

Up to 10 attachments (including images) can be used with a maximum of 1.0 MB each and 10.5 MB total.