on 07-28-2009 10:38 AM
Anyone has experience with using table comparison transforming for large volume data, e.g. 10M records.
How is the performance?
Any experience and suggestions is welcome.
Performance will depend on a lot of factors, but the two big ones are:
1. how many columns you are using in your compare (more columns will be slower)
2. how you do the compare (sorted input is WAY faster than row comparison)
Without knowing any details about your data or the process you are trying to implement it's difficult to make any additional recommendations. But keep in mind that there are other options - especially for really big data sets.
For example, you can always use a two step process that first deletes existing records matching the incoming set of primary keys, and then do a straight insert of the rows. This will avoid the whole comparison step - and avoid doing updates which are much slower than inserts.
However, this only works if you are replacing the existing records. It wouldn't work if your table compare is part of a type 2 dimension load or something that requires you to track history.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
85 | |
23 | |
11 | |
9 | |
8 | |
5 | |
5 | |
5 | |
5 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.