Skip to Content
avatar image
Former Member

Slow insertion in Amazon Redshift

I'm testing the BO Data Services as ETL tool for data extraction having as Amazon Redshift destination. However, the performance for the inserts into Redshift is very bad.

On average, the Data Services inserts about 5 records per second. (I made tests with Talend and it already entered 22000 reg / sec)


We have a BODS running on the Linux platform x64.

For connection, we use an ODBC connection created and configured as Amazon recommendations.

http://docs.aws.amazon.com/redshift/latest/mgmt/install-odbc-driver-linux.html

Does anyone have any idea how I can provide more performance in data inserts in Redshift using bods?

Thanks

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

4 Answers

  • avatar image
    Former Member
    Oct 20, 2015 at 12:10 PM

    No ideias?

    Add comment
    10|10000 characters needed characters exceeded

  • Oct 20, 2015 at 02:51 PM

    I have no idea of your target Database but in general either of the following must help

    1) increasing number of rows per commit on your target table

    2) If your destination supports any bulk loading techniques, you can try using those in the target table bulk loader options.

    3) Please check the buffer size set in your target DB which is being loaded.

    Thanks

    Sameera.

    Add comment
    10|10000 characters needed characters exceeded

    • Hi Vitor,


      I have replied to your old post.

      Can you add the necessary jar file RedshiftJDBC41-1.1.7.1007.jar to the job server location and check for the jdbc adapter issue.

      ODBC driver for Redshift database may or may not work with DS 4.2.

      Regards

      Arun Sasi

  • avatar image
    Former Member
    Oct 20, 2015 at 07:00 PM

    What does your transformation do? Just dump loading on to target? I see a warning on your error screen. Also why did you chose ODBC? What is your DS version, i guess DS 4.2 onwards supports JDBC adapter.

    Add comment
    10|10000 characters needed characters exceeded

  • Oct 21, 2015 at 06:56 AM

    There's something in the error file. That message might give a hint. Can you post?

    Add comment
    10|10000 characters needed characters exceeded