Fast way to push data from source IQ to target IQ?

[steffen_preissler]

Hello everyone,

is there a way to copy data from a source IQ to a target IQ (via push from source) in a fast way like it is "insert into ... location" (pull) from the target system? It should be a push from the source system, because of security/compliance reasons, the connection from target to source has to be avoided (which is needet to execute "insert into ... location" on the target system.

Unfortunately, insert data into proxy tables on the source system is way slower than executing "insert into ... location" on the target system.

Does anyone has an idea to accomplish that?

Thank you very much in advance,

Steffen

[markmumy · Accepted solution]

tayeb.hadjou's response is spot on. If the goal is to move data as fast as possible, that process will work. This still runs an insert location, but it is started from the source, so perhaps that will get around security issues? You just need to coordinate the logins on both sides so that things are mapped properly.

If not, then you would need to look at using an external ETL tool like SAP Data Services. Something that supports fast loading into IQ.

The only other way to "push", would be to extract the data to filesystem, then issue a LOAD TABLE USING CLIENT FILE. This is a client side load of IQ. It uses LOAD TABLE, which is fast, but the USING CLIENT FILE piece forces the file to be streamed from the client to IQ over a single network connection, so you lose some performance because of that. May not be much more of a performance loss than INSERT LOCATION. Both have to stream the data over a single, serial connection.

[steffen_preissler]

Hey guys.

thank you both for your fast responses! We already considered triggering the "insert into ... location" on the target system (test system) from the source (a production system) via "forward to". Due to security compliance reasons, an explicit connection from the target system (test) to the source system (production) via "create server" should to be avoided and so the "forward to" and "insert into ... location" on the target system wouldn't work.

But thank you for clarify, that the only other fast way of copying data would be ETL with support for IQ fast load and LOAD TABLE via file.

Thank you.

[tayeb_hadjou]

Without using proxy tables, you can (if security constrains allow) define remote IQ server (create server ...), connect into using "forward to ..." and execute the insert location.

Connect to <local_IQ>:

1. forward to <remote_IQ>
2. insert.... location, ....
3. forward to