How to speed up Impex imports?

3601 Views

Hello

We need to import about one million records every 6 months and actually, the import takes 3 days on a 4 core system. We use a hot folder configuration to do this. Our import job uses the DefaultImportServer and we set legacy mode to TRUE (also tested with FALSE) and the validation mode RELAXED. The first 20000 records are fast (about 500 items/sec) but at the end, it is very slow (2 items/sec). With logging we found out, that there are many validations in background like checking the existence of the record first bevor importing (within the depths of the jalo-layer).

Before we import, we clean all neccessary tables and the records to import have no dependencies/relations and we do not need the any check!!! Is there any way to speed up the import? 3 days is abolutly not acceptable. The only way we see at the moment is to generate a SQL file and import the data directly into the DB (takes about 30 sec, wich is acceptable).

Any idears?