Hi,
I have multiple input files in a SFTP folder and need to be zipped into 1 zip file. The files in the SFTP folder could reach 20k daily.
I have done this successfully with:
Looping Process Call -> Enrich Content with SFTP (combine each content as it is in XML) -> Gather & Split the file again to split each content to a separate file and gather using ZIP strategy.
It's working successfully and iflow will be triggered via a timer that is scheduled to run every 5 mins. But the problem is it runs very slow when there are thousands of files exist in the SFTP folder. For example, when there are 4k files and I set the loop to max process 100 files it runs for 20 mins, and this causes the files in the SFTP to never be processed all since the rate of new files generated is faster than the rate of the files processed and archived.
Last time, I try to change the max loop to process 2k files, but it runs for 3hours+ and in the end there is a "Error occurred during aggregation - No space left on device".

(Total files should be around 13mb)
Is there a better way to do zip multiple files into one with a good performance? Or am I doing the Looping process call/Enrich content/split & gather process wrongly?
Thank you.