We are working on a use case to read zip files from s3 and unzip , parse xml files and ingest data into HANA. We used a custom python operator to unzip files and used "ListFile" standard operator to process files from unzipped folder. The average file count in a zip file comes around 700 and most of the cases its above 1K.
We found an issue after processing 1000 files and "Listfiles" not working/processing after 1000 files. Could you please advice any solution on this as we need to process multiple zip files in sequence.