Skip to Content
0
Oct 14, 2020 at 03:58 PM

ADF - How to identify the number of partition needed, while using SAP source?

246 Views

Hi Team,

I'm using Azure Data Factory(ADF) to pull data from SAP table and it's having billions of records, i'm using CopyData activity with Partition option as "On Int" to retrieve the data faster in ADF. I'm passing "20000" as "Max partitions number", the problem is, if the data is available for only 10000 partitions, then for the remaining 10000 partitions, it's creating empty files, which is making more difficult when running other jobs.

  1. How to identify, what is the maximum number of partitions needed upfront for an SAP table data in ADF?

Thanks!