ADF - How to identify the number of partition needed, while using SAP source?

[former_member1319961]

Hi Team,

I'm using Azure Data Factory(ADF) to pull data from SAP table and it's having billions of records, i'm using CopyData activity with Partition option as "On Int" to retrieve the data faster in ADF. I'm passing "20000" as "Max partitions number", the problem is, if the data is available for only 10000 partitions, then for the remaining 10000 partitions, it's creating empty files, which is making more difficult when running other jobs.

1. How to identify, what is the maximum number of partitions needed upfront for an SAP table data in ADF?

Thanks!

[former_member30]

Hi and welcome to the SAP Community!

Thank you for visiting SAP Community to get answers to your questions. Since you're asking a question here for the first time, I recommend that you familiarize yourself with https://community.sap.com/resources/questions-and-answers (if you haven't already), as it provides tips for preparing questions that draw responses from our members. For example, you can outline what steps you took to find answers (and why they weren't helpful). The more details you provide, the more likely it is that members will be able to assist you. Should you wish, you can revise your question by selecting Actions, then Edit (although once someone answers your question, you'll lose the ability to edit the question -- but if that happens, you can leave more details in a comment). Finally, if you're hoping to connect with readers, please consider adding a picture to your profile. Here's how you do it: https://www.youtube.com/watch?v=F5JdUbyjfMA&list=PLpQebylHrdh5s3gwy-h6RtymfDpoz3vDS. By personalizing your profile with a photo of you, you encourage readers to respond.

Cheers,

Julia SAP Community Moderator