Skip to Content

DI XIR2 Limitation on reading source file

Are there any limitations on the number of source flat files that a data flow can read or the max file size that a dataflow can read? I know a dataflow can read multiple files using a wildcard... but is there a limit to the number of files that can be read. We're expecting to receive files that when combined will have a size of 15G+ where each individual file is ~10MB each.

Edited by: Chris Sam on Dec 18, 2008 10:25 PM

Add a comment
10|10000 characters needed characters exceeded

Assigned Tags

Related questions

1 Answer

  • Best Answer
    author's profile photo Former Member
    Former Member
    Posted on Dec 18, 2008 at 09:38 PM

    No, there's no limit on number of files read via the wildcard or sizes of the file.

    Of course it also depends on what you want to do with the data. If you try to load this 15 GB in memory, a lot of paging will occur and your job might get extremely slow. But for regular (well designed) data flows, the data will just stream through the engine.

    Add a comment
    10|10000 characters needed characters exceeded

Before answering

You should only submit an answer when you are proposing a solution to the poster's problem. If you want the poster to clarify the question or provide more information, please leave a comment instead, requesting additional details. When answering, please include specifics, such as step-by-step instructions, context for the solution, and links to useful resources. Also, please make sure that you answer complies with our Rules of Engagement.
You must be Logged in to submit an answer.

Up to 10 attachments (including images) can be used with a maximum of 1.0 MB each and 10.5 MB total.