Dataflow - Automatically read from multiple text file

[Former Member]

I have multiple text files generated from different system.
I have currently created multiple dataflows in BODS to read those files and set them to populate in a single table. All the dataflows have been set to run in parallel.
So whenever a new file is placed, i replicate the dataflow and change the file location and name(All files and path have standard naming convention).
I'm trying to automate this setup by maintaining certain parameters in a table and the job will automatically pick the text files and read them.
We can achieve this by means of While Loop but we cannot set them to run in Parallel. The reason i want to achieve parallel execution here is that, there are lots of dataflows that are available. Running them sequentially will take lot of time.
How can i handle this situation ?
Any help would be greatly appreciated.

[former_member208402]

Hi Parsuwanath AnanthaVijayan,

If your loading all these files to a single target table and all these files are placed in a single location then give filename as *.txt in the file format editor. it will load all the files that exists in that path. You can built this in one dataflow only.

if all your files are have same naming convention, example :- sales_india, sales_us,sales_eur then give filename as sales*.txt.

you can use substitution parameter in DS where you can parameterize your file location and call parameter in the place of file location.

Please let me know if your requirement is different.

thanks,

Ravi kiran

[Former Member]

Hi Ravi ,
Thanks for spending your valuable time to help me out.

We have files in separate folders for different countries.

Lemme give you an example.
For an object, Say Sales Order from India,
the folder path will be C://India//Sales Order//Sales_Order_India.txt
And for canada it will be C://Canada//Sales Order//Sales_Order_Canada.txt
and this goes one.
Currently i have created two dataflows for reading the above two files. And suppose a new country comes into Picture, i will create another dataflow by replicating one of the dataflow above and change the file path to point towards the new country.

Also we have separate table maintained to check whether a country is in scope or not.

With the help of Global Variables, while loop and the above table i can achieve this automation but the dataflow will be executed sequentially. I want to run this in parallel because we have more than 100 dataflows for each object and running sequentially will not help here.

[former_member208402]

Hi Parsuwanath AnanthaVijayan,

In this case try if you can move all files to a single location daily and start processing all the files in a single dataflow. But it still process file by file. You can maintain a table where you can store filepath for diff files, file names in a table and when ever you get any new file you just add file path and file name in the table, so build a script with while loop which automatically moves all files from diff location to single location using to table.

Other wise whenever you get a new file, replicate dataflow as you are doing now.

Thanks,

Ravi kiran.