Skip to Content

Improving performace of the job while loading the data into Json format

Jul 15, 2017 at 09:36 AM


avatar image
Former Member

Hi All,

I am trying to load data into json format and from json to ODL(operation data layer which is cloud).

The json format I am loading has nested schemas.

I have to load 40 millions of data which is transactional data.

so to improve the performance of the job, I have designed 5 parallel dfs which process 500 transactions each at a time. 500 transactions are processed in 5 mnts by single df. so in 5 mnts, I can process max 2500 transactions. if I process more than 500 records( lets say for 1000-40 mnts and 1500-1 hr 20 mnts), it takes more time.

I have tried this with query transforms(better performance),xml_map transform(not better than query). Is there a way to improve the performance of the job ?

10 |10000 characters needed characters left characters exceeded
* Please Login or Register to Answer, Follow or Comment.

0 Answers