DS run longer with scheduled job as compare to manual run

[Former Member]

I have scheduled a job through Management Console (MC) to run everyday once at 

certain time. After some time, maybe after 15 days of running, the execution 

time has increased to double from 17 mins to 67 mins by one time jump. After 

that, the job kept running and spent 67 mins to complete.

The nature of the job is to generate around 400 output flat files from a 

source db2 table. Under efficient running time, 1 file took around 2 seconds to 

generate. But now it became taking 8 seconds to generate one file. The data 

volume and nature of the source table didn't change, so that was not the root 

cause of increasing time.

I have done several investigations and the results as such:

1) I scheduled again this job at MC to run for testing, it would take 67 mins 

to complete. However, if I manually run this job thorough MC, it would take 

17 mins efficient time to run.

2) I replicated this job to a second job as a copy. Then I scheduled this 

copied job at MC to run, it would take 67 mins to run. But if I manually run 

this job through MC, it will take 17 mins to run.

3) I created another test repo and load this job in. I scheduled the job to 

run at this new repo, it would take 67 mins to run. If I manually run the job 

through MC, it only took 17 mins to run.

4) Finally, I manually executed the job through unix job scripts command, 

which is one of the scheduled job entry in the cron file, such as 

./DI__4c553b0d_6fe5_4083_8655_11cb0fe230f4_2_r_3_w_n_6_40.sh, the job also would take 17 

mins to run to finish.

5) I have recreated the repo to make it clean and reload back the jobs and 

recreated again the schedule. Yet it still took 67 mins to run scheduled job.

Therefore, the conclusion is why it takes longer time to run by scheduling 

method as compare to manually running method?

Please provide me a way to troubleshoot this problem. Thank you.

OS : HPUX 11.31

DS : BusinessObjects Data Services 12.1.1.0

databasee : DB2 9.1

[Former Member]

executing from schedule is same as what you did in step 4) executing the .bat file from the command line

you also mentioned that eah file generation takes 8sec instead of 2 sec, investigate around that area, how are you reading data and writing data to a file, in a single DF ? or calling a same DF in while loop ?

when the job is scheduled to run what else is running on the System ? is there any backup utility for DB or OS running at that time?

[Former Member]

That is the valid point you mention in 4). However, the result was different as stated, as long as I use schedule at MC to run, that will be longer time. However if I manually run it either through MC or by scripts command in item 4), it will be shorter.

There is only one DF will be looped many times by changing its input parameters to select records from a db2 table, does a grouping and summation and output to a flat file. It is a single DF to read and output to flat file, and it will be called many times by looping.

DS support personnel from China said it may have something to do with the cron job, but he couldn't identify any problem because he was not familiar with cron job.

[Former Member]

if the problem is with the cron, then it will take time to launch the job, not during execution of the job

is the job getting started as per schedule ? or there is a delay in starting the job

if you have opened a case with support and its still open you can attach the log file to the incident, I can take the log file from there just give in the incident number (message number)

[Former Member]

The job is getting started as per schedule, and no delay. I have uploaded the trace logs to the support site.

The incident that I logged is:

Job execution time has increase suddenly ( 786739 / 2009 )

[Former Member]

I am checking the log files, one more thing have you tried scheduling the job for different time or running the job manually during the time its scheduled

from the log file the jobs from the schedule started at different time and the one executed manually was run at different time, it hard to tell from the trace log about the system load, and what other activities were happeing at that time that could affect the performance of the job

check the monitor log of both the files, are both job execution processing the same number of rows ?

[Former Member]

During the investigation, we did schedule the job to differrent time to run, but it still gave 67 mins. I uploaded again this adhoc scheduled test run trace to support system for your reference. For the investigation, we run same data twice, one with schedule run, which slowed down, and another run with manual trigger, which was faster.

[Former Member]

Hi,

The lastest investigation shows that even the job is run manually, the time also has slowed down to 67 minutes. I am sorry to misled your guys to think that this problem only affect scheduled job. I also not sure why suddenly manually run job has increased the time.

So I will further make some more testings and find out the reason and provide update to you all.

[Former Member]

Yesterday we had done another test and indirectly made the problem to go 

away. We changed the generated output flat file directory from current directory 

of /fdminst/cmbc/fdm_d/bds/gl to /fdminst/cmbc/fdm_d/bds/config directory to 

run, to see any difference would make. We changed the directory pointing 

inside Substitution Parameter Configurations windows. Surprisingly, job had 

started to run fast and completed in 15 minutes and not 67 minutes anymore.

Then we shifted back and pointed the output directory back to original 

/fdminst/cmbc/fdm_d/bds/gl and the job has started to run fast ever since and all 

completed in 15 minutes. Even we created ad hoc schedule to run and it was 

still running fast.

We not sure why it was solved by shifting directory away and shifting back, 

and whether this had to do with BODS problem or HP Unix system environment 

problem. Nonetheless, the job is started to run normally and fast now as we 

test.

[Former Member]

Can you upload or paste the Trace log from each execution? That might help in diagnosis... Thanks, -rs-

[Former Member]

How do I upload the trace? If I paste it here, it will be very long.

[Former Member]

Open up the trace log in the Designer

Click on the first line

Scroll to the last line

Hold down Shift and click on the last line

right-mouse / Copy

You should be able to paste that into a Forum Reply

[Former Member]

I can't paste the trace log here, becaue it is too long and exceeds the maximum length of 15000 characters. The trace log has more then 6000 lines. Is there a way for me to post it?