Skip to Content
0

SAP Data Services Jobs failing Intermittently

Oct 12, 2016 at 01:36 PM

165

avatar image

Hi Experts,

We are 3 ETL jobs that runs in a sequence and often one of the job gets failed daily. Failing pattern is not fixed.

These 3 jobs are loading data to Teradata and have one thing in common that it parallel flows loading data to separate tables. We do have other ETL jobs too that loads perfectly to Teradata except these 3 jobs.

We are using Bulkloader Utility within DS while loading to Teradata.

Below is the trace and error log we see whenever one of the jobs gets failed.

Error log:

(14.2) 10-09-16 04:03:53 (E) (34078764:2057) SYS-050612: |Data flow DF_EQUIPMENT_INSTANCE|Loader Equipment_INST_Qry2_EQUIPMENT_INSTANCE
                                                         Cannot create process with image file </BODI/prod/home/dataservices/log/bulkloader> and command line
                                                         </BODI/prod/home/dataservices/log/bulkloader/TD1_PUTLST_EQUIPMENT_INSTANCE_3.sh>, due to error <26>.
 (14.2) 10-09-16 04:03:53 (E) (29098064:2057) DBS-120823: |Data flow DF_EQUIPMENT_INSTANCE|Loader Equipment_INST_Qry2_EQUIPMENT_INSTANCE
                                                         Teradata command <fastload> failed.  Check $LINK_DIR/bin/td_env.config and run <fastload> with script file
                                                         </BODI/prod/home/dataservices/log/bulkloader/TD1_PUTLST_EQUIPMENT_INSTANCE_3.ctl> on the command line to debug or notify
                                                         Customer Support.
 (14.2) 10-09-16 04:04:06 (E) (46137502:0001) RUN-050316: |Session JOB_CDC_Initial_EQUIPMENT_INSTANCE
                                                         Error during the execution of the job:JOB_CDC_Initial_EQUIPMENT_INSTANCE
 (14.2) 10-09-16 04:04:06 (E) (46137502:0001) RUN-050304: |Session JOB_CDC_Initial_EQUIPMENT_INSTANCE
                                                         Function call <raise_exception ( Error during the execution of the job:JOB_CDC_Initial_EQUIPMENT_INSTANCE ) > failed, due to
                                                         error <50316>: <Error during the execution of the job:JOB_CDC_Initial_EQUIPMENT_INSTANCE>.
 (14.2) 10-09-16 04:04:06 (E) (46137502:0001) RUN-050316: |Session JOB_CDC_Initial_EQUIPMENT_INSTANCE
                                                         Error during the execution of the job:JOB_CDC_Initial_EQUIPMENT_INSTANCE

Trace Log:


Log: trace_10_09_2016_04_00_09_1367956__f301de55_c6c7_4ce6_bd01_c5036963bd8e.txt
(14.2) 10-09-16 04:00:11 (46137502:0001)      JOB: Reading job <f301de55_c6c7_4ce6_bd01_c5036963bd8e> from the repository; Server version is <14.2.3.549>; Repository version is
                                                   <14.2.3.0000>.
 (14.2) 10-09-16 04:00:11 (46137502:0001)      JOB: Current directory of job <f301de55_c6c7_4ce6_bd01_c5036963bd8e> is </BODI/prod/home/dataservices/bin>.
 (14.2) 10-09-16 04:00:11 (46137502:0001)      JOB: Starting job on job server host <sapbods1p>, port <3500>.
 (14.2) 10-09-16 04:00:12 (46137502:0001)      JOB: Job <JOB_CDC_Initial_EQUIPMENT_INSTANCE> of runid <2016100904001146137502001> is initiated by user <bodipadm>.
 (14.2) 10-09-16 04:00:12 (46137502:0001)      JOB: Processing job <JOB_CDC_Initial_EQUIPMENT_INSTANCE>.
 (14.2) 10-09-16 04:00:13 (46137502:0001)      JOB: Optimizing job <JOB_CDC_Initial_EQUIPMENT_INSTANCE>.
 (14.2) 10-09-16 04:00:13 (46137502:0001)      JOB: Job <JOB_CDC_Initial_EQUIPMENT_INSTANCE> is started.
 (14.2) 10-09-16 04:00:13 (46137502:0001) WORKFLOW: Work flow <WF_CDC_Initial_EQUIPMENT_INSTANCE> is started.
 (14.2) 10-09-16 04:00:13 (46137502:0001)  PRINTFN: #######Inside the Start Script#######
 (14.2) 10-09-16 04:00:13 (46137502:0001)  PRINTFN: #######Calling Stored Procedure:sp_etl_load_id_gen#######
 (14.2) 10-09-16 04:00:13 (46137502:0001)   DB_LIB: Load odbc driver library <libodbc.a(odbc.so)>.
 (14.2) 10-09-16 04:00:16 (29098062:0001) DATAFLOW: Process to execute data flow <DF_EQUIPMENT_MATERIAL> is started.
 (14.2) 10-09-16 04:00:18 (29098062:0001) DATAFLOW: The specified locale <eng_us.iso-8859-1> has been coerced to <Unicode (UTF-16)> for data flow <DF_EQUIPMENT_MATERIAL> because
                                                   the datastore <ERP_TeraData> obtains data in <Unicode (UTF-16)> codepage.
 (14.2) 10-09-16 04:00:18 (29098062:0001)      JOB: Initializing transcoder for datastore <ERP_TeraData> to transcode between engine codepage<Unicode (UTF-16)> and datastore
                                                   codepage <<DEFAULT>>
 (14.2) 10-09-16 04:00:18 (29098062:0001)      JOB: Initializing transcoder for datastore <Transport_Format> to transcode between engine codepage<Unicode (UTF-16)> and datastore
                                                   codepage <<DEFAULT>>
 (14.2) 10-09-16 04:00:18 (29098062:0001)      JOB: Initializing transcoder for datastore <TD1> to transcode between engine codepage<Unicode (UTF-16)> and datastore codepage
                                                   <<DEFAULT>>
 (14.2) 10-09-16 04:00:20 (29098062:0001)     ABAP: The codepage of transport file /P11/interface/BODI/TeraData/ZIFACO_Equip_Mat_EXTRACT.dat (ABAP Data Flow
                                                   ABAPDF_EQUIPMENT_MATERIAL) has been coerced to utf-8 for SAP ABAP Server above 4.6.
 (14.2) 10-09-16 04:00:20 (29098062:0001) DATAFLOW: Data flow <DF_EQUIPMENT_MATERIAL> is started.
 (14.2) 10-09-16 04:00:20 (29098062:0001)     ABAP: ABAP flow <ABAPDF_EQUIPMENT_MATERIAL> is started.
 (14.2) 10-09-16 04:00:20 (29098062:0001)     ABAP: Batch job <ZIFACO_Equip_Mat_EXTRACT 10/09/16 04:00:20> is submitted. Job number is <04002000>
 (14.2) 10-09-16 04:00:41 (29098062:0001)     ABAP: Batch job <ZIFACO_Equip_Mat_EXTRACT 10/09/16 04:00:20> for ABAP program <ZIFACO_EQUIP_MAT_EXTRACT> is finished.
 (14.2) 10-09-16 04:00:41 (29098062:0001)     ABAP: ABAP flow <ABAPDF_EQUIPMENT_MATERIAL> is completed.
 (14.2) 10-09-16 04:00:41 (29098062:0001) DATAFLOW: Data flow <DF_EQUIPMENT_MATERIAL> using PAGEABLE Cache with <3583 MB> buffer pool.
 (14.2) 10-09-16 04:00:46 (29098062:1543)  BLKLOAD: Starting Bulk Loader Utility <fastload> in <Dataflow DF_EQUIPMENT_MATERIAL>. Data file is
                                                   </BODI/prod/home/dataservices/log/bulkloader/TD1_PUTLST_EQUIPMENT_MATERIAL_11_0.dat>. Log file is
                                                   </BODI/prod/home/dataservices/log/bulkloader/TD1_PUTLST_EQUIPMENT_MATERIAL_11.log>.  Command line used is
                                                   </BODI/prod/home/dataservices/log/bulkloader/TD1_PUTLST_EQUIPMENT_MATERIAL_11.sh>.
 (14.2) 10-09-16 04:00:56 (29098062:0001) DATAFLOW: Data flow <DF_EQUIPMENT_MATERIAL> is completed successfully.
 (14.2) 10-09-16 04:00:56 (29098062:0001) DATAFLOW: Process to execute data flow <DF_EQUIPMENT_MATERIAL> is completed.
 (14.2) 10-09-16 04:00:58 (29098064:0001) DATAFLOW: Process to execute data flow <DF_EQUIPMENT_INSTANCE> is started.
 (14.2) 10-09-16 04:00:59 (29098064:0001) DATAFLOW: The specified locale <eng_us.iso-8859-1> has been coerced to <Unicode (UTF-16)> for data flow <DF_EQUIPMENT_INSTANCE> because
                                                   the datastore <ERP_TeraData> obtains data in <Unicode (UTF-16)> codepage.
 (14.2) 10-09-16 04:00:59 (29098064:0001)      JOB: Initializing transcoder for datastore <ERP_TeraData> to transcode between engine codepage<Unicode (UTF-16)> and datastore
                                                   codepage <<DEFAULT>>
 (14.2) 10-09-16 04:00:59 (29098064:0001)      JOB: Initializing transcoder for datastore <Transport_Format> to transcode between engine codepage<Unicode (UTF-16)> and datastore
                                                   codepage <<DEFAULT>>
 (14.2) 10-09-16 04:00:59 (29098064:0001)      JOB: Initializing transcoder for datastore <TD1> to transcode between engine codepage<Unicode (UTF-16)> and datastore codepage
                                                   <<DEFAULT>>
 (14.2) 10-09-16 04:01:00 (29098064:0001)     ABAP: The codepage of transport file /P11/interface/BODI/TeraData/ZIFACO_Equip_Inst_EXTRACT.txt (ABAP Data Flow
                                                   ABAPDF_EQUIPMENT_INSTANCE) has been coerced to utf-8 for SAP ABAP Server above 4.6.
 (14.2) 10-09-16 04:01:00 (29098064:0001) DATAFLOW: Data flow <DF_EQUIPMENT_INSTANCE> is started.
 (14.2) 10-09-16 04:01:00 (29098064:0001)     ABAP: ABAP flow <ABAPDF_EQUIPMENT_INSTANCE> is started.
 (14.2) 10-09-16 04:01:00 (29098064:0001)     ABAP: Batch job <ZIFACO_Equip_Inst_EXTRACT 10/09/16 04:01:00> is submitted. Job number is <04010000>
 (14.2) 10-09-16 04:03:11 (29098064:0001)     ABAP: Batch job <ZIFACO_Equip_Inst_EXTRACT 10/09/16 04:01:00> for ABAP program <ZIFACO_EQUIP_INST_EXTRACT> is finished.
 (14.2) 10-09-16 04:03:11 (29098064:0001)     ABAP: ABAP flow <ABAPDF_EQUIPMENT_INSTANCE> is completed.
 (14.2) 10-09-16 04:03:11 (29098064:0001) DATAFLOW: Data flow <DF_EQUIPMENT_INSTANCE> using PAGEABLE Cache with <3582 MB> buffer pool.
 (14.2) 10-09-16 04:03:39 (29098064:2571)  BLKLOAD: Starting Bulk Loader Utility <fastload> in <Dataflow DF_EQUIPMENT_INSTANCE>. Data file is
                                                   </BODI/prod/home/dataservices/log/bulkloader/TD1_PUTLST_METER_11_0.dat>. Log file is
                                                   </BODI/prod/home/dataservices/log/bulkloader/TD1_PUTLST_METER_11.log>.  Command line used is
                                                   </BODI/prod/home/dataservices/log/bulkloader/TD1_PUTLST_METER_11.sh>.
 (14.2) 10-09-16 04:03:39 (29098064:2057)  BLKLOAD: Starting Bulk Loader Utility <fastload> in <Dataflow DF_EQUIPMENT_INSTANCE>. Data file is
                                                   </BODI/prod/home/dataservices/log/bulkloader/TD1_PUTLST_EQUIPMENT_INSTANCE_3_0.dat>. Log file is
                                                   </BODI/prod/home/dataservices/log/bulkloader/TD1_PUTLST_EQUIPMENT_INSTANCE_3.log>.  Command line used is
                                                   </BODI/prod/home/dataservices/log/bulkloader/TD1_PUTLST_EQUIPMENT_INSTANCE_3.sh>.
 (14.2) 10-09-16 04:04:03 (29098064:0001) DATAFLOW: Data flow <DF_EQUIPMENT_INSTANCE> is terminated due to error <120823>.
 (14.2) 10-09-16 04:04:03 (29098064:0001) DATAFLOW: Process to execute data flow <DF_EQUIPMENT_INSTANCE> is completed.
 (14.2) 10-09-16 04:04:06 (46137502:0001) WORKFLOW: Work flow <WF_CDC_Initial_EQUIPMENT_INSTANCE> is terminated due to an error <120823>.
 (14.2) 10-09-16 04:04:06 (46137502:0001)  PRINTFN: #######Inside Catch Block#######
 (14.2) 10-09-16 04:04:06 (46137502:0001)  PRINTFN: Error in Job:JOB_CDC_Initial_EQUIPMENT_INSTANCE
 (14.2) 10-09-16 04:04:06 (46137502:0001)  PRINTFN: Error Number:120823
 (14.2) 10-09-16 04:04:06 (46137502:0001)  PRINTFN: Error Message:Teradata command <fastload> failed.  Check $LINK_DIR/bin/td_env.config and run <fastload> with script file
                                                   </BODI/prod/home/dataservices/log/bulkloader/TD1_PUTLST_EQUIPMENT_INSTANCE_3.ctl> on the command line to debug or notify
                                                   Customer Support.
 (14.2) 10-09-16 04:04:06 (46137502:0001)  PRINTFN: #######Raising Exception#######
 (14.2) 10-09-16 04:04:06 (46137502:0001)      JOB: Job <JOB_CDC_Initial_EQUIPMENT_INSTANCE> is terminated due to error <50316>.

Let me know if any further info required.

We tried running .sh script at Unix server too but does not show up much information.

Thanks,

Amit.

10 |10000 characters needed characters left characters exceeded
* Please Login or Register to Answer, Follow or Comment.

1 Answer

Abhi K Oct 12, 2016 at 02:42 PM
0

Hi Amit,

Have you set the required parameters; as shown in error below? It seems it is not able to fetch the required env...

PRINTFN: Error Message:Teradata command <fastload> failed.  Check $LINK_DIR/bin/td_env.config and run <fastload> with script file

Thanks

Share
10 |10000 characters needed characters left characters exceeded