Skip to Content
avatar image
Former Member

Work processes of shadow instance (SAP upgrade) not starting

HI,

I'm in the process of upgrading our system to EhP4. I'm now in MAIN_SHADOW/START_SHDI_FIRST phase. Now the shadow instance is not starting up and here's the contents of the dev_w0 of the shadow instance work directory:

M  kernel runs with dp version 241(ext=110) (@(#) DPLIB-INT-VERSION-241)
M  length of sys_adm_ext is 364 bytes
M  ***LOG Q01=> ThInit, WPStart (Workproc 0 1 27066) [thxxhead.c   1290]
M  ThInit: running on host a65od007
M  calling db_connect ...
C  Oracle Client Version: '10.2.0.4.0'
C  Client NLS setting (OCINlsGetInfo): 'AMERICAN_AMERICA.WE8DEC'
C  Logon as OPS$-user to get SAPP01SHD's password
C  Connecting as /@Q03 on connection 0 (nls_hdl 0) ... (dbsl 700 151208)
C  Nls CharacterSet                 NationalCharSet              C      EnvHp      ErrHp ErrHpBatch
C    0 WE8DEC                                                    0  0x467e180  0x4685c60  0x46895c8
C  Attaching to DB Server Q03 (con_hdl=0,svchp=0x46894f8,srvhp=0x468bc08)
M  ------------------ C-STACK ----------------------
(CTrcStack2+0x82)[0x6c5852]
(SigIGenAction+0x2a2)[0x1938d62]
/lib64/libpthread.so.0[0x2b5b14e13d60]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(gslcgvg_GetValues+0x75)[0x2b5b1d3984c5]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(ora_ldap_get_values+0x71)[0x2b5b1d3839b9]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(ldap_get_values+0x44)[0x2b5b1d383936]
/lib64/libnss_ldap.so.2[0x2b5b1e9777fc]
/lib64/libnss_ldap.so.2[0x2b5b1e979c30]
/lib64/libnss_ldap.so.2[0x2b5b1e9773ab]
/lib64/libnss_ldap.so.2(_nss_ldap_gethostbyname2_r+0x6d)[0x2b5b1e97a10d]
/lib64/libnss_ldap.so.2(_nss_ldap_gethostbyname_r+0x66)[0x2b5b1e97a1b6]
/lib64/libc.so.6(gethostbyname_r+0x121)[0x2b5b14ff4371]
/lib64/libc.so.6(gethostbyname+0xd3)[0x2b5b14ff3c23]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(snlpcgthstbynm+0x42)[0x2b5b1d4e7f52]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(snlinGetAddrInfo+0x408)[0x2b5b1d4ebf00]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(nttbnd2addr+0x2b2)[0x2b5b1d558ce2]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(ntacbbnd2addr+0x1b8)[0x2b5b1d436ea6]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(ntacbnd2addr+0x4e)[0x2b5b1d436c62]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(nsc2addr+0x8a)[0x2b5b1d4b1ed6]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(nscall1+0x17c)[0x2b5b1d3fe1c2]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(nscall+0x332)[0x2b5b1d3fd5d8]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(niotns+0x6c2)[0x2b5b1d441efc]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(nigcall+0x3b)[0x2b5b1d4bfa53]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(osncon+0x444)[0x2b5b1d447442]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(kpuadef+0x88)[0x2b5b1d0f05e8]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(upiini+0x2ec)[0x2b5b1d2651ac]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(upiah0+0x72)[0x2b5b1d24224a]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(kpuatch+0x101a)[0x2b5b1d0effe8]
/oracle/Q03/102_64/lib/libclntsh.so.10.1(OCIServerAttach+0xa7)[0x2b5b1d2689d9]
/usr/sap/Q03/upg/abap/exe/dboraslib.so[0x2b5b1cc94f99]
/usr/sap/Q03/upg/abap/exe/dboraslib.so(oci_logon+0xec)[0x2b5b1cc9a61c]
/usr/sap/Q03/upg/abap/exe/dboraslib.so[0x2b5b1ccab47a]
/usr/sap/Q03/upg/abap/exe/dboraslib.so[0x2b5b1ccab841]
/usr/sap/Q03/upg/abap/exe/dboraslib.so[0x2b5b1ccad0ee]
(open_con+0xf1)[0x129e631]
(supply_con+0x31d)[0x12a065d]
(dbsl_connect+0x11)[0x12acab1]
(db_connect+0x11e)[0x12a30ae]
(ThInit+0x977)[0x556de7]
(ThStart+0x11b)[0x5590db]
(DpMain+0x285)[0x4bd985]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2b5b14f3c304]
M  -------------------------------------------------
M
M Mon Jan 24 11:42:20 2011
M  ***LOG Q0E=> SigIGenAction, signal ( 11) [sigux.c      1067]
M  in_ThErrHandle: 1
M  ThSigHandler: signal (step 1, th_errno 11, action 4)
M
M  Info for wp 0
M
M    pid = 27066
M    severity = 0
M    status = 0
M    stat = WP_RUN
M    waiting_for = NO_WAITING
M    reqtype = DP_RQ_DIAWP
M    act_reqtype = NO_REQTYPE
M    rq_info = 0
M    tid = -1
M    mode = 255
M    len = -1
M    rq_id = 65535
M    rq_source =
M    last_tid = 0
M    last_mode = 0
M    semaphore = 0
M    act_cs_count = 0
M    csTrack = 0
M    csTrackRwExcl = 0
M    csTrackRwShrd = 0
M    mode_cleaned_counter = 0
M    control_flag = 0
M    int_checked_resource(RFC) = 0
M    ext_checked_resource(RFC) = 0
M    int_checked_resource(HTTP) = 0
M    ext_checked_resource(HTTP) = 0
M    report = >                                        <
M    action = 0
M    tab_name = >                              <
M    attachedVm = no VM
M  PfStatDisconnect: disconnect statistics
M  Entering TH_CALLHOOKS
M  ThCallHooks: call hook >ThrSaveSPAFields< for event BEFORE_DUMP
M  *** ERROR => ThrSaveSPAFields: no valid thr_wpadm [thxxrun1.c   723]
M  *** ERROR => ThCallHooks: event handler ThrSaveSPAFields for event BEFORE_DUMP failed [thxxtool3.c  261]
M  ThCallHooks: call hook >BtcCallLgCl< for event BEFORE_DUMP
M  Entering ThSetStatError
M  ThIErrHandle: do not call ThrCoreInfo (no_core_info=0, in_dynp_env=0)
M  Entering ThReadDetachMode
M  ***LOG Q02=> wp_halt, WPStop (Workproc 0 27066) [dpuxtool.c   269] 

I have already tried to restart the shadow instance manually using SAP Note 525677 (with cleanipc) but the problem still persists

Edited by: Lee Hwei on Jan 24, 2011 6:22 AM

Edited by: Lee Hwei on Jan 24, 2011 6:27 AM

Edited by: Lee Hwei on Jan 24, 2011 6:30 AM

Add comment
10|10000 characters needed characters exceeded

  • Follow
  • Get RSS Feed

3 Answers

  • avatar image
    Former Member
    Jan 24, 2011 at 05:51 AM

    Hi,

    Below note may help you.

    Note 525677 - Problems when starting shadow instance

    If this is not your Production System then you can shutdown your syste and repeat the phase.

    Regards,

    Nirmal.K

    Add comment
    10|10000 characters needed characters exceeded

    • Former Member

      In addition to my earlier reply, you can try this :

      1) Stop upgrade tool and main instance.

      2) use cleanipc command to remove semaphores.

      cleanipc <instance number> remove. Remove from both shadow and main instance.

      3) Start shadow instance.

      4) Start upgrade tool and repeat the phase.

      Regards,

      Nirmal.k

  • avatar image
    Former Member
    Jan 24, 2011 at 06:40 AM

    Could you please also paste dev_disp and dev_ms logs from shadow instance directory.

    Thanks

    Sunny

    Add comment
    10|10000 characters needed characters exceeded

    • Former Member

      >

      > Could you please also paste dev_disp and dev_ms logs from shadow instance directory.

      >

      > Thanks

      > Sunny

      Hi,

      here's dev_disp of shadow instance (is there a way here in SCN to paste this properly???)

      -


      trc file: "dev_disp.new", trc level: 1, release: "701"

      -


      sysno 12

      sid Q03

      systemid 390 (AMD/Intel x86_64 with Linux)

      relno 7010

      patchlevel 0

      patchno 32

      intno 20020600

      make: single threaded, ASCII, 64 bit, optimized

      pid 27006

      Mon Jan 24 11:41:41 2011

      kernel runs with dp version 241(ext=110) (@(#) DPLIB-INT-VERSION-241)

      length of sys_adm_ext is 364 bytes

      • SWITCH TRC-HIDE on ***

      ***LOG Q00=> DpSapEnvInit, DPStart (12 27006) [dpxxdisp.c 1286]

      shared lib "dw_xml.so" version 32 successfully loaded

      shared lib "dw_xtc.so" version 32 successfully loaded

      shared lib "dw_stl.so" version 32 successfully loaded

      shared lib "dw_gui.so" version 32 successfully loaded

      shared lib "dw_mdm.so" version 32 successfully loaded

      rdisp/softcancel_sequence : -> 0,5,-1

      Mon Jan 24 11:42:07 2011

      • WARNING => DpNetCheck: NiHostToAddr(www.doesnotexist0101.qqq.nxst) took 26 seconds

      Mon Jan 24 11:42:17 2011

      • WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 10 seconds

      ***LOG GZZ=> 2 possible network problems detected - check tracefile and adjust the DNS settings [dpxxtool2.c 5529]

      MtxInit: 30000 0 0

      DpSysAdmExtInit: ABAP is active

      DpSysAdmExtInit: VMC (JAVA VM in WP) is not active

      DpIPCInit2: start server >a65wd007_Q03_12 <

      DpShMCreate: sizeof(wp_adm) 26928 (1224)

      DpShMCreate: sizeof(tm_adm) 5351424 (26624)

      DpShMCreate: sizeof(wp_ca_adm) 22912 (76)

      DpShMCreate: sizeof(appc_ca_adm) 7600 (76)

      DpCommTableSize: max/headSize/ftSize/tableSize=500/8/548040/548048

      DpShMCreate: sizeof(comm_adm) 548048 (1088)

      DpSlockTableSize: max/headSize/ftSize/fiSize/tableSize=0/0/0/0/0

      DpShMCreate: sizeof(slock_adm) 0 (104)

      DpFileTableSize: max/headSize/ftSize/tableSize=0/0/0/0

      DpShMCreate: sizeof(file_adm) 0 (72)

      DpShMCreate: sizeof(vmc_adm) 0 (1760)

      DpShMCreate: sizeof(wall_adm) (22440/36712/72/104)

      DpShMCreate: sizeof(gw_adm) 48

      DpShMCreate: SHM_DP_ADM_KEY (addr: 0x2b5b1f96d000, size: 6023992)

      DpShMCreate: allocated sys_adm at 0x2b5b1f96d000

      DpShMCreate: allocated wp_adm at 0x2b5b1f96ed98

      DpShMCreate: allocated tm_adm_list at 0x2b5b1f9756c8

      DpShMCreate: allocated tm_adm at 0x2b5b1f975728

      DpShMCreate: allocated wp_ca_adm at 0x2b5b1fe8ff28

      DpShMCreate: allocated appc_ca_adm at 0x2b5b1fe958a8

      DpShMCreate: allocated comm_adm at 0x2b5b1fe97658

      DpShMCreate: system runs without slock table

      DpShMCreate: system runs without file table

      DpShMCreate: allocated vmc_adm_list at 0x2b5b1ff1d328

      DpShMCreate: allocated gw_adm at 0x2b5b1ff1d3a8

      DpShMCreate: system runs without vmc_adm

      DpShMCreate: allocated ca_info at 0x2b5b1ff1d3d8

      DpShMCreate: allocated wall_adm at 0x2b5b1ff1d3e0

      MBUF state OFF

      DpCommInitTable: init table for 500 entries

      rdisp/queue_size_check_value : -> off

      ThTaskStatus: rdisp/reset_online_during_debug 0

      EmInit: MmSetImplementation( 2 ).

      MM global diagnostic options set: 0

      <ES> client 0 initializing ....

      <ES> InitFreeList

      <ES> block size is 4096 kByte.

      Using implementation map

      <EsMap> EsMapUnamFileMapInit: Initialize the memory 4096 MB

      Linux: Kernel supports shared memory disclaiming

      Linux: using madvise(<pointer>, <size>, 9).

      Linux: disclaiming for shared memory enabled

      <ES> 1021 blocks reserved for free list.

      ES initialized.

      • WARNING => System running without ICM - check rdisp/start_icman [dpxxdisp.c 13169]

      Mon Jan 24 11:42:19 2011

      rdisp/http_min_wait_dia_wp : 1 -> 1

      Mon Jan 24 11:42:20 2011

      ***LOG Q0K=> DpMsAttach, mscon ( a65wd007) [dpxxdisp.c 12483]

      DpStartStopMsg: send start message (myname is >a65wd007_Q03_12 <)

      DpStartStopMsg: start msg sent

      CCMS: AlInitGlobals : alert/use_sema_lock = TRUE.

      CCMS: start to initalize 3.X shared alert area (first segment).

      • ERROR => DpWPCheck: W0 (pid 27066) died (severity=0, status=11) [dpxxdisp.c 15662]

      child (pid=27066) killed with signal 11

      my types changed after wp death/restart 0x1f --> 0x1d

      • ERROR => DpWPCheck: W1 (pid 27067) died (severity=0, status=11) [dpxxdisp.c 15662]

      child (pid=27067) killed with signal 11

      • ERROR => DpWPCheck: W3 (pid 27069) died (severity=0, status=11) [dpxxdisp.c 15662]

      child (pid=27069) killed with signal 11

      • ERROR => DpWPCheck: W4 (pid 27070) died (severity=0, status=11) [dpxxdisp.c 15662]

      child (pid=27070) killed with signal 11

      • ERROR => DpWPCheck: W5 (pid 27071) died (severity=0, status=11) [dpxxdisp.c 15662]

      child (pid=27071) killed with signal 11

      • ERROR => DpWPCheck: W7 (pid 27073) died (severity=0, status=11) [dpxxdisp.c 15662]

      child (pid=27073) killed with signal 11

      • ERROR => DpWPCheck: W10 (pid 27076) died (severity=0, status=11) [dpxxdisp.c 15662]

      child (pid=27076) killed with signal 11

      DpMsgAdmin: Set release to 7010, patchlevel 0

      MBUF state PREPARED

      MBUF component UP

      DpMBufHwIdSet: set Hardware-ID

      ***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c 1048]

      DpMsgAdmin: Set patchno for this platform to 32

      Release check o.K.

      Mon Jan 24 11:42:33 2011

      • ERROR => DpWPCheck: W0 (pid 27066) died (severity=0, status=0) [dpxxdisp.c 15662]

      my types changed after wp death/restart 0x1d --> 0x0

      • ERROR => DpWPCheck: W1 (pid 27067) died (severity=0, status=0) [dpxxdisp.c 15662]

      • ERROR => DpWPCheck: W2 (pid 27068) died (severity=0, status=11) [dpxxdisp.c 15662]

      child (pid=27068) killed with signal 11

      • ERROR => DpWPCheck: W3 (pid 27069) died (severity=0, status=0) [dpxxdisp.c 15662]

      • ERROR => DpWPCheck: W4 (pid 27070) died (severity=0, status=0) [dpxxdisp.c 15662]

      • ERROR => DpWPCheck: W5 (pid 27071) died (severity=0, status=0) [dpxxdisp.c 15662]

      • ERROR => DpWPCheck: W6 (pid 27072) died (severity=0, status=0) [dpxxdisp.c 15662]

      • ERROR => DpWPCheck: W7 (pid 27073) died (severity=0, status=0) [dpxxdisp.c 15662]

      • ERROR => DpWPCheck: W8 (pid 27074) died (severity=0, status=0) [dpxxdisp.c 15662]

      • ERROR => DpWPCheck: W9 (pid 27075) died (severity=0, status=11) [dpxxdisp.c 15662]

      child (pid=27075) killed with signal 11

      • ERROR => DpWPCheck: W10 (pid 27076) died (severity=0, status=0) [dpxxdisp.c 15662]

  • avatar image
    Former Member
    Jan 24, 2011 at 08:52 AM

    Hi,

    Thank you all for responding. This is already solved. I changed the value of SAPLOCALHOSTFULL in all profiles of the shadow instance to the value of the virtual hostname (initially the value was the physical hostname)

    Add comment
    10|10000 characters needed characters exceeded