Very Very Urgent : Error during MAIN_UPTRANS/STARTSAP_PUPG phase

[Former Member]

ThDear All,

We are upgrading our ECC6 system to EHP5.

during MAIN_UPTRANS/STARTSAP_PUPG phase we are getting error :

Last error code set: Process /usr/sap/PRD/DVEBMGS01/exe/sapcontrol exited with 2, see '/oracle/PRD/sapdata1/SUM/abap/log/SAPup.ECO' for details<br/> System start failed<br/> Earlier, validating the profile failed, see INSTANCE.OUT

************************************************

Below is the dev_disp log files :

*********************************************

---------------------------------------------------

trc file: "dev_disp.new", trc level: 1, release: "720"

---------------------------------------------------

sysno      01

sid        PRD

systemid   274 (HP (IA-64) with HP-UX)

relno      7200

patchlevel 0

patchno    401

intno      20020600

make       single threaded, Unicode, 64 bit, optimized

profile    /usr/sap/PRD/SYS/profile/PRD_DVEBMGS01_dbciPRD

pid        9493

kernel runs with dp version 138000(ext=119000) (@(#) DPLIB-INT-VERSION-138000-UC)

length of sys_adm_ext is 588 bytes

*** SWITCH TRC-HIDE on ***

***LOG Q00=> DpSapEnvInit, DPStart (01 9493) [dpxxdisp.c   1316]

        shared lib "dw_xml.sl" version 401 successfully loaded

        shared lib "dw_xtc.sl" version 401 successfully loaded

        shared lib "dw_stl.sl" version 401 successfully loaded

        shared lib "dw_gui.sl" version 401 successfully loaded

        shared lib "dw_mdm.sl" version 401 successfully loaded

        shared lib "dw_rndrt.sl" version 401 successfully loaded

        shared lib "dw_abp.sl" version 401 successfully loaded

        shared lib "dw_sym.sl" version 401 successfully loaded

        shared lib "dw_aci.sl" version 401 successfully loaded

rdisp/softcancel_sequence :  -> 0,5,-1

use internal message server connection to port 3900

rdisp/dynamic_wp_check : 1

rdisp/calculateLoadAverage : 1

MtxInit: 30000 0 0

DpSysAdmExtInit: ABAP is active

DpSysAdmExtInit: VMC (JAVA VM in WP) is not active

DpIPCInit2: write dp-profile-values into sys_adm_ext

DpIPCInit2: start server >dbciPRD_PRD_01                          <

DpShMCreate: sizeof(wp_adm)             63168   (2256)

DpShMCreate: sizeof(tm_adm)             55563776        (27768)

DpShMCreate: sizeof(wp_ca_adm)          80000   (80)

DpShMCreate: sizeof(appc_ca_adm)        160000  (80)

DpCommTableSize: max/headSize/ftSize/tableSize=2000/16/5536064/5536080

DpShMCreate: sizeof(comm_adm)           5536080 (2744)

DpSlockTableSize: max/headSize/ftSize/fiSize/tableSize=0/0/0/0/0

DpShMCreate: sizeof(slock_adm)          0       (296)

DpFileTableSize: max/headSize/ftSize/tableSize=0/0/0/0

DpShMCreate: sizeof(file_adm)           0       (80)

DpShMCreate: sizeof(vmc_adm)            0       (2304)

DpShMCreate: sizeof(wall_adm)           (416064/352496/80/192)

DpShMCreate: sizeof(gw_adm)     48

DpShMCreate: sizeof(j2ee_adm)   3952

DpShMCreate: SHM_DP_ADM_KEY             (addr: c000000022000000, size: 62190192)

DpShMCreate: allocated sys_adm at c000000022000010

DpShMCreate: allocated wp_adm_list at c000000022003180

DpShMCreate: allocated wp_adm at c000000022003370

DpShMCreate: allocated tm_adm_list at c000000022012a40

DpShMCreate: allocated tm_adm at c000000022012a90

DpShMCreate: allocated wp_ca_adm at c0000000255100a0

DpShMCreate: allocated appc_ca_adm at c000000025523930

DpShMCreate: allocated comm_adm at c00000002554aa40

DpShMCreate: system runs without slock table

DpShMCreate: system runs without file table

DpShMCreate: allocated vmc_adm_list at c000000025a923a0

DpShMCreate: system runs without vmc_adm

DpShMCreate: allocated gw_adm at c000000025a92450

DpShMCreate: allocated j2ee_adm at c000000025a92490

DpShMCreate: allocated ca_info at c000000025a93410

DpShMCreate: allocated wall_adm at c000000025a934c0

DpCommAttachTable: attached comm table (header=c00000002554aa40/ft=c00000002554aa50)

DpSysAdmIntInit: initialize sys_adm

rdisp/test_roll : roll strategy is DP_NORMAL_ROLL

dia token check not active (15 token)

MBUF state OFF

DpCommInitTable: init table for 2000 entries

DpRqQInit: keep protect_queue / slots_per_queue 0 / 2001 in sys_adm

rdisp/queue_size_check_value :  -> on,50,30,40,500,50,500,80

EmInit: MmSetImplementation( 2 ).

MM global diagnostic options set: 0

<ES> client 0 initializing ....

<ES> EsILock: use spinlock for locking

<ES> InitFreeList

<ES> block size is 4096 kByte.

Using implementation std

<ES> Info: use normal pages (no huge table support available)

EsStdUnamFileMapInit: ES base = 0xc000000300000000

EsStdInit: Extended Memory 6144 MB allocated

<ES> 1535 blocks reserved for free list.

ES initialized.

mm.dump: set maximum dump mem to 96 MB

DpVmcSetActive: set vmc state DP_VMC_NOT_ACTIVE

MPI: dynamic quotas disabled.

MPI init: pipes=4000 buffers=1279 reserved=383 quota=10%

Sun Jul 28 17:24:42 2013

rdisp/http_min_wait_dia_wp : 1 -> 1

***LOG CPS=> DpLoopInit, ICU ( 4.0.1 4.0.1 5.1) [dpxxdisp.c   1703]

***LOG Q0I=> NiPConnect: 192.168.0.15:3900: connect (239: Connection refused) [nixxi.cpp 2897]

*** ERROR => NiPConnect: SiConnect failed for hdl 17/sock 12

    (SI_ECONN_REFUSE/239; I4; ST; 192.168.0.15:3900) [nixxi.cpp    2897]

*** ERROR => MsIAttachEx: NiBufConnect to dbciPRD/3900 failed (rc=NIECONN_REFUSED) [msxxi.c      713]

***LOG Q0L=> DpLoopInit, nomscon () [dpxxdisp.c   1807]

CCMS: SemInMgt: Semaphore Management initialized by AlAttachShm_Doublestack.

CCMS: SemInit: Semaphore 38 initialized by AlAttachShm_Doublestack.

CCMS: start to initalize 3.X shared alert area (first segment).

DpAppcBlksLow: max appc blocks in queue is 1000 (50 %)

DpWpBlksLow: max wp blocks in queue is 800 (80 %)

Sun Jul 28 17:25:22 2013

***LOG Q0I=> NiPConnect: 192.168.0.15:3900: connect (239: Connection refused) [nixxi.cpp 2897]

*** ERROR => NiPConnect: SiConnect failed for hdl 43/sock 15

    (SI_ECONN_REFUSE/239; I4; ST; 192.168.0.15:3900) [nixxi.cpp    2897]

*** ERROR => MsIAttachEx: NiBufConnect to dbciPRD/3900 failed (rc=NIECONN_REFUSED) [msxxi.c      713]

Sun Jul 28 17:25:42 2013

***LOG Q0I=> NiPConnect: 192.168.0.15:3900: connect (239: Connection refused) [nixxi.cpp 2897]

*** ERROR => NiPConnect: SiConnect failed for hdl 44/sock 15

    (SI_ECONN_REFUSE/239; I4; ST; 192.168.0.15:3900) [nixxi.cpp    2897]

*** ERROR => MsIAttachEx: NiBufConnect to dbciPRD/3900 failed (rc=NIECONN_REFUSED) [msxxi.c      713]

Sun Jul 28 17:26:02 2013

***LOG Q0I=> NiPConnect: 192.168.0.15:3900: connect (239: Connection refused) [nixxi.cpp 2897]

*** ERROR => NiPConnect: SiConnect failed for hdl 45/sock 15

    (SI_ECONN_REFUSE/239; I4; ST; 192.168.0.15:3900) [nixxi.cpp    2897]

*** ERROR => MsIAttachEx: NiBufConnect to dbciPRD/3900 failed (rc=NIECONN_REFUSED) [msxxi.c      713]

Sun Jul 28 17:26:22 2013

***LOG Q0I=> NiPConnect: 192.168.0.15:3900: connect (239: Connection refused) [nixxi.cpp 2897]

*** ERROR => NiPConnect: SiConnect failed for hdl 46/sock 15

    (SI_ECONN_REFUSE/239; I4; ST; 192.168.0.15:3900) [nixxi.cpp    2897]

Sun Jul 28 17:26:42 2013

***LOG Q0I=> NiPConnect: 192.168.0.15:3900: connect (239: Connection refused) [nixxi.cpp 2897]

*** ERROR => NiPConnect: SiConnect failed for hdl 40/sock 15

    (SI_ECONN_REFUSE/239; I4; ST; 192.168.0.15:3900) [nixxi.cpp    2897]

*** ERROR => MsIAttachEx: NiBufConnect to dbciPRD/3900 failed (rc=NIECONN_REFUSED) [msxxi.c      713]

Sun Jul 28 17:27:02 2013

***LOG Q0I=> NiPConnect: 192.168.0.15:3900: connect (239: Connection refused) [nixxi.cpp 2897]

*** ERROR => NiPConnect: SiConnect failed for hdl 41/sock 15

    (SI_ECONN_REFUSE/239; I4; ST; 192.168.0.15:3900) [nixxi.cpp    2897]

*** ERROR => MsIAttachEx: NiBufConnect to dbciPRD/3900 failed (rc=NIECONN_REFUSED) [msxxi.c      713]

Sun Jul 28 17:27:22 2013

***LOG Q0I=> NiPConnect: 192.168.0.15:3900: connect (239: Connection refused) [nixxi.cpp 2897]

*** ERROR => NiPConnect: SiConnect failed for hdl 42/sock 15

    (SI_ECONN_REFUSE/239; I4; ST; 192.168.0.15:3900) [nixxi.cpp    2897]

*** ERROR => MsIAttachEx: NiBufConnect to dbciPRD/3900 failed (rc=NIECONN_REFUSED) [msxxi.c      713]

Sun Jul 28 17:27:42 2013

***LOG Q0I=> NiPConnect: 192.168.0.15:3900: connect (239: Connection refused) [nixxi.cpp 2897]

*** ERROR => NiPConnect: SiConnect failed for hdl 43/sock 15

    (SI_ECONN_REFUSE/239; I4; ST; 192.168.0.15:3900) [nixxi.cpp    2897]

*** ERROR => MsIAttachEx: NiBufConnect to dbciPRD/3900 failed (rc=NIECONN_REFUSED) [msxxi.c      713]

Sun Jul 28 17:28:02 2013

***LOG Q0I=> NiPConnect: 192.168.0.15:3900: connect (239: Connection refused) [nixxi.cpp 2897]

*** ERROR => NiPConnect: SiConnect failed for hdl 44/sock 15

    (SI_ECONN_REFUSE/239; I4; ST; 192.168.0.15:3900) [nixxi.cpp    2897]

*** ERROR => MsIAttachEx: NiBufConnect to dbciPRD/3900 failed (rc=NIECONN_REFUSED) [msxxi.c      713]

[Former Member · Accepted solution]

DEar All,

Our problem is solved after below activity

There was diff. version of "sapstartsrv" in "/usr/sap/PRD/ASCS00/exe"

and

"/sapmnt/PRD/exe". So I have done following activity to reolve error :

/sapmnt/PRD/exe/sapcpe pf=/usr/sap/PRD/SYS/profile/PRD_DVEBMGS01_dbciPRD

/sapmnt/PRD/exe/sapcpe pf=/usr/sap/PRD/SYS/profile/PRD_ASCS00

Thanks and Regards,

Nirav

[Former Member]

Hi Nirav,

Was there some change in the host/services-file or network configuration?

Check the /etc/hosts and /etc/services entry and check the result of uname -n, if this is "consistent".

Also check niping -v -H localhost, it should be consistent as well.

Also check your profiles for the host/port/services definition.

I'm curious why it is trying to use port 3900, because Dispatcher usually uses 32NN and Message Server 36NN. I saw also in your case, you are using CI with 01 (DVEBMGS01)? There are some "mixed" stuff here I think.

And check also R3trans -d to see if it does connect to the database successfully.

-->Earlier, validating the profile failed, see INSTANCE.OUT (/usr/sap/PRD/SYS/profile/PRD_DVEBMGS01_dbciPRD)

-->NiPConnect: 192.168.0.15:3900: connect (239: Connection refused)

-->NiBufConnect to dbciPRD/3900 failed (rc=NIECONN_REFUSED)

I hope this will help you.

Regards,

Andre

[Former Member]

Hi Nirav,

Please let me know the version of the SUM tool that you are using . I can see this error in SUM tools 1 SP06 and SP07

Note 1707161 - Central Note - Software Update Manager 1.0 SP06 [lmt_007]

Dual stack: Timeout after phase MAIN_UPTRANS/STARTSAP_PUPG due to longer start up of JAVA instance

In phase MAIN_UPTRANS/STARTSAP_PUPG, the Software Update Manager starts the ABAP and JAVA instance. If the starting up of the JAVA instance takes much time, the phase can fail, even though the ABAP instance might be started successfully and the JAVA instance might complete its succesful start up some minutes later.

In this case, the SAPup.ECO logfile in SUM/abap/log has at the end for example the following entry:

EXECUTING /usr/sap/<SID>/<instance>/exe/sapcontrol (sapcontrol) -p

NI_HTTP -host <hostname> -nr <instancenum> -function StartWait 1000

<date> <time>

Start

OK

<date> <time>

StartWait

FAIL: Timeout

The timeout can indicate that the start up just needs more time, such as 16,6 minutes based on the above-mentioned example entry in which the StartWait parameter is 1000, and the calculation is 1000/60.

As you usually know how long your systems needs for a starting up, you can use this calculation to get a hint if it's a timeout or a real start error.

In case it is just a timeout and the system starts, you can wait until the JAVA instance has started and then try the phase again using the Repeat-option.

Note 1732061 - Central Note - Software Update Manager 1.0 SP07 [lmt_008]

(Windows only:) Phase MAIN_UPTRANS/STARTSAP_PUPG: Start of AAS instances failed
During an update or upgrade of distributed and high availability systems, the start of additional application server instances fails in phase MAIN_UPTRANS/STARTSAP_PUPG. The reason is that the SAPGLOBALHOST parameter in the instance profiles cannot be analyzed.
To solve the issue, proceed as follows:

1. 3. Check the instance profiles of all additional application server instances for a line that starts with #SAPGLOBALHOST.
2. 4. Remove the # at the beginning of the line, because this sign changes the line to a comment so that the value of the parameter cannot be analyzed.
3. 5. Repeat phase MAIN_UPTRANS/STARTSAP_PUPG.

Additional checks:

Add parameter rdisp/ignore_gw_death = 1 it is useful when you have problems with your message server, in this way you can start SAP and troubleshout the system in an easier way.

Be sure to set sapms and listener in /etc/services:

sapmsSID 39NN/tcp #sap service for SID

lsnrSID 15NN/tcp #sap Oracle listener for SID

Please check that rdisp/msserv parameter is set or not, for some reason it is not set in the source system.

Thanks

Rishi Abrol

[Former Member]

Hi Rishi,

What you are writing about sap ms and listener ports are wrong!!

SAP Message Server default port is in the range of 36NN, and Oracle listener default installation is 1527, unless you change this.

See this doc: http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/4e515a43-0e01-0010-2da1-9bcc452c2...

Even on a newest version of SAP: http://help.sap.com/saphelp_nw73ehp1/helpdata/en/4e/c26cdc58e968b9e10000000a42189e/frameset.htm

And by the way, port 3900 is officially registered/used:

So please double-check your info before providing your answer.

Also Nirav is doing a EhP upgrade, not a new installation, so all entries on services/host-file should exist anyway, especially Oracle listener-port.

Regards,

Andre

[Former Member]

Hi Ander,

Its not a generate thing that I was talking here. The purpose of asking to check those things was as its listening on the port 3900 and if some clustering is done on the server and they have used specific port rather than standard.

any how to me more specific i know.

Thanks

Rishi Abrol

[Former Member]

Hi Nirav,

So why is it calling 3900 :

Port Number of the SAP Message Server:

If you do not specify a value, the default port number is used.

ABAP Message Server Port

There is an external message server port and an internal message server port.

The ABAP message server uses both the internal and the external message server

ports. The default profile contains the configuration for both message server ports.

The external message server port uses the parameter rdisp/msserv with default

value 36<nn>, where <nn> is the instance number of the ABAP message server

instance.

The internal message server port uses the parameter rdisp/msserv_internal

with default value 39<nn>, where <nn> is the instance number of the ABAP

message server instance

For more information about the parameters used for message server ports, see

SAP Note 821875.

Can you please check if its not occupied by some other service and the actual system ia not able to start on those port.

So please stop all the process and try to start the system manually and then rerun the upgrade process.

Thanks

Rishi abrol

[Manas_Vincit]

Hi Nirav ,

Please manually start System and repeat this Phase . Clean shared memory before starting .

Thanks

Manas Behra