NW 71 abap instance does not come up in phase STARTSAP_PUPG

[Former Member]

Dears,

we are upgrading our Solution Manaher from Ehp1 for Nw 70, to rel Nw.71.

The upgrade on the J2EE tab is finished successfully while on the ABAP tab the upgrade is stopped in phase STARTSAP_PUPG because the Abap instance fails to start.

Analyzing the logs into the DVEBMGS00\work directory we noticed the ABAP instance does not come up at all, while the J2EE instance starts regularly but fails later because it cannot connect to the Abap Ume.

The strange thing is that no one dev_w* file is generated into the DVEBMGS00\work ; all the files written here during the startsap are related to the J2EE instance.

The only log we can found for the Abap\instance come out from the SCS01\work directory and it's the "dev_ms";  I attach an exstract from this file :

[Thr 01] ***LOG Q01=> MsSInit, MSStart (Msg Server 1 9109540) [msxxserv_mt. 2278]
[Thr 01] SigISetDefaultAction : default handling for signal SIGCHLD
[Thr 01] load acl file = /usr/sap/ABC/SYS/global/ms_acl_info
[Thr 01] MsGetOwnIpAddr: my host addresses are :
[Thr 01]   1 : [xx.xx.xx.xx] saphost (HOSTNAME)
[Thr 01]   2 : [127.0.0.1] loopback (LOCALHOST)
[Thr 01] MsHttpInit: full qualified hostname = saphost

[Thr 01] HTTP logging is switch off
[Thr 01] set HTTP state to LISTEN
[Thr 01] *** HTTP port 8101 state LISTEN ***
[Thr 01] *** I listen to internal port 3901 (3901) ***
[Thr 01] *** HTTP port 8101 state LISTEN ***
[Thr 01] CUSTOMER KEY: >abcdefghilmno<
[Thr 01] build version=720.2011.10.26

[Thr 01] Tue Mar 27 11:21:14 2012
[Thr 01] MsJ2EE_AddLoggedInNode: add node [3536000] into logged in list
[Thr 01] MsJ2EE_AddLoggedInNode: add node [3536050] into logged in list

[Thr 01] Tue Mar 27 11:21:39 2012
[Thr 01] *** ERROR => MsSRead: NiBufReceive (rc=NIECONN_BROKEN) [msxxserv_mt. 12853]
[Thr 01] *** ERROR => MsSClientHandle: MsSRead C3 (J2EE3536050), MSEINTERN [msxxserv_mt. 5359]
[Thr 01] MsJ2EE_AddDisconnectedNode: add node [3536050] into disconnect list

[Thr 01] Tue Mar 27 11:21:47 2012
[Thr 01] MsJ2EE_CleanDisconnectedNodes: clean disconnect list
[Thr 01] MsJ2EE_AddLoggedInNode: node [3536050] is already in logged in list

[Thr 01] Tue Mar 27 11:21:58 2012
[Thr 01] *** ERROR => MsSRead: NiBufReceive (rc=NIECONN_BROKEN) [msxxserv_mt. 12853]
[Thr 01] *** ERROR => MsSClientHandle: MsSRead C3 (J2EE3536050), MSEINTERN [msxxserv_mt. 5359]
[Thr 01] MsJ2EE_AddDisconnectedNode: add node [3536050] into disconnect list

What's wrong ?

Regards

[former_member182307 · Accepted solution]

Hello,

It's like your message server in SCS01 can't start for some reason.

The one thing i find weird in  you log file is :

[Thr 01]   1 : [xx.xx.xx.xx] saphost (HOSTNAME)

As if your host  was unable to perform its own resolution.

You should have a look at your hosts file and see if saphost is resolved in it.

Regards,

Steve.

[Former Member · Accepted solution]

Thanks for the feedback.

The host file is ok, I tested it

I'm looking into all the J2EE log files produced during the startsap, in order to see if something could prevent the Abap instance to come up.

I found several suspicious items, but actually I'm not able to evaluate what is related with the error and what not; I found them in different files:

1)

....... 

Service tc~lm~nzdm~crrsrv started. (197 ms).
Mar 29, 2012 12:07:37...        com.sap.security.core.persistence [SAPEngine_System_Thread[impl:5]_74] Fatal: Initialization
of ABAP data source (com.sap.security.core.persistence.datasource.imp.R3Persistence) failed: "Connect to SAP gateway failed
Connect_PM  TYPE=A ASHOST=localhost SYSNR=00 GWHOST=localhost GWSERV=sapgw00 PCS=1

........

2)

......

[Thr 01] JStartupICheckFrameworkPackage: can't find framework package /usr/sap/TSM/DVEBMGS00/work/jstartup.jar
[Thr 01] JStartupICheckFrameworkPackage: can't find framework package /usr/sap/TSM/DVEBMGS00/work/jvmx.jar

.....

3)

............

[Thr 35726] Thu Mar 29 12:07:37 2012
[Thr 35726] JHVM_RegisterNatives: com.sap.mw.rfc.driver.CpicDriver
[Thr 35726] ***LOG Q0I=> NiPConnect: 127.0.0.1:3300: connect (79: A remote host refused an attempted connect operation.) [nixxi.cpp 2898]
[Thr 35726] *** ERROR => NiPConnect: SiConnect failed for hdl 1/sock 305
    (SI_ECONN_REFUSE/79; I4; ST; 127.0.0.1:3300) [nixxi.cpp    2898]
</verbosegc>
[Thr 35726] JLaunchIExitJava: exit hook is called (rc = -11113)
[Thr 35726] **********************************************************************
[Thr 35726] *** ERROR => The Java VM terminated with a non-zero exit code.
[Thr 35726] *** Please see SAP Note 943602 , section 'J2EE Engine exit codes'
[Thr 35726] *** for additional information and trouble shooting.
[Thr 35726] **********************************************************************
[Thr 35726] SigISetIgnoreAction : SIG_IGN for signal SIGCHLD
[Thr 35726] JLaunchCloseProgram: good bye (exitcode = -11113)

.............

4)

..........
Fatal: Couldn't complete initial communication!
Exception is : The socket name is not available on this system.
Mar 29, 2012 12:07:24... ...tup.impl1.dispatcher.R3StartupService [SAPEngine_System_Thread[impl:6]_35] Fatal: StackTrace is :
java.net.NoRouteToHostException: The socket name is not available on this system.
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:355)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:220)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:207)
at java.net.Socket.connect(Socket.java:479)
at java.net.Socket.connect(Socket.java:429)
at java.net.Socket.<init>(Socket.java:336)
at java.net.Socket.<init>(Socket.java:152)
at com.sap.engine.services.r3startup.impl1.dispatcher.R3StartupService.initializeR3Socket(R3StartupService.java:417)
at com.sap.engine.services.r3startup.impl1.dispatcher.R3StartupService.start(R3StartupService.java:150)
at com.sap.engine.services.r3startup.dispatcher.R3StartupService.start(R3StartupService.java:34)
at com.sap.engine.core.service630.container.ServiceRunner.startCommunicationServiceFrame(ServiceRunner.java:228)
at com.sap.engine.core.service630.container.ServiceRunner.run(ServiceRunner.java:158)
at com.sap.engine.frame.core.thread.Task.run(Task.java:64)
at com.sap.engine.core.thread.impl6.SingleThread.execute(SingleThread.java:82)
at com.sap.engine.core.thread.impl6.SingleThread.run(SingleThread.java:154)

Any suggestion please ?

regards

[Amitjaiswal]

Hi Robert ,

Can you please check the bindHost property is configured correctly  as per note 609603 Also please check whether you have  defined gw service correctly, in the etc/hosts and etc/services per attached note 52959. also Please check note 124562.

Also it sometime due to  Incorrect version of the JMON library So kindly do check oss note 1163719 as well

[former_member182307]

Hi,

The j2ee startup failure is a consequence of the failed startup of the ABAP stack.

The log file states there is an issue with the connection to the gateway process which should be listening on port 3300. And this may be normal as there is an issue in your ABAP stack startup.

Do you have any thing written in the dev_disp log file in the work directory of the ABAP stack ?

Thinking of this connection problem, Is there any firewall running on your host that could prevent the usage of port 3300 ?

Regards,

Steve.

[Former Member]

Hi,

unfortunately the dev_disp file is missing at all.

A part the sapstartsrv e sapstart log file all the others files present into the DVEBMGS00\work are for the J2EE instance.

The sapstartsrv log file  is :

  ---------------------------------------------------
trc file: "sapstartsrv.log", trc level: 0, release:
"720"
---------------------------------------------------
pid        2949216
[Thr 01] Tue Mar 27 12:00:32 2012
No halib defined => HA support disabled
CCMS agent initialization: return code 255.
Initializing SAPControl Webservice
Starting AutoRestart thread
AutoRestart thread started
Starting WebService SSL thread
Starting WebService thread
Webservice SSL thread started, listening on port 50014
Webservice thread started, listening on port 50013
Trusted http connect via Unix domain socket '/tmp/.sapstream50013'
enabled.
Trusted https connect via Unix domain socket '/tmp/.sapstream50014' enabled.
trusted unix domain socket user is starting SAP System at 2012/03/27 12:00:52
trusted unix domain socket user is starting SAP System at 2012/03/27
12:02:36
trusted unix domain socket user is stopping SAP System at 2012/03/27
12:20:23
trusted unix domain socket user is starting SAP System at 2012/03/29
12:07:02

The startsap log is:

SAP-R/3-Startup Program Rel 720 V2.0
(2008/11/15)
-------------------------------------------------
Starting at 2012/03/29 12:07:02
Startup Profile:
"/usr/sap/ABC/SYS/profile/START_DVEBMGS00_saphost"
Setup Environment Variables
---------------------------
(10551446) SETENV LD_LIBRARY_PATH=/usr/sap/ABC/DVEBMGS00/j2ee/os_libs:
(10551446) SETENV SHLIB_PATH=/usr/sap/ABC/DVEBMGS00/j2ee/os_libs:
(10551446) SETENV
LIBPATH=/usr/sap/ABC/DVEBMGS00/j2ee/os_libs:/usr/sap/ABC/DVEBMGS00/exe:/usr/lib:/lib:/usr/sap/ABC/SYS/exe/run:/oracle/client/10x_64/instantclient
Execute Pre-Startup Commands
----------------------------
(2752640) Local: rm -f jc.sapABC_DVEBMGS00
(2752642) Local: ln -s -f /usr/sap/ABC/DVEBMGS00/exe/jcontrol
jc.sapABC_DVEBMGS00
(2752644) Local: rm -f ig.sapABC_DVEBMGS00
(2752646) Local: ln -s -f /usr/sap/ABC/DVEBMGS00/exe/igswd_mt
ig.sapABC_DVEBMGS00
Starting Programs
-----------------
(2752650) Starting: local jc.sapABC_DVEBMGS00
pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS00_saptsm1 -DSAPSTART=1
(10551446) Waiting for Child Processes to terminate.
(3277000) Starting: local ig.sapABC_DVEBMGS00 -mode=profile
pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS00_saphost
(2752650) New Child Process created.
(2752650) Starting local Command:
Command:
jc.sapABC_DVEBMGS00     
pf=/usr/sap/TSM/SYS/profile/ABC_DVEBMGS00_saphost           -DSAPSTART=1
(3277000) New Child Process created.
(3277000) Starting local Command:
Command:
ig.sapABC_DVEBMGS00
           -mode=profile     
pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS00_saphost

[rupali_karbhari3]

Hi Roberto,

I hope you have referred SAP Note:

Note 1252365 - Upgrades based on SAP NetWeaver 7.1 including EHP 1

It also talks about prerequsits to take care before upgrade start:

1. Check the Parameter 'icm/max_conn'

2. Create Internal Service User with UME Role Administrator

3. Increased Space Requirements in the Database

4. Downloading Kernel DVDs for the Upgrade

Refer section :

Phase: STARTSAP_PUPG

Description: You are upgrading a dual-stack distributed system (with SCS instance running on a remote host) and phase STARTSAP_PUPG fails. You can find the following error message in the 'dev_icm' trace file:

*** ERROR => can't attach to message server (rc=-100)

Solution: Ensure that the remote SCS instance is started and repeat the phase.

On Windows platforms, you must install an SCS Windows service prior to starting the SCS instance. Proceed as described in the upgrade guide for Windows - step 3 in section "Phase DOWNTIME_END_".

[Former Member]

Hi , thanks for the feedback

Yes I checked the note Note 1252365 - Upgrades based on SAP NetWeaver 7.1 including EHP 1

The parameter icm/max_conn is not present at all, so the default is used.

We did not create the UME role anyway the error is diferent as from note 1349854.

It is not a Database problem space nor filesystem space problem.

We downloaded all the software available for the upgrade from Sapnet; by the way the upgrade is arrived to the Downtime module; if some cd was wrong I suppose the upgrade to be unable to arrive till this point.

I'm thinking to change also the /usr/sap/sapservices profile as for note 1252365 , despite t says to do it after the upgrade.

The main dubt is that is not clear how to do it:

to copy the whole content of the instance profile (which ? of the SCS01 or the DVEBMGS00 istance) in a new file called /usr/sap/sapservices or simply copying one of these under the /usr/sap.

regards

[rupali_karbhari3]

Hi Roberto,

Just a basic question , have you triggered EHP upgrade via CI server?

You must start it via CI

Check Note 1156185 - Central Note: Upgrade to Systems on SAP NetWeaver 7.1 EHP 1

• <instance-id>: This is the primary application server instance ID. Depending on your source release, determine it as follows:
  • SAP NetWeaver 2004: Open the file /usr/sap/<SAPSID>/<instance-name>/j2ee/cluster/bootstrap/bootstrap.properties and search for the value of the instance.prefix property.
  • SAP NetWeaver 7.0: Open the central instance profile /usr/sap/<SAPSID>/SYS/profile/<SAPSID>_<instance-name>_<host> and search for the value of the j2ee/instance_id entry.

[Former Member]

Hi

yes we run the upgrade from the CI ; this is a Solution Manager we do not have other application servers a part the CI.

Regards