During Installation sapcontrol is not able to start SCS instance

[janh90]

Hey SAP-Gurus,
I am trying to install a fresh Java NW 7.50 System with SWPM SP18 Pl 6 on AIX with DB2 database.
Unfortunately, the installations hangs in step "starting Central instance". After some time of waiting I exited the SWPM process and tried to start the instance manually.
I killed sapstartsrv process for the SCS instance
I deleted the sapstream files in /tmp
Executed "cleanipc 50 remove"
Checked /usr/sap/sapservices - Instance is maintained correctly
Checked /etc/services Re-named folder /usr/sap/J02/SCS50/sec to sec_2 and renamed it back to sec (Because I had the same error for DEV System before and there this procedure made it work 🙂

Then I started the Service again: sapcontrol -nr 50 -function StartService J02 -worked

However, when I try to start the System with: sapcontrol -nr 50 -debug -function Start

[Thr 01] Thu Jan 12 14:14:54 2017
[Thr 01] NiIInit: allocated nitab (2048 at 11088a510)
[Thr 01] NiIHSBufInit: initialize hostname buffer (IPv4)
[Thr 01] NiHLInit: alloc host buf (100 entries)
[Thr 01] NiSrvLInit: alloc serv bufs (100 entries)
[Thr 01] NiISapServNameCheck: sapctrl50 not supported (len 9)
[Thr 01] NiPGetServInfoError: func NiPGetServByName; sys getaddrinfo; log 1; isErr 1; errorNo 8
[Thr 01] ***LOG Q0I=> NiPGetServByName: 'sapctrl50' not found: getaddrinfo [/bas/745_REL/src/base/ni/niuxi.c 1819]
[Thr 01] NiSrvLGetServNo: service name 'sapctrl50' not found by operating system
[Thr 01] NiInit3: NI already initializes (init=1;cur=2048)
[Thr 01] addrinfo of 'localhost':
[Thr 01] 0: 127.0.0.1:0 'loopback' RAW (0-2-3-0-16)
[Thr 01] NiHLGetNodeAddr: got hostname 'localhost' from operating system
[Thr 01] NiIGetNodeAddr: hostname 'localhost' = addr 127.0.0.1
[Thr 01] NiIGetServNo: servicename '55013' = port 55013
[Thr 01] NiICreateHandle: hdl 1 state NI_INITIAL_CON
[Thr 01] NiILocalCheck: local loopback address: 127.0.0.1
[Thr 01] NiIInitSocket: set default settings for new hdl 1/sock 4 (UD; ST)
[Thr 01] NiIBlockMode: set blockmode for hdl 1 FALSE
[Thr 01] NiITraceByteOrder: CPU byte order: big endian, network, high val..low val
[Thr 01] NiIConnectSocket: hdl 1 is connecting to /tmp/.sapstream55013 (timeout=-1)
[Thr 01] NiIConnectSocket: connection of hdl 1 established to /tmp/.sapstream55013
[Thr 01] NiIConnect: state of hdl 1 NI_CONNECTED
[Thr 01] NiIBlockMode: set blockmode for hdl 1 TRUE
[Thr 01] NiIWrite: hdl 1 SSLState 0
[Thr 01] NiIWrite: hdl 1 sent data (wrt=785,pac=1,RAW_IO)
[Thr 01] NiIRead: hdl 1 SSLState 0

When I do the same on my DEV System the Output is exactly the same, but the following lines are at the end:

...
[Thr 01] Thu Jan 12 14:18:38 2017
[Thr 01] NiIRead: hdl 1 received data (rcd=758,pac=1,RAW_IO)

I hope you have some further hints for me ;)
Thanks in advance
Jan

[janh90 · Accepted solution]

Hello all,

We fixed the issue 🙂

D02:

[Thr 3087] NiIRead: hdl 27 SSLState 0

[Thr 3087] NiIRead: hdl 27 received data (rcd=785,pac=1,RAW_IO)

trusted unix domain socket user is starting SAP System at 2017/01/23 17:01:58 [Thr 3087] *** ERROR => JsfCloseShm: Static shared memory is not initialized. [jsfxxshm.c 2061]

[Thr 3087] RqQDetach: Queue not initialized

[Thr 3087] ShmDelete2( 85 )

[Thr 3087] *** ERROR => ShmDelete2: Inv. Key=85 [shmux.c 2819]

[Thr 3087] ShmDelete2( 84 ) ......

_______________________________________________________________________________

J02:

[Thr 2057] NiIRead: hdl 17 SSLState 0

[Thr 2057] NiIRead: hdl 17 received data (rcd=785,pac=1,RAW_IO)

is starting SAP System at 2017/01/23 16:17:12

[Thr 2057] *** ERROR => JsfCloseShm: Static shared memory is not initialized. [jsfxxshm.c 2061]

[Thr 2057] RqQDetach: Queue not initialized

[Thr 2057] ShmDelete2( 85 )

[Thr 2057] *** ERROR => ShmDelete2: Inv. Key=85 [shmux.c 2819]

[Thr 2057] ShmDelete2( 84 )

[Thr 2057] *** ERROR => ShmDelete2: Inv. Key=84 [shmux.c 2819]

At the end it probably was an issue with the parameter "service/protectedwebmethods".

After the SWPM failed to start the SCS, we set the entry to "NONE". Could be that this was not secure enough to start the System with sapcontrol. BUt this are just presumptions.

However, it is solved and I thank you guys. 🙂

Best regards Jan

[isaias_freitas · Accepted solution]

Nice to hear the good news!

Please close this question 😉

And from a security perspective, I strongly recommend that you set the parameter "service/protectedwebmethods" back to its previous value. Having it as NONE is not such a good idea 🙂

[janh90 · Accepted solution]

Hey Isaias,

sorry for the missunderstanding. The Parameter is now set to "service/protectedwebmethods = DEFAULT -J2EEGetProcessList -ReadLogFile -AbapReadSysLog" like it is set on our DEV system. And probably this solved the issue. With setting "NONE" it did not work, and also not with default value SDEFAULT it did not work.

Best regards

Jan

[isaias_freitas · Accepted solution]

Hi Jan,

No reason to apologize :-).

I just like to take into account the security aspect, and using NONE is not secure at all :D.

Best regards,

Isaías

[isaias_freitas]

Hello Jan,

Please try executing the following command:

sapcontrol -nr 50 -function OSExecute "/bin/ls" 0 0 LOG

In addition, what happens with the "... -function Start" command? Does it "hang forever", blocking the shell until you cancel it?

Best regards,

Isaías

[janh90]

Hey Bartosz,

I don't think that we have a problem with this error message. I checked other /etc/services files in our SAP landscapes and we never maintained such kind of entries.

And as I already mentioned also DEV and QA system are showing these errors, but these systems are working fine.

Thanks for the reply and best regards

Jan

[BJarkowski]

What about /etc/services?

[Thr 01] NiSrvLGetServNo: service name 'sapctrl50' not found by operating system

Best regards

Bartosz

[isaias_freitas]

Hi!

This can be safely ignored ;-).

Isaías

[janh90]

Hey Guys,

I am still at a loss, but I set the trace level for sapstartsrv to 3. So I am able to provide a logfile which might help you to investigate the error:
sapstartsrv.txt

Thanks and best regards

Jan Hofmann

[isaias_freitas]

Hello Jan,

From the log file:

[Thr 1800] NiIAccept: state of hdl 18 NI_ACCEPTED
[Thr 1800] NiIAccept: hdl 9 accepted hdl 18 from /tmp/.sapstream55013
[Thr 1800] NiIBlockMode: set blockmode for hdl 18 TRUE
[Thr 2829] NiIRead: hdl 18 SSLState 0
[Thr 2829] NiIRead: hdl 18 received data (rcd=785,pac=1,RAW_IO)

 is starting SAP System at 2017/01/20  7:41:58
[Thr 2829] *** ERROR => JsfCloseShm: Static shared memory is not initialized. [jsfxxshm.c  2061]

We see that the sapstartsrv received the connection and the 785 bytes sent by sapcontrol.

It logs that SAP is being started and then there is an error related to shared memory.

Is there any other instance running on this server (even if from a system other than J02) with instance number 50 as well?

If yes, this could be causing the issue. In fact, this is not supported (the instance number must be unique at a server; having the same instance number on different servers is ok).

If no, please stop all SAP processes for the instance 50 and clear the shared memory with "cleanipc 50 remove".

You can run cleanipc twice, and the second execution must indicate that there was nothing to cleanup.

If SAP still does not start, please execute "sapcontrol -nr 50 -function GetVersionInfo" and confirm that all files are from the same kernel release and patch level.

Best regards,

Isaías

[janh90]

Hello Isaias,

Thanks for your hint.
There is no other SAP System running with instance 50 on this host. The "cleanipc 50 remove" cleaned up all shared memories. And executing the command again nothind was cleaned, so everything as expected.

Executing the "sapcontrol -nr 50 -function GetVersionInfo", indeed we have some differneces in the patch Level.

But we patched disp+work yesterday to PL 323 to exclude a kernel bug. Do you think this is a Problem now?

GetVersionInfo
OK
Filename, VersionInfo, Time
/usr/sap/J02/SCS50/exe/sapstartsrv, 745, patch 323, changelist 1731759, RKS compatibility level 0, optU (Jan 13 2017, 11:37:34), rs6000_64, 2017 01 13 11:58:57
/usr/sap/J02/SCS50/exe/disp+work, 745, patch 323, changelist 1731759, RKS compatibility level 0, optU (Jan 13 2017, 11:37:34), rs6000_64, 2017 01 13 16:07:14
/usr/sap/J02/SCS50/exe/gwrd, 745, patch 323, changelist 1731759, RKS compatibility level 0, optU (Jan 13 2017, 11:37:34), rs6000_64, 2017 01 13 11:39:41
/usr/sap/J02/SCS50/exe/msg_server, 745, patch 323, changelist 1731759, RKS compatibility level 0, optU (Jan 13 2017, 11:37:34), rs6000_64, 2017 01 13 11:39:34
/usr/sap/J02/SCS50/exe/dboraslib.so, 745, patch 301, changelist 1718173, RKS compatibility level 0, optU (Nov 11 2016, 23:55:44), rs6000_64, 2016 11 11 23:58:38
/usr/sap/J02/SCS50/exe/dbdb2slib.so, 745, patch 301, changelist 1718173, RKS compatibility level 0, optU (Nov 11 2016, 23:55:44), rs6000_64, 2016 11 11 23:58:32
/usr/sap/J02/SCS50/exe/dbdb4slib.so, 745, patch 301, changelist 1718173, RKS compatibility level 0, optU (Nov 11 2016, 23:55:44), rs6000_64, 2016 11 12 01:06:45
/usr/sap/J02/SCS50/exe/dbdb6slib.so, 745, patch 301, changelist 1718173, RKS compatibility level 0, optU (Nov 11 2016, 23:55:44), rs6000_64, 2016 11 11 23:58:37
/usr/sap/J02/SCS50/exe/dbsybslib.so, 745, patch 301, changelist 1718173, RKS compatibility level 0, optU (Nov 11 2016, 23:55:44), rs6000_64, 2016 11 11 23:58:39
/usr/sap/J02/SCS50/exe/enserver, 745, patch 301, changelist 1718173, RKS compatibility level 0, optU (Nov 11 2016, 23:55:44), rs6000_64, 2016 11 11 23:59:34
/usr/sap/J02/SCS50/exe/icman, 745, patch 323, changelist 1731759, RKS compatibility level 0, optU (Jan 13 2017, 11:37:34), rs6000_64, 2017 01 13 11:39:42
/usr/sap/J02/SCS50/exe/sapwebdisp, 745, patch 323, changelist 1731759, RKS compatibility level 0, optU (Jan 13 2017, 11:37:34), rs6000_64, 2017 01 13 11:39:44
/usr/sap/J02/SCS50/exe/jcontrol, 745, patch 323, changelist 1731759, RKS compatibility level 0, optU (Jan 13 2017, 11:37:34), rs6000_64, 2017 01 13 11:37:39
/usr/sap/J02/SCS50/exe/jlaunch, 745, patch 323, changelist 1731759, RKS compatibility level 0, optU (Jan 13 2017, 11:37:34), rs6000_64, 2017 01 13 11:50:50
/usr/sap/J02/SCS50/exe/jstart, 745, patch 323, changelist 1731759, RKS compatibility level 0, optU (Jan 13 2017, 11:37:34), rs6000_64, 2017 01 13 11:55:54

Thanks and best regards

Jan

[isaias_freitas]

Hello Jan,

No problem about the version, as in this case you have applied the DW package as well.

Isaías

[janh90]

Hey Sriram,

also the ping is possible from other servers or PCs.

Thanks and best regards

Jan

[Sriram2009]

Hi Jan.

Are you able to ping the AIX host from external Pc's or any other servers? or is it firewall blocking?

BR

[janh90]

Hey Sriram,

thanks for your reply, but /etc/hosts file is also maintained correctly.

Best regards

Jan

[Sriram2009]

Hi Jan.

I think you have add the host name & IP address in \etc\hosts file and then try the installation

Regards

Sriram