Error after kernel upgrade

[former_member645149]

Hi all,

I have upgraded kernel on a java which is a dual stack ECC system from

720_REL - 500

to

753_REL - 400.

After upgrade , I am getting below error after starting the system.

-------------------

4 ETW000 R3trans version 6.26 (release 753 - 29.11.18 - 20:18:11).
4 ETW000 unicode enabled version
4 ETW000 ===============================================
4 ETW000
4 ETW000 date&time : 17.10.2019 - 09:49:11
4 ETW000 control file: <no ctrlfile>
4 ETW000 R3trans was called as follows: R3trans -d
4 ETW000 trace at level 1 opened for a given file pointer
4 ETW000 [ dev trc,00000] Thu Oct 17 09:49:11 2019
4 ETW000 [ dev trc,00000] Loading DB library '/usr/sap/SID/SYS/exe/run/dboraslib.so' ...
4 ETW000 [ dev trc,00000] Library '/usr/sap/SID/SYS/exe/run/dboraslib.so' loaded
4 ETW000 [ dev trc,00000] Version of '/usr/sap/SID/SYS/exe/run/dboraslib.so' is "753.02", patchlevel (0.400)
4 ETW000 [dbsloci.c ,00000] *** ERROR => DbSlConnect(ci=NULL): missing connection info (vers=-1)
4 ETW000 [dbsloci.c ,00000] *** ERROR => DbSlConnect() failed with 18=DBSL_ERR_CONNECT
2EETW169 no connect possible: "DBMS = ORACLE --- dbs_ora_tnsname = 'SID'"
trans.log: END

----------------------

Kindly help.

Thanks,

Uday

[Sriram2009]

Hi Udaya

1. Could you share you SAP version detail?

2. If you are having kernel level 720_REL - 500 then max you can upgrade to kernel 7.22 not on 7.53

Regards

SS

[former_member645149]

Hi Sriram,

Thank you.

I have reverted the kernel back to 720_REL-500 now.

I am still getting below errors in jsmon and R3trans

sidadm> R3trans -d
This is R3trans version 6.24 (release 720 - 08.07.13 - 20:13:05 ).
unicode enabled version
2EETW169 no connect possible: "maybe someone set invalid values for DIR_LIBRARY ('/usr/sap/SID/SYS/exe/run') or dbms_type ('ORA')"
R3trans finished (0012).

JSMON:

----------

JStart Monitor Program

Enter [prefix [arg]] <section> [command [options]]
+-------+----------------+---------------------------------------------------+
|Section|Command |Description |
+-------+----------------+---------------------------------------------------+
|jsmon |exit/quit |exit JSMon program |
| |help |show help |
| |view |show instance overview |
| |repeat <command>|repeat <command> every 5 seconds |
| |trace <level> |set JSMon trace-level |
Fri Oct 18 06:18:29 2019> p v

JsfOpenShm failed: object not found
Cannot access shared memory.

-----------

[former_member645149]

Hi Sriram,

There is some configuration issues as well, the system was not up from April.

We have planned oracle upgrade on all systems in the landscape.

If I start the DB + sap, ORA processes are coming up in sidadm environment as well apart from orasid.

So , I tried killing ora processes as sidadm,(as they should only be there in orasid), db is also going down.

Please help.

Thanks,

Uday

[former_member645149]

[Oct 22, 2019 1:35:22 PM ] [Bootstrap module]> Problem occurred while performing synchronization.
[Oct 22, 2019 1:47:25 PM ] --------------------------------------------------
[Oct 22, 2019 1:47:25 PM ] Bootstrap MODE:
[Oct 22, 2019 1:47:25 PM ] <SINGLE INSTANCE>
[Oct 22, 2019 1:47:25 PM ] determined by parameter [ID515823150].
[Oct 22, 2019 1:47:25 PM ] --------------------------------------------------
[Oct 22, 2019 1:47:26 PM ] Exception occurred:
java.lang.ExceptionInInitializerError
at javax.crypto.Cipher.getInstance(Cipher.java:570)
at javax.crypto.Cipher.getInstance(Cipher.java:511)
at iaik.security.provider.IAIK.a(Unknown Source)
at iaik.security.provider.IAIK.addAsJDK14Provider(Unknown Source)
at iaik.security.provider.IAIK.addAsJDK14Provider(Unknown Source)
at com.sap.security.core.server.secstorefs.Crypt.<clinit>(Crypt.java:82)
at com.sap.security.core.server.secstorefs.SecStoreFS.setSID(SecStoreFS.java:158)
at com.sap.engine.core.configuration.impl.persistence.rdbms.DBConnectionPool.<init>(DBConnectionPool.java:113)
at com.sap.engine.core.configuration.impl.persistence.rdbms.PersistenceHandler.<init>(PersistenceHandler.java:38)
at com.sap.engine.core.configuration.impl.cache.ConfigurationCache.<init>(ConfigurationCache.java:149)
at com.sap.engine.core.configuration.bootstrap.ConfigurationManagerBootstrapImpl.init(ConfigurationManagerBootstrapImpl.java:214)
at com.sap.engine.core.configuration.bootstrap.ConfigurationManagerBootstrapImpl.<init>(ConfigurationManagerBootstrapImpl.java:46)
at com.sap.engine.bootstrap.Synchronizer.<init>(Synchronizer.java:76)
at com.sap.engine.bootstrap.Bootstrap.initDatabaseConnection(Bootstrap.java:474)
at com.sap.engine.bootstrap.Bootstrap.<init>(Bootstrap.java:156)
at com.sap.engine.bootstrap.Bootstrap.main(Bootstrap.java:946)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sap.engine.offline.OfflineToolStart.main(OfflineToolStart.java:81)
Caused by: java.lang.SecurityException: Cannot set up certs for trusted CAs
at javax.crypto.JceSecurity.<clinit>(JceSecurity.java:267)
... 21 more
Caused by: java.lang.SecurityException: Jurisdiction policy files are not signed by trusted signers!
at javax.crypto.JceSecurity.loadPolicies(JceSecurity.java:593)
at javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:524)
at javax.crypto.JceSecurity.access$700(JceSecurity.java:37)
at javax.crypto.JceSecurity$1.run(JceSecurity.java:258)
at java.security.AccessController.doPrivileged(Native Method)
at javax.crypto.JceSecurity.<clinit>(JceSecurity.java:234)
... 21 more

[Oct 22, 2019 1:47:26 PM ] [Bootstrap module]> Problem occurred while performing synchronization.

[former_member645149]

[Thr 772] Tue Oct 22 13:47:25:920 2019
[Thr 772] JHVM_RegisterNatives: com.sap.bc.krn.perf.PerfTimes
[Thr 772] JHVM_RegisterNatives: 8 of 9 methods in com.sap.bc.krn.perf.PerfTimes registered.

[Thr 3085] Tue Oct 22 13:47:26:823 2019
[Thr 3085] JLaunchIExitJava: exit hook is called (rc = 66)
[Thr 3085] **********************************************************************
[Thr 3085] *** ERROR => The Java VM terminated with a non-zero exit code.
[Thr 3085] *** Please see SAP Note 943602 , section 'J2EE Engine exit codes'
[Thr 3085] *** for additional information and trouble shooting.
[Thr 3085] **********************************************************************
[Thr 3085] SigISetIgnoreAction : SIG_IGN for signal SIGCHLD
[Thr 3085] JLaunchCloseProgram: good bye (exitcode = 66)
dev_bootstrap_ID515823150: END

[former_member645149]

[Thr 01] Tue Oct 22 15:41:31 2019
I [Thr 01] MtxInit: ANONYMOUS 0 0
F
F [Thr 01] Tue Oct 22 15:41:33 2019
F [Thr 01] *** ERROR => JsfOpenShm: ShmCreate(79, 4096, ATTACH) failed (rc = 3 SHM segment doesn't exist) [jsfxxshm.c 1739]
F
F [Thr 01] Tue Oct 22 15:41:42 2019
F [Thr 01] *** ERROR => JsfCloseShm: Static shared memory is not opened. [jsfxxshm.c 2105]
dev_jsmon: END

[former_member645149]

Hi SS,

Thanks.

R3trans -d gives 16 but theer are below logs in trans.log

[dbsloci.c ,00000] *** ERROR => ORA-00942 occurred at SQL stmt (parse error offset=917504)
4 ETW000 [ dev trc,00000] Dump statement cache (after SQL error):
4 ETW000 [ dev trc,00000] sc_p=111b10a28,no=19,idc_p=(nil),con=0,act=1,slen=45,smax=256,#vars=0,stmt=111cb09c0,table=XXXXT
4 ETW000 [ dev trc,00000] SELECT * FROM "XXXXT" WHERE TABNAME = 'SVERS';
4 ETW000 [dbntab.c ,00000] *** ERROR => NTAB: get_ntab_db_structure(): ddntt_column_check() returned dbsl error 99
4 ETW000 [ dev trc,00000] NTAB: db_ntab(): init_ntab failed, returning 4

[Sriram2009]

Hi Udaya.

Refer the SAP Note 1860519 - SAP fails to connect to the database

Regards

SS

[former_member645149]

error-during-kernel-upgrade.txt

please find trans.log attached.

[Sriram2009]

Hi Udaya.

1. First correct the R3trams-d then the java will start, could you attach the r3trans log file as text format.

2. Are you copied all the old kernel files in right path? because i am not good in Unix.

Regards

SS

[former_member645149]

SQL> shutdown immediate;
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
IBM AIX RISC System/6000 Error: 2: No such file or directory
SQL>

[former_member645149]

In available.log,

Unavailable 17.10.2019 07:58:22 - 17.10.2019 08:03:22
Unavailable 17.10.2019 08:05:38 - 17.10.2019 08:15:38
Unavailable 17.10.2019 08:20:39 - 17.10.2019 09:24:40
Unavailable 17.10.2019 09:27:01 - 17.10.2019 09:32:01
Unavailable 17.10.2019 09:37:25 - 17.10.2019 10:04:25
Unavailable 17.10.2019 10:09:58 - 17.10.2019 13:24:58

[former_member645149]

Found below in work dir,

[Thr 01] Thu Oct 17 12:48:39 2019 F [Thr 01] *** ERROR => JsfOpenShm: ShmCreate(79, 4096, ATTACH) failed (rc = 3 SHM segment doesn't exist) [jsfxxshm.c 1739] [Thr 01] Thu Oct 17 10:10:11 2019 [Thr 01] JStartupICheckFrameworkPackage: can't find framework package /usr/sap/SID/JCXX/exe/jvmx.jar [Thr 01] Thu Oct 17 10:10:11 2019 [Thr 01] SigISetDefaultAction : default handling for signal SIGCHLD [Thr 01] JStartupICheckFrameworkPackage: can't find framework package /usr/sap/SID/JCXX/work/jstartup.jar [Thr 01] JStartupICheckFrameworkPackage: can't find framework package /usr/sap/SID/JCXX/work/jvmx.jar

[former_member645149]

Also found,

[Thr 01] JStartupICreateProcess: fork process (pid 6291530)

[Thr 01] Thu Oct 17 10:10:21 2019
[Thr 01] *** ERROR => invalid return code of process [bootstrap_ID515823150] (exitcode = 255) [jstartxx.c 1647]
[Thr 01] JControlExecuteBootstrap: error executing bootstrap node [bootstrap_ID515823150] (rc = 255)
[Thr 01] JControlCloseProgram: started (exitcode = 255)
[Thr 01] JControlCloseProgram: good bye... (exitcode = 255)

[former_member645149]

Hi All,

In work directory - dev_bootstrap, I found below.

[Thr 01] Thu Oct 17 10:10:11 2019
[Thr 01] JStartupICheckFrameworkPackage: can't find framework package /usr/sap/SID/JCXX/exe/jvmx.jar

**********************************************************************
JStartupIReadSection: read node properties [bootstrap]
-> node name : bootstrap
-> node type : bootstrap
-> node execute : yes
-> java path : /usr/sap/SID/JCxx/exe/sapjvm_4
-> java parameters : -Djco.jarm=1
-> java vm version : 4.1.026 21.1-b02
-> java vm vendor : SAP Java Server VM (SAP AG)
-> java vm type : server
-> java vm cpu : ppc64
-> heap size : 512M
-> root path : /usr/sap/SID/JCXX/j2ee/cluster
-> class path : ./bootstrap/launcher.jar
-> OS libs path : /usr/sap/SID/JCXX/j2ee/os_libs
-> main class : com.sap.engine.offline.OfflineToolStart
-> framework class : com.sap.bc.proj.jstartup.JStartupFramework
-> registr. class : com.sap.bc.proj.jstartup.JStartupNatives
-> framework path : /usr/sap/SID/JCXX/exe/jstartup.jar:/usr/sap/SID/JCXX/exe/jvmx.jar
-> parameters : com.sap.engine.bootstrap.Bootstrap ./bootstrap ID5158231
-> debuggable : yes
-> debug mode : no
-> debug port : 60000
-> shutdown timeout : 120 sec.
**********************************************************************

Please check if this is of use.

Thanks,

Uday

[former_member645149]

Hi all,

Also, jsmon gives below error.

Thu Oct 17 12:48:35 2019> process view

JsfOpenShm failed: object not found
Cannot access shared memory.

Uday