hdbnameserver error loading shared libraries libhdbpythonbase.so

[gadde_shrinivas]

Hi,

I have installed HANA Database version 2.00.020.00.1500920972 (server component only in single DB environment) on SLES 12. I am unable to connect via HANA Studio as I get an incorrect password message for 'SYSTEM' user id. Now, I am following the instructions from " SAP HANA Administration Guide" to reset this user's password but when I execute the "hananameserver" command I get the error:

./hdbnameserver: error while loading shared libraries: libhdbpythonbase.so: cannot open shared object file: No such file or directory

I can see that the file "libhdbpythonbase.so" is available under "/hana/shared/HDB/exe/linuxx86_64/HDB_2.00/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/" but not in PATH:

ls -l HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdbpythonbase.so
-r-xr-xr-x 1 hdbadm sapsys 8335888 Jul 24 15:11 HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdbpythonbase.so

So, I even added the path manually:

hdbadm > echo $PATH
/home/ec2-user/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/lib/mit/bin

hdbadm > export PATH=/usr/sap/HDB/HDB02/exe:$PATH
hdbadm > echo $PATH
/usr/sap/HDB/HDB02/exe:/home/ec2-user/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/lib/mit/bin

But even after this, I get the same error.

Below is the list of my commands execution for your reference:

SESSION 01
------------
# su hdbadm
Password:
hdbadm > /usr/sap/HDB/HDB02/exe/sapcontrol -nr 02 -function StopSystem HDB

SESSION 02
------------
# su hdbadm
Password: 
hdbadm > /usr/sap/HDB/HDB02/hdbenv.sh
hdbadm > /usr/sap/HDB/HDB02/exe/hdbnameserver
./hdbnameserver: error while loading shared libraries: libhdbpythonbase.so: cannot open shared object file: No such file or directory

Can you please advise how I can resolve this? I checked other questions on this same topic but none of them helped.

Thanks!

[BJarkowski]

Do you have any values in the environment variable LD_LIBRARY_PATH? If not, please try to point to the directory with the missing files.

Also, you can execute the command ldd to check what underlying libraries are failing.

ldd libhdbpythonbase.so

[gadde_shrinivas]

Thanks for your answer.

Below is the output for "ldd libhdbpythonbase.so" command:

sh-4.3$ cd /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca
sh-4.3$ ldd libhdbpythonbase.so
        linux-vdso.so.1 (0x00007fff675ac000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fe94533a000)
        libhdbbasement.so => not found
        libcrypto_base.so => not found
        libhdbbasis.so => not found
        libaio.so.1 => /lib64/libaio.so.1 (0x00007fe945138000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe944f1b000)
        libhdblttbase.so => not found
        libutil.so.1 => /lib64/libutil.so.1 (0x00007fe944d18000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fe944b14000)
        libpam.so.0 => /lib64/libpam.so.0 (0x00007fe944905000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fe9446ca000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fe944341000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fe944044000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe943e2d000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fe943a8a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe94578d000)
        libaudit.so.1 => /usr/lib64/libaudit.so.1 (0x00007fe943867000)

And LD_LIBRARY_PATH was not maintained so I added it manually and then check above command again:

sh-4.3$ printenv | grep LD_LIBRARY_PATH
sh-4.3$ export LD_LIBRARY_PATH=/hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca
sh-4.3$ ldd libhdbpythonbase.so
        linux-vdso.so.1 (0x00007ffeb6253000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f7750776000)
        libhdbbasement.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdbbasement.so (0x00007f774f643000)
        libcrypto_base.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libcrypto_base.so (0x00007f7750c7c000)
        libhdbbasis.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdbbasis.so (0x00007f774eab8000)
        libaio.so.1 => /lib64/libaio.so.1 (0x00007f774e8b6000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f774e699000)
        libhdblttbase.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdblttbase.so (0x00007f774e4d9000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f774e2d6000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f774e0d2000)
        libpam.so.0 => /lib64/libpam.so.0 (0x00007f774dec3000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f774dc88000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f774d8ff000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f774d602000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f774d3eb000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f774d048000)
        libhdbbasementapi.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdbbasementapi.so (0x00007f7750c62000)
        libhdbtransactionmanager.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdbtransactionmanager.so (0x00007f774cfc4000)
        libhdbdataaccess.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdbdataaccess.so (0x00007f774b736000)
        libhdbdebuggingbase.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdbdebuggingbase.so (0x00007f7750c31000)
        libhdbunifiedtypes.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdbunifiedtypes.so (0x00007f774afd6000)
        libhdbsqlhint.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdbsqlhint.so (0x00007f7750bfc000)
        libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x00007f774add1000)
        libhdbversion.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdbversion.so (0x00007f7750bea000)
        libicuuc.so.53 => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libicuuc.so.53 (0x00007f774ac2e000)
        libicui18n.so.53 => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libicui18n.so.53 (0x00007f774a9d2000)
        libicudata.so.53 => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libicudata.so.53 (0x00007f7749549000)
        libv2catalystanalyzer.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libv2catalystanalyzer.so (0x00007f77494d3000)
        libv2catalystbasis.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libv2catalystbasis.so (0x00007f77494cd000)
        libv2tbclient.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libv2tbclient.so (0x00007f77494b1000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7750bc9000)
        libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00007f77492a5000)
        libaudit.so.1 => /usr/lib64/libaudit.so.1 (0x00007f7749082000)
        libhdbspatial.so => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libhdbspatial.so (0x00007f7748c4f000)
        libprotobuf.so.9 => /hana/shared/HDB/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeaca/libprotobuf.so.9 (0x00007f7748b1b000)

Now, when I execute "hdbnameserver" command I get a long output (uploaded here) and it doesn't seem like its starting normally. If I specify argument "-resetUserSystem" I still get the same output.

Can you please tell me what's wrong? This is a brand new installation and no changes have been done yet.

Thanks!

[BJarkowski]

OK, let's give it another try by setting additional environment variable SAP_RETRIEVAL_PATH pointing to

/usr/sap/HDB/HDB02/<hostname>

[gadde_shrinivas]

I added "SAP_RETRIEVAL_PATH" and seems like a different kind of error.

I have uploaded the latest output for "hdbnameserver" command "hdbnameserver2.log.md" file along with trace file "nameserver.crashdump.017572.trc" here .

Thanks!

[BJarkowski]

OK, now it's asking for DIR_INSTANCE.

This should point to /usr/sap/HDB/HDB02

But I'm concerned we have so many issues. Did you check you have fullfilled all prerequisites for HANA? Is HANA running at all? can you see indexserver process running after executing "ps aux"?

[gadde_shrinivas]

I have added entry for DIR_INSTANCE and now below is the output:

sh-4.3$ export DIR_INSTANCE=/usr/sap/HDB/HDB02

sh-4.3$ printenv | grep DIR_INSTANCE
DIR_INSTANCE=/usr/sap/HDB/HDB02

sh-4.3$ /usr/sap/HDB/HDB02/exe/hdbnameserver
Environment variable DIR_EXECUTABLE not set, using SAP_RETRIEVAL_PATH/../exe as fallback
unclean shutdown of service instance with pid 11460
initialization of communication failed!
prepare for shutting service down...
Disabling signal handler...
Stopping self watchdog...
Stopping request dispatcher...
Shutting service down...
Stopping threads...
Stopping responder...
Stopping communication...
Deleting self watchdog...
Deleting request dispatcher...
Deleting responder...
Deleting service...
Deleting threads...
Deleting pools...
Deleting configuration...
Removing pidfile...
shutdown is completed.

If I execute "hdbnameserver" again I get same output. Shouldn't the service start?

I did another install of this same version on another server and there as well I saw similar errors and no environment variables were maintained there either but I could still start/stop services from HANA Studio.

Thanks!

[BJarkowski]

Is your HANA instance starting at all? How much memory do you have available there?

[gadde_shrinivas]

Yes, it does. The server has 61GB memory.

[BJarkowski]

Could you please provide the log file for hdbnameserver please?

[gadde_shrinivas]

Uploaded all nameserver* trace files here .

[BJarkowski]

Thanks. I was able to see some errors there regarding SSL:

Can you check the value of configuration parameter "ssl" in [communication] section in global.ini file (

/usr/sap/<sid>/SYS/global/hdb/custom/config/global.ini)?

[gadde_shrinivas]

I do not see any setting for SSL. Below is the content of "global.ini" file:

# global.ini last modified 2017-11-16 06:04:09.375446 by hdbnameserver
[multidb]
mode = multidb
database_isolation = low
singletenant = yes

[persistence]
basepath_datavolumes = /hana/data/HDB
basepath_logvolumes = /hana/log/HDB

[system_information]
usage = development

Thanks!

[BJarkowski]

Hi,

I have checked the logs one more time and I can't see there anything more why the instance is not starting. As this is your new installation I would suggest to uninstall and install again the database.

Please also go through the HANA installation notes (depending on your operating system, I'm pasting here a note for SLES 12). Ensure that all required steps are executed.

2205917 - SAP HANA DB: Recommended OS settings for SLES 12 / SLES for SAP Applications 12

[gadde_shrinivas]

Hi Bartosz,

Thanks for spending time and looking into this.

I had followed the said note (# 2205917) for configuring my SLES before HANA installation. From the error, I am wondering if there is any other environment variable to point to the "CommonCrypto Lib"? Or, maybe I should pre-install any SAP Crypto Library? I couldn't find any other notes or posts on this topic so it feels like I missed something during install. I am now also in doubt if SLES 12 SP3 is supported for HANA installation since as per Note 2235581 only SLES until SP2 is supported.

Would you know if a standalone HANA server component alone works? I am wondering if I must include Studio, Client and/or XS to satisfy any dependencies?

Thanks!

[BJarkowski]

Hi,

yes, the issue with CommonCryptoLib also concerned me and this is why I asked you to check the ssl parameter in global.ini file. You could also check the existance of environment variable SECUDIR, which should point to the CommonCryptoLib library. It should be installed automatically together with HANA, so I don't think you have missed any step from the installation.

I'm also suprised that your HANA instance is starting and working without a problem. It looks like you encounter the issue only when trying to reset the password. Maybe it would be worth checking again if you can see the hdbindexserver process running in the background.

In the log I can see the line saying there was unclean shutdown of the instance. Did you by any chance try to run the hdbnameserver more than once? I just thought that it might work during the second execution? But it's more guessing now.

And finally I would like to answer your question. Yes, HANA can work as standalone component and it doesn't require Studio or XS to work.

[benpoon_123]

you are quire right. stop hdb, stop b1 service...then it works