hdbindexserver and hdbxsengine not working

[former_member445524]

A moderator having wrongly rejected this post, in an e-mail
exchange with the SCN Moderation and Collaboration team
(KMM732464V46915L0KM), we have agreed that I will re-post it
with the following preamble:

  I hereby confirm that prior to asking this question I have
  searched the SCN forums for an answer.  I used 'HANS',
  'hdbindexserver', 'hdbxsengine', and 'crash' as keywords.

  Let me also stipulate that this question is not a
  duplicate of http://scn.sap.com/thread/3938580
  because it asks help with a different error.

My question follows:

In my HANA intallation the services hdbindexserver and
hdbxsengine are not starting either automatically or after

                        > HDB start

Could it be the reason that I am unable to connect to my
HANA instance from HANA Studio?  If so, where can I find
information about why those services failed to start?  They
are not mentioned in the log files under

                       /var/tmp/hdb_*

I am not sure if it is correct to run them manually, but
when I run hdbxsengine from under the "system administrator
user", I get:

Crash at /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/
FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_maint_rel/sys/src/
Basis/System/impl/SharedMemorySegment.cpp : 346 Reason: File
/var/tmp/.SHM_SAP_IPMM_Mutex_0028_no-sid could not be opened
/ created. Ensure existing, writable directory /var/lib/hdb.
RC:13 Permission denied

I am running HANA SPS10 on SLES 11 SP 4.

[bryan_bingay · Accepted solution]

Anton, did HANA ever run on this system?

What do you get when you run HDB info as sidam?

Not able to connect to the system with HANA studio will always fail if HANA is stopped. Connecting to the HANA system even if it is stopped should still allow you to connect as sidadm user. Then you can look at the diagnostic files. indexserver nameserver ect.... These should give you some more information on what is wrong. A full Linux file system could also cause HANA from not starting. 

[jitendra_kumar01 · Accepted solution]

Hi Anton,

As you mentioned at last  that the there might be either some file is not present or if the file present then it does not has sufficient permission. Can you check the diagnosis file of Indexserver and XS engine,nameserver or provide more info on that.

Regards

Jitendra

[former_member445524 · Accepted solution]

Bryan Bingay:

>Anton, did HANA ever run on this system?

It never did, for it is our brand-new server.  I installed
HANA onto it for the first time.

>What do you get when you run HDB info as sidam?

Here you are: http://pastebin.com/ERtkZLem .

>Not able to connect to the system with HANA studio will
>always fail if HANA is stopped. Connecting to the HANA
>system even if it is stopped should still allow you to
>connect as sidadm user.

Well, I can connect to the Linux machine and run the HANA
studio.  But when I try to add my instance via

                   Systems -> Add system

I get this error:

  The system cannot be reached.  The logon data could not be
  used.

Do you know which HANA program or service should be
listening to connections from HANA Studio?

>Then you can look at the diagnostic files. indexserver
>nameserver ect....  These should give you some more
>information on what is wrong.

If I choose "Connect anyway", I can view the diagnosis files
from Studio via

      System monitor -> my system -> Administration ->
      Diagnosis files

There are some twenty files there, and although I have
searched them using grep I didn't find any mention of
'hdbindexserver' or 'hdbxsengine'.  Do you perchance know
which file should contain errors related to the starting of
the two services in question?

>A full Linux file system could also cause HANA from not
>starting.

Free hard-disk space is abundant on my server.  I have just
check it.

[former_member445524 · Accepted solution]

Jitendra Kumar:

>As you mentioned at last  that the there might be either
>some file is not present or if the file present then it
>does not has sufficient permission.  Can you check the
>diagnosis file of Indexserver and XS engine,nameserver or
>provide more info on that.

Yes, I should have liked to, but I don't know where those
files might be.  The closest I have found is:

  Holder.cc(00158) : Waiting for the next indexserver startup
  Holder.cc(00206) : Terminated by request

in:
/usr/sap/NDB/HDB00/

[bryan_bingay · Accepted solution]

What user are you trying HDB start? Are you able to login as or switch to the siadm user? can you run HDB info and HDB version?

[bryan_bingay · Accepted solution]

How much memory does the server have? Anything less than 32GB and HANA will not start.

[former_member445524 · Accepted solution]

Jitendra Kumar:

>check the permission of /var/lib/hdb and change the
>ownership for sidadm if it is root .

It was root indeed.  Sorry, Michael -- the subdirectories
were owned by <sid>adm, and I didn't check the hdb directory
itself.

When I changed its owner to <sid>adm and ran
'hdhindexserver' it printed:

ndbadm
/hana/shared/NDB/exe/linuxx86_64/HDB_1.00.102.05.1455873859_2714180> ./dbindexserver
unclean shutdown of service instance with pid 24585.
cannot write to /usr/sap/NDB/HDB00/<host>/lock/hdbindexserver@30003.pid: Permision denied (13)

Changing the owner and group of the whole /usr/sap:

              chown -R ndbadm:sapsys /usr/sap

didn't help.

This message is mentioned in
http://scn.sap.com/thread/3656538
in connexion with the nameserver log, where futher and more
usefult information is present.  Could you tell me where
these per-service logs are stored?  I don't remember
explicitly specifying the log directory during the gui
installation, and neither could I find the answer in the
installation and administration guides.

Bryan Bingay:

>What user are you trying HDB start?

<sid>adm.

>Are you able to login as or switch to the siadm user? can
>you run HDB info and HDB version?

Sure.  I had already done HDB info and uploaded the result.
The current output is here:
http://pastebin.com/FC231JrB .

And here is what HDB version prints:

version:             1.00.102.05.1455873859
branch:              fa/newdb100_maint_rel
git hash:            830c2b6a5adee46f25adb36f7c1abc95b2a16ad0
git merge time:      2016-02-19 10:24:19
weekstone:           0000.00.0
compile date:        2016-02-19 10:44:08
compile host:        ld7272
compile type:        rel

>How much memory does the server have? Anything less than
>32GB and HANA will not start.

50 GB, but surely if insufficient memory were the culprit
there would be an appropriate error message in one of the
log files?

[bryan_bingay · Accepted solution]

Can you connect to the system with hdbsql? Try this.

hdbsql-i00  -n hostname or ip:30015 -u SYSTEM -p password

select* from dummy

[former_member445524 · Accepted solution]

Bryan Bingay:

>Can you connect to the system with hdbsql? Try this.
>
>hdbsql-i00 -n hostname or ip:30015 -u SYSTEM -p password

I tried:

  hdbsql -i00 -n localhost -u SYSTEM -p <hidden>

and got:

  * -10709: Connection failed (RTE:[89006]
  System call 'connect' failed, rc=111:Connection refused)

[former_member445524 · Accepted solution]

I believe that at this stage I need to answer the following
two simple questions:

  1.  In what file(s) is the start-up of HANA services
      logged?

  2.  Which HANA component is responsible for listening to
      incoming connections from HANA Studio, hdbsql, &c?

[bryan_bingay · Accepted solution]

Anton, it is obvious there is a configuration problem of the OS, HANA or both. I would verify the OS settings per SAP notes 1954788 and 1855805. Check the firewall, this has burned me in the past. I also would uninstall HANA and reinstall. Here are some helpful links.

Earlier Releases – SAP Help Portal Page

https://www.youtube.com/user/saphanaacademy

Good Luck

Bryan

[former_member445524 · Accepted solution]

Bryan Bingay:

>I would verify the OS settings per SAP notes 1954788 and
>1855805.

Thanks, I will.

>Check the firewall, this has burned me in the past.

Rather unlikely, for I am connecting to HANA from the same
machine, i.e. to localhost.

[former_member445524 · Accepted solution]

Bryan Bingay:

>I also would uninstall HANA and reinstall.

That did it.  A miracle and naught less!

[former_member183326]

Are you trying to start the DB with root user?

/var/lib/hdb.- Try looking here. I'm guessing some files are owned by the root user....it should be owned by <sid>adm.

It would be good if next time you actually include some snippets from the nameserver + indexserver.

[jitendra_kumar01]

check  the permission of  /var/lib/hdb and change the ownership for sidadm if it is root . Moreover you

can find more about the nameserver and indexserver  in the directory trace present inside /usr/sap/sid/<sid instance no>hostname/trace

[former_member445524]

Michael Healy:

>Are you trying to start the DB with root user?

What do you mean by starting the DB?  I installed HANA via
the GUI installer, which created a special <sid>adm user and
is starting the services automatially, supposedly from that
user.

>/var/lib/hdb. - Try looking here.  I'm guessing some files
>are owned by the root user....it should be owned by
><sid>adm.

Nope, they are owned by the <sid>adm user.

>It would be good if next time you actually include some
>snippets from the nameserver+indexserver.

Just let me know whence to get them.

[former_member445524]

Jitendra Kumar:

>Moreover you can find more about the nameserver and
>indexserver in the directory trace present inside
>/usr/sap/sid/<sid instance no>hostname/trace

Thanks.  I have found a crash dump for the indexserver
there:

  find . -exec grep -l indexserver {} \;

but nothing for 'xsengine'.  Any idea where that might be?

[jitendra_kumar01]

Hi Anton,

If you are able to connect the Hana system, then you can search in the filter  in the diagnosis file tab for XS engine trace. No matter system is running or not.

There might be another reason of  firewall issue or SAPUI5 Delivery unit content might not present.

[former_member445524]

Jitendra Kumar to Anton Shepelev:

>>Thanks.  I have found a crash dump for the indexserver
>>there:
>>
>>  find . -exec grep -l indexserver {} \;
>
>If you are able to connect the Hana system, then you can
>search in the filter  in the diagnosis file tab for XS
>engine trace. No matter system is running or not.

Those filters are by file name, not by content.

[former_member445524]

Michael Healy:

>IF you have a crash dump and the system is still currently
>down then open an incident with SAP and send me the
>incident number.

The crash dump I have, but the system is working after I
reinstalled HANA.  Thank you kindly.