Skip to Content
0

SAP HANA Mutlitenant - firs tenant shuts down when server reaches 70% memory usage

Mar 29, 2017 at 06:24 AM

75

avatar image

Hi guys,

Story:
I have HAVA 122.05 running on SLES 11.4 that has 512GB RAM.
I am running few tenant databases on it.

My main and most important tenant is HDB.

Problem:

For some unknown reason this HANA system can't run more than 70% of system RAM. It always runs around 69,8% utilisation.

Whenever server starts to work more intensive and memory usage reach 70,1% than first tenant shuts down automatically (well second, because first is SYSTEMDB)

- I have never set any limits.
- I have tried to adjust RAM utilisation by HDB by messing with

ALTER SYSTEM ALTER CONFIGURATION ('indexserver.ini', 'DATABASE', 'Tenent Db') SET ('memorymanager', 'allocationlimit') = 'memory in kb' WITH RECONFIGURE

but I was never able to force this system to exceed 70% of RAM utilisation.

Any ideas?

8i8ux.png (6.5 kB)
fmzgr.png (12.7 kB)
koqwg.png (21.2 kB)
10 |10000 characters needed characters left characters exceeded
* Please Login or Register to Answer, Follow or Comment.

4 Answers

Best Answer
Lukasz Konieczny Jul 06, 2017 at 10:40 AM
0

The best part was that there ware no traces of this events in logs. However with SAP Support we ware able to solve this mystery.

It turned out, that standard linux tools like htop or gnome system monitor does not provide information of real RAM utilisation. I thought that I had 30% of RAM available but it turned out that this was not true.
Real value will show you free -m command on OS level.

There also was an info about those events but on OS level, not in HANA itself.

This was Out-of-Memory situation(OOM) in OS. What it means, the memory is not enough to run all application including hana tenants in the system right now.
When OS ran into this, OS chose one of the victim who normally is the process using the largest of memory in the system. That is why HDB(which is the largest) was killed by OS every time.
This is a general concept of linux regardless of redhat or suse.

Conclusion:
Server was in fact running at 100% RAM utilisation, but this 30% was considered as "non-important stuff". Too bad that OS was taking my hdbindexserver as a junk that can be removed...

Solution:
Upgrading to latest HANA version (122.09) solved the problem.

Share
10 |10000 characters needed characters left characters exceeded
Lucas Oliveira
Mar 29, 2017 at 09:45 PM
0

Hi Lukasz,

Are you sure they shutdown? What does the indexserver traces of these tenants says? Any chances the kernel oom-killer jumped in and murdered one of the tenant indexserver processes?

BRs,

Lucas de Oliveira

Share
10 |10000 characters needed characters left characters exceeded
Lukasz Konieczny Apr 03, 2017 at 12:04 PM
0

Yes i'm sure.
RAM utilisation drops to about 40%, application servers crash and tenant ACTIVE_STATUS is 'Starting'.
Then I must shut it down with ALTER DATABASE statement and turn in on again.

Share
10 |10000 characters needed characters left characters exceeded
Lucas Oliveira
Apr 04, 2017 at 01:47 AM
0

Well, what does the indexserver traces of these tenants says by the time they get restarted?

Share
10 |10000 characters needed characters left characters exceeded