Skip to Content
0
May 06, 2016 at 05:33 PM

Out-Of-Memory Errors

2186 Views

Hi guys

Lately, we are experiencing several Out-of-Memory situations on our SAP Landscape, after it was migrated to SAP HANA Database

It could occur randomly in ERP, NW PO or BW, since all of them were migrated from SQL Server to HANA Database

Basically, SAP Instance freezes, and only option is to restart it.

Under alert it shows

preprocessor on host uyclsxxx port 31002 was restarted at 2000-01-01 00:00:00.0000000.

webdispatcher on host uyclsxxx port 31006 was restarted at 2000-01-01 00:00:00.0000000.

Memory Overview seems to be normal and there are no peaks thay might hint Memory is below required

I was able to obtain some diagnostic files and all seems to be related to HANA Server unable to allocate more memory.

Nameserver, Indexserver and webdispatcher Out-Of-Memory dumps all shows the same information.

Any idea on how to troubleshoot these out-ot-memory errors? Bear in mind it could occur at any time, even on non-working hours or weekend when there are no activity on the SAP System

[MEMORY_OOM] Information about current out of memory situation: (2016-05-04 21:38:05 562 Local)

OUT OF MEMORY occurred.

Host: uycls157.mdp.local

Executable: hdbnameserver

PID: 22961

Failed to allocate 32816 byte.

Out of memory reasons:

!:! <1> MemoryPool::allocate(big/huge) failure.

/sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <2> BigBlockAllocator failure, no precharge.

/sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <3> IPMM early exit.

/sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)

!:! <4> MemoryPool::allocate(big/huge) failure.

/sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (1416)

!:! <5> BigBlockAllocator failure, no precharge.

/sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/MemoryPool.cpp (2085)

!:! <6> IPMM early exit.

/sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC47HAPPY_rel_fa~newdb100_rel/sys/src/Basis/MemoryManager/impl/GlobalMemoryHandler.cpp (6866)

At most one OOM dump is written in approximately 86400s.

[OS_MEMORY] Operating system information about memory: (2016-05-04 21:38:26 465 Local)

/proc/meminfo:

MemTotal: 154912776 kB

MemFree: 1644660 kB

Buffers: 155040 kB

Cached: 37670000 kB

SwapCached: 0 kB

Active: 120563596 kB

Inactive: 30454160 kB

Active(anon): 113194280 kB

Inactive(anon): 22856080 kB

Active(file): 7369316 kB

Inactive(file): 7598080 kB

Unevictable: 0 kB

Mlocked: 0 kB

SwapTotal: 6291448 kB

SwapFree: 6291448 kB

Dirty: 8836 kB

Writeback: 0 kB

AnonPages: 113192916 kB

Mapped: 23249932 kB

Shmem: 22857644 kB

Slab: 749560 kB

SReclaimable: 624456 kB

SUnreclaim: 125104 kB

KernelStack: 5416 kB

PageTables: 417168 kB

NFS_Unstable: 0 kB

Bounce: 0 kB

WritebackTmp: 0 kB

CommitLimit: 83747836 kB

Committed_AS: 136213928 kB

VmallocTotal: 34359738367 kB

VmallocUsed: 542572 kB

VmallocChunk: 34359179772 kB

HardwareCorrupted: 0 kB

AnonHugePages: 0 kB

HugePages_Total: 0

HugePag

/proc/vmstat:

nr_free_pages 411165

nr_inactive_anon 5714020

nr_active_anon 28298570

nr_inactive_file 1899520

nr_active_file 1842329

nr_unevictable 0

nr_mlock 0

nr_anon_pages 28298229

nr_mapped 5812483

nr_file_pages 9456260

nr_dirty 2209

nr_writeback 0

nr_slab_reclaimable 156114

nr_slab_unreclaimable 31276

nr_page_table_pages 104292

nr_kernel_stack 677

nr_unstable 0

nr_bounce 0

nr_vmscan_write 433

nr_writeback_temp 0

nr_isolated_anon 0

nr_isolated_file 0

nr_shmem 5714411

numa_hit 3058504450

numa_miss 0

numa_foreign 0

numa_interleave 114388

numa_local 3058504450

numa_other 0

nr_anon_transparent_hugepages 0

pgpgin 2381614114

pgpgout 3146033750

pswpin 433

pswpout 433

pgalloc_dma 2

pgalloc_dma32 155436

pgalloc_normal 3065907965

pgalloc_movable 0

pgfree 3066476428

pgactivate 9823911

pgdeactivate 46976

pgfault 3140334036

pgmajfault 1262

pgrefill_dma 0

pgrefill_dma32 46976

pgrefill_normal 0

pgrefill_movable 0

pgsteal_dma 0

pgsteal_dma32 433

pgsteal_normal 2869225

pgsteal_movable 0

pgscan_kswapd_dma 0

pgscan_kswapd_dma32 2720

pgsc

/proc/sysvipc/shm:

key shmid perms size cpid lpid nattch uid gid cuid cgid atime dtime ctime rss swap

0 67010560 740 1024 22935 22943 1 1000 79 1000 79 1461791037 1461791037 1461791037 0 0

11001 67043329 600 268435456 22943 24417 7 1000 79 1000 79 1461792167 1461792167 1461791037 268435456 0

1446068590 67076098 660 117 22961 24369 6 1000 500 1000 79 1461792151 1461791363 1461791039 4096 0

351291229 67108867 660 1060911 22961 24369 24 1000 500 1000 79 1461792151 1461791363 1461791039 1060864 0

785145360 67141636 660 108 22961 24369 6 1000 500 1000 79 1461792151

1461791363 1461791040 4096 0

Attachments