SA CR 728597 / Linux Kernel direct i/o bug & huge pages

[Former Member]

Last year, April -> October, I asked the question about IQ supporting Huge Pages on Linux.  It was mentioned that under SA CR 728597 and Red Hat Bug 891857 that there was a bug in the Linux kernel handling of direct I/O while using transparent huge memory pages (a variant of Linux Huge memory pages). 

CR 728597:

This problem is related to a possible bug in the transparent huge pages (THP) feature introduced in these operating system versions.  Red Hat bug 891857 has been created to track this issue.

The problem can be triggered by calling an external environment, xp_cmdshell, or other procedure that causes a fork while other I/O is occurring.  A known limitation with the Linux kernel limits the use of fork while doing O_DIRECT I/O operations.  Essentially what can happen is that the data can come from or go to the wrong process’ memory after the fork.  SQL Anywhere performs O_DIRECT I/O operations according to the documented safe usage.  However, THP appears to cause further problems and the O_DIRECT I/O data comprising database page reads/writes appears to get lost.

http://scn.sap.com/thread/3338917 and http://froebe.net/blog/2013/06/17/does-anyone-have-any-details-on-redhat-linux-bug-891857/

Does anyone know the status of this ongoing FIVE year old issue?

jason

[former_member182258]

The bug was closed with this comment: SQL Anywhere now disables direct I/O if transparent huge pages are enabled.  A warning will be printed as the database file is being opened to indicate that direct I/O is disabled due to this bug.  This is similar to how SQL Anywhere handles file systems that do not support direct I/O.

[saroj_bagai]

My understanding is that for performance reasons 'Annonymous  huge pages'  should be disabled

[markmumy]

That's my understanding, too.  We had an issue at a customer site just this past week where Anon HP were enabled, as they are by default.  We saw that the OS was spending a lot of time in system.  When the kernel was reset to disallow Anon HP, the server was restarted and the IQ workload resubmitted.  Performance was increased and there was no more wasted system time.

Mark

[Former Member]

umm..  the performance penalty flies contrary to how Oracle, DB2, even ASE perform.  So what it sounds like is that instead of fixing the bug, SAP will ignore it.  Nice.

[markmumy]

Jason,

SAP is not ignoring the bug.  The bug is a Linux bug that must be fixed by the Linux community.  All we can do is impress upon that community that we would like it fixed.  Build our business case.  So far that has fallen on deaf ears at RedHat. 

Our choice was to either work around the bug by disabling direct I/O or to continue with both direct I/O and anonymous huge pages in place.  The latter is known to cause memory corruption.  What would you have us do until RHEL and Linux fix it?  Should we force our customers to run in a mode that can cause memory corruption and then database corruption?  Database consistency is above all else.  We have to work around the Linux bug so that we can maintain that consistency.

What performance penalty do you refer to?  Have you done any specific tests with IQ that shows transparent hugepages actually help?  Please share your results that show IQ is faster with transparent hugepages enabled.

You cannot compare IQ to ASE, Oracle, DB2, etc.  We all interact quite differently with the OS.  So much so that each engine has a different set of tunables and requirements for the OS.  What gains ASE may have with transparent hugepages has no bearing on IQ.

I can say that in the few limited tests I have done with transparent hugepages (including all the Guinness Records tests that I am working) I have not seen a performance difference for IQ with transparent hugepages on or off.

Unfortunately, all we (the entire IQ community) can do is take the issue up with RedHat or switch to SUSE where this is not an issue.

Mark

[markmumy]

I should have mentioned, too, that Oracle has actually issued an alert on this topic and that they recommend disabling transparent hugepages:

Disabling Transparent HugePages (RHEL6/OL6)

Starting from RHEL6/OL6, Transparent HugePages are implemented and enabled by default. They are meant to improve memory management by allowing HugePages to be allocated dynamically by the "khugepaged" kernel thread, rather than at boot time like conventional HugePages. That sounds like a good idea, but unfortunately Transparent HugePages don't play well with Oracle databases and are associated with node reboots in RAC installations and performance problems on both single instance and RAC installations. As a result Oracle recommends disabling Transparent HugePages on all servers running Oracle databases, as described in this MOS note

http://www.oracle-base.com/articles/linux/configuring-huge-pages-for-oracle-on-linux-64.php

https://support.oracle.com/epmos/faces/DocContentDisplay?id=1557478.1

The same type of performance issues are even being seen in the Hadoop world:

http://structureddata.org/2012/06/18/linux-6-transparent-huge-pages-and-hadoop-workloads/

Some are claiming the same performance hits on Greenplum on that last link.

Mark

[Former Member]

#1 :  RedHat is NOT the Linux community.  RedHat is simply a vendor with a distribution.  Has anyone contacted the Linux kernel team or other Linux distributions? (e.g. SuSE)

#2 : THP is just one possible option.  The other is huge memory pages.

#3 : SAP *is* ignoring the problem by not spending a little bit of time on it and proposing a patch.  Other DBMS vendors do so, why not SAP?  Even Microsoft and Apple submit patches.

If you go back to the original discussion, I brought up huge memory pages.  Oracle and IBM both heavily promote the use of huge memory pages.  From Oracle:  Very Large Memory and HugePages  When you're dealing with a lot of memory, huge pages can make a big difference in performance.

Perhaps it is a matter of still thinking Sybase has being independent and not utilizing the resources of SAP?  (SAP to direct i/o Linux team:  We'll sponsor X hours at Y per hour to fix this problem.  Any one up for it?)

Then again, it might be possible that SAP itself may not know how to approach the actual developers of the Linux kernel (open source developers I mean)?  It is very different than approaching another vendor.

jason

[Former Member]

SAP, what's the status on this bug?

[Former Member]

bump