cancel
Showing results for 
Search instead for 
Did you mean: 

SA CR 728597 / Linux Kernel direct i/o bug & huge pages

Former Member
0 Kudos
145

Last year, April -> October, I asked the question about IQ supporting Huge Pages on Linux.  It was mentioned that under SA CR 728597 and Red Hat Bug 891857 that there was a bug in the Linux kernel handling of direct I/O while using transparent huge memory pages (a variant of Linux Huge memory pages). 

CR 728597:

This problem is related to a possible bug in the transparent huge pages (THP) feature introduced in these operating system versions.  Red Hat bug 891857 has been created to track this issue.

The problem can be triggered by calling an external environment, xp_cmdshell, or other procedure that causes a fork while other I/O is occurring.  A known limitation with the Linux kernel limits the use of fork while doing O_DIRECT I/O operations.  Essentially what can happen is that the data can come from or go to the wrong process’ memory after the fork.  SQL Anywhere performs O_DIRECT I/O operations according to the documented safe usage.  However, THP appears to cause further problems and the O_DIRECT I/O data comprising database page reads/writes appears to get lost.

http://scn.sap.com/thread/3338917 and http://froebe.net/blog/2013/06/17/does-anyone-have-any-details-on-redhat-linux-bug-891857/

Does anyone know the status of this ongoing FIVE year old issue?

jason

Accepted Solutions (0)

Answers (3)

Answers (3)

former_member182258
Participant
0 Kudos

The bug was closed with this comment: SQL Anywhere now disables direct I/O if transparent huge pages are enabled.  A warning will be printed as the database file is being opened to indicate that direct I/O is disabled due to this bug.  This is similar to how SQL Anywhere handles file systems that do not support direct I/O.

saroj_bagai
Contributor
0 Kudos

My understanding is that for performance reasons 'Annonymous  huge pages'  should be disabled

markmumy
Advisor
Advisor
0 Kudos

That's my understanding, too.  We had an issue at a customer site just this past week where Anon HP were enabled, as they are by default.  We saw that the OS was spending a lot of time in system.  When the kernel was reset to disallow Anon HP, the server was restarted and the IQ workload resubmitted.  Performance was increased and there was no more wasted system time.

Mark

Former Member
0 Kudos

umm..  the performance penalty flies contrary to how Oracle, DB2, even ASE perform.  So what it sounds like is that instead of fixing the bug, SAP will ignore it.  Nice.

markmumy
Advisor
Advisor
0 Kudos

Jason,

SAP is not ignoring the bug.  The bug is a Linux bug that must be fixed by the Linux community.  All we can do is impress upon that community that we would like it fixed.  Build our business case.  So far that has fallen on deaf ears at RedHat. 

Our choice was to either work around the bug by disabling direct I/O or to continue with both direct I/O and anonymous huge pages in place.  The latter is known to cause memory corruption.  What would you have us do until RHEL and Linux fix it?  Should we force our customers to run in a mode that can cause memory corruption and then database corruption?  Database consistency is above all else.  We have to work around the Linux bug so that we can maintain that consistency.

What performance penalty do you refer to?  Have you done any specific tests with IQ that shows transparent hugepages actually help?  Please share your results that show IQ is faster with transparent hugepages enabled.

You cannot compare IQ to ASE, Oracle, DB2, etc.  We all interact quite differently with the OS.  So much so that each engine has a different set of tunables and requirements for the OS.  What gains ASE may have with transparent hugepages has no bearing on IQ.

I can say that in the few limited tests I have done with transparent hugepages (including all the Guinness Records tests that I am working) I have not seen a performance difference for IQ with transparent hugepages on or off.

Unfortunately, all we (the entire IQ community) can do is take the issue up with RedHat or switch to SUSE where this is not an issue.

Mark

markmumy
Advisor
Advisor
0 Kudos

I should have mentioned, too, that Oracle has actually issued an alert on this topic and that they recommend disabling transparent hugepages:

Disabling Transparent HugePages (RHEL6/OL6)

Starting from RHEL6/OL6, Transparent HugePages are implemented and enabled by default. They are meant to improve memory management by allowing HugePages to be allocated dynamically by the "khugepaged" kernel thread, rather than at boot time like conventional HugePages. That sounds like a good idea, but unfortunately Transparent HugePages don't play well with Oracle databases and are associated with node reboots in RAC installations and performance problems on both single instance and RAC installations. As a result Oracle recommends disabling Transparent HugePages on all servers running Oracle databases, as described in this MOS note

http://www.oracle-base.com/articles/linux/configuring-huge-pages-for-oracle-on-linux-64.php

https://support.oracle.com/epmos/faces/DocContentDisplay?id=1557478.1

The same type of performance issues are even being seen in the Hadoop world:

http://structureddata.org/2012/06/18/linux-6-transparent-huge-pages-and-hadoop-workloads/

Some are claiming the same performance hits on Greenplum on that last link.

Mark

Former Member
0 Kudos

#1 :  RedHat is NOT the Linux community.  RedHat is simply a vendor with a distribution.  Has anyone contacted the Linux kernel team or other Linux distributions? (e.g. SuSE)

#2 : THP is just one possible option.  The other is huge memory pages.

#3 : SAP *is* ignoring the problem by not spending a little bit of time on it and proposing a patch.  Other DBMS vendors do so, why not SAP?  Even Microsoft and Apple submit patches.

If you go back to the original discussion, I brought up huge memory pages.  Oracle and IBM both heavily promote the use of huge memory pages.  From Oracle:  Very Large Memory and HugePages  When you're dealing with a lot of memory, huge pages can make a big difference in performance.

Perhaps it is a matter of still thinking Sybase has being independent and not utilizing the resources of SAP?  (SAP to direct i/o Linux team:  We'll sponsor X hours at Y per hour to fix this problem.  Any one up for it?)

Then again, it might be possible that SAP itself may not know how to approach the actual developers of the Linux kernel (open source developers I mean)?  It is very different than approaching another vendor.

jason

Former Member
0 Kudos

SAP, what's the status on this bug?

Former Member
0 Kudos

bump