Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » Huge Pages »

Huge Pages

php.internals

Rasmus Lerdorf

10 years ago
We have a bit of a problem with how Linux handles huge pages when you run out of them. I've been discussing it with Dmitry and he recently committed a way to disable Huge Pages in the main allocator to master: https://github.com/php/php-src/commit/945a661912612cdecd221cd126feda7d4335c33c Unfortunately it looks like Linux is very unhappy when a CoW happens to a huge page allocated in its parent process and it tries to re-map this page. If there are no available huge pages you get random seg faults/bus errors and everything blows up. I don't understand why Linux can't fall back to a regular page here in this case, but it doesn't. It seems like an OS bug to me, but nonetheless this affects mod_php/php-fpm since we have a huge page allocated before the fork, but it also affects any cli script that does a pcntl_fork(). As far as I can tell there is no way to fix it. We could try to make sure we never do a MAP_HUGETLB prior to the fork in mod_php/php-fpm but that still doesn't solve the pcntl_fork() case. Anatol, we should merge Dmitry's patch into PHP-7.0 and further, I think we should flip the default to zend_mm_use_huge_pages = 0 in it. If you have well-behaved code that never tries to allocate a lot of memory you can turn on huge pages, but I think we should default to safety here. Those SIGBUS storms are quite alarming and hard to figure out. There have also been some other issues with huge pages as per these bugs: https://bugs.php.net/70984 https://bugs.php.net/71272 https://bugs.php.net/71355 This is very much an expert feature that we need to document better and people should know what they are stepping in when they turn this feature on. -Rasmus

Anatol Belski

10 years ago
Hi Rasmus,
> -----Original Message----- > From: Rasmus [mailto:rasmus@lerdorf.com] > Sent: Sunday, March 20, 2016 5:29 AM > To: php-internals <internals@lists.php.net> > Subject: [PHP-DEV] Huge Pages > > We have a bit of a problem with how Linux handles huge pages when you run > out of them. > > I've been discussing it with Dmitry and he recently committed a way to disable > Huge Pages in the main allocator to master: > > > https://github.com/php/php- > src/commit/945a661912612cdecd221cd126feda7d4335c33c > > Unfortunately it looks like Linux is very unhappy when a CoW happens to a huge > page allocated in its parent process and it tries to re-map this page. If there are > no available huge pages you get random seg faults/bus errors and everything > blows up. I don't understand why Linux can't fall back to a regular page here in > this case, but it doesn't. It seems like an OS bug to me, but nonetheless this > affects mod_php/php-fpm since we have a huge page allocated before the fork, > but it also affects any cli script that does a pcntl_fork(). As far as I can tell there > is no way to fix it. We could try to make sure we never do a MAP_HUGETLB prior > to the fork in mod_php/php-fpm but that still doesn't solve the pcntl_fork() case. > > Anatol, we should merge Dmitry's patch into PHP-7.0 and further, I think we > should flip the default to zend_mm_use_huge_pages = 0 in it. > > If you have well-behaved code that never tries to allocate a lot of memory you > can turn on huge pages, but I think we should default to safety here. Those > SIGBUS storms are quite alarming and hard to figure out. > > There have also been some other issues with huge pages as per these > bugs: > > https://bugs.php.net/70984 > https://bugs.php.net/71272 > https://bugs.php.net/71355 > > This is very much an expert feature that we need to document better and people > should know what they are stepping in when they turn this feature on. >
Dmitry has already merged this patch into the 7.0 branch. I also know from Remi that Fedora doesn't play well with the huge pages. Huge pages require special system settings which are usually not enabled in OSes by default. Having it off by default in PHP would therefore not cause any functional disadvantage while addressing the stability concerns. Thus reversing the default setting sounds feasible, it could be already done for the upcoming 7.0.5 if there are no objections. Regards Anatol

Rasmus Lerdorf

10 years ago
On 03/20/2016 01:58 PM, Anatol Belski wrote:
> Dmitry has already merged this patch into the 7.0 branch. I also know from Remi that Fedora doesn't play well with the huge pages. Huge pages require special system settings which are usually not enabled in OSes by default. Having it off by default in PHP would therefore not cause any functional disadvantage while addressing the stability concerns. Thus reversing the default setting sounds feasible, it could be already done for the upcoming 7.0.5 if there are no objections.
Yes, and since the special system settings, namely setting vm.nr_hugepages is a system-wide shared setting and not PHP-specific it makes even more sense to not enable huge pages by default. If you happen to install PHP on a system that has vm.nr_hugepages set to non-zero for some other service on that machine, PHP is going blow up as soon as it runs out of huge pages. And the 2nd reason to disable huge pages by default is that, as you say, on most systems vm.nr_hugepages is going to be 0 but MAP_HUGETLB will be defined because every modern OS supports huge pages even if they are not actually enabled. But the allocator doesn't know that. It will try to do a MAP_HUGETLB mmap every single time here: https://github.com/php/php-src/blob/master/Zend/zend_alloc.c#L470 which is going to fail with an ENOMEM error and fall back to a non-HUGETLB mmap. So every alloc there incurs an extra syscall in the default case. So we have 3 things to do: 1. Default it to off 2. Provide a way to specify the huge page size as per bug 71355 3. Document the use of USE_ZEND_ALLOC_HUGE_PAGES and how to set vm.nr_hugepages appropriately along with a warning about what might happen if don't allocate enough 1 and 3 are trivial. 2 is a little trickier if we want to try to detect the page size. We could just not try and let the user specify it. Perhaps even use ZEND_ALLOC_HUGE_PAGES directly. As in: ZEND_ALLOC_HUGE_PAGES=2048 would turn on huge pages in the allocator with 2k pages. -Rasmus

Remi Collet

10 years ago
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Le 20/03/2016 05:29, Rasmus a écrit :
> We have a bit of a problem with how Linux handles huge pages when > you run out of them.
Yes, especially on old kernel (2.6.32 on RHEL-6) For the build in my repository : - - RHEL-6 : build with --disable-huge-code-pages - - RHEL-7 and Fedora : only enabled on x86_64 (php.ini)
> I think we should flip the default to zend_mm_use_huge_pages = 0 in > it.
+1 Remi. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlbvnskACgkQYUppBSnxahhifACgjys/mVvAwJEiT40njsSRnVYC ZrUAoMKxgxkdktKDZnNsLNUkqYUqv2o5 =nBDN -----END PGP SIGNATURE-----

Brian Moon

10 years ago
On 3/19/16 23:29 , Rasmus wrote:
> We have a bit of a problem with how Linux handles huge pages when you > run out of them. > > I've been discussing it with Dmitry and he recently committed a way to > disable Huge Pages in the main allocator to master: > > https://github.com/php/php-src/commit/945a661912612cdecd221cd126feda7d4335c33c > > > Unfortunately it looks like Linux is very unhappy when a CoW happens to > a huge page allocated in its parent process and it tries to re-map this > page. If there are no available huge pages you get random seg faults/bus > errors and everything blows up. I don't understand why Linux can't fall > back to a regular page here in this case, but it doesn't. It seems like > an OS bug to me, but nonetheless this affects mod_php/php-fpm since we > have a huge page allocated before the fork, but it also affects any cli > script that does a pcntl_fork(). As far as I can tell there is no way to > fix it. We could try to make sure we never do a MAP_HUGETLB prior to the > fork in mod_php/php-fpm but that still doesn't solve the pcntl_fork() case. > > Anatol, we should merge Dmitry's patch into PHP-7.0 and further, I think > we should flip the default to zend_mm_use_huge_pages = 0 in it. > > If you have well-behaved code that never tries to allocate a lot of > memory you can turn on huge pages, but I think we should default to > safety here. Those SIGBUS storms are quite alarming and hard to figure out. > > There have also been some other issues with huge pages as per these bugs: > > https://bugs.php.net/70984 > https://bugs.php.net/71272 > https://bugs.php.net/71355 > > This is very much an expert feature that we need to document better and > people should know what they are stepping in when they turn this feature > on. > > -Rasmus >
This explains a recent issue reported for GearmanManager on PHP 7 as well. PHP Warning: pcntl_fork(): Error 12 in /usr/local/share/gearman-manager/GearmanManager.php on line 929 Error 12 is a memory allocation error. https://github.com/brianlmoon/GearmanManager/issues/136
-- Brian. -------- http://brian.moonspot.net/