Everything

php

recover from a segfault

php.internals

Nuno Lopes

20 years ago

Hi,

In the last days I've exchanged some e-mails with PCRE's author because of
one more bug that appeared in our database about segfaults in PCRE (related
to stack overflows).
PCRE can consume a lot of stack, because of backtracking (thus segfaulting
PHP). Yesterday I've discovered that when using the setrlimit() function,
most segfaults can be avoided
(http://mega.ist.utl.pt/~ncpl/php_pcre_stack_limits.txt)  :)

But I was thinking why we can't fail more silently from segfaults. A
recursive PHP function can easily segfault PHP. Why can't we catch the
signal and recover from the error? (and print a "stack overflow" message,
like java does).

I've done a little program for fun to show myself how to catch the SIGSEGV
signals and print a nice message.
(http://mega.ist.utl.pt/~ncpl/break-stack.html)

So, catching the signal is easy. What about recovery? Doesn't anyone has
experience in this area? Can this be done? (and in most
SAPIs/architectures?)


Nuno

P.S.: sorry if my "bright" idea is dumb and/or undoable :)

Ilia A.

20 years ago

Nuno Lopes wrote:

> In the last days I've exchanged some e-mails with PCRE's author because
> of one more bug that appeared in our database about segfaults in PCRE
> (related to stack overflows).
> PCRE can consume a lot of stack, because of backtracking (thus
> segfaulting PHP). Yesterday I've discovered that when using the
> setrlimit() function, most segfaults can be avoided
> (http://mega.ist.utl.pt/~ncpl/php_pcre_stack_limits.txt)  :)

This sounds like an interesting idea, I think we need to consider it
doing for PHP in general rather then jut when PCRE is being used. The
only thing is that rather then setting the stack to infinity, perhaps a
small value can be used ;-)

> I've done a little program for fun to show myself how to catch the
> SIGSEGV signals and print a nice message.
> (http://mega.ist.utl.pt/~ncpl/break-stack.html)
>
> So, catching the signal is easy. What about recovery? Doesn't anyone has
> experience in this area? Can this be done? (and in most
> SAPIs/architectures?)
>

After SEGV or any memory problem has happened the situation is
undefined. Another words you cannot reliably continue the operation of
the program. Which is why termination and dumping of core is the lesser
of all evils in this case.

Ilia

Markus Fischer

20 years ago

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

[response is at the bottom]

Ilia Alshanetsky wrote:

> Nuno Lopes wrote:
>> In the last days I've exchanged some e-mails with PCRE's author
>> because of one more bug that appeared in our database about segfaults
>> in PCRE (related to stack overflows).
>> PCRE can consume a lot of stack, because of backtracking (thus
>> segfaulting PHP). Yesterday I've discovered that when using the
>> setrlimit() function, most segfaults can be avoided
>> (http://mega.ist.utl.pt/~ncpl/php_pcre_stack_limits.txt)  :)
>
> This sounds like an interesting idea, I think we need to consider it
> doing for PHP in general rather then jut when PCRE is being used. The
> only thing is that rather then setting the stack to infinity, perhaps a
> small value can be used ;-)
>
>> I've done a little program for fun to show myself how to catch the
>> SIGSEGV signals and print a nice message.
>> (http://mega.ist.utl.pt/~ncpl/break-stack.html)
>>
>> So, catching the signal is easy. What about recovery? Doesn't anyone
>> has experience in this area? Can this be done? (and in most
>> SAPIs/architectures?)
>>
>
> After SEGV or any memory problem has happened the situation is
> undefined. Another words you cannot reliably continue the operation of
> the program. Which is why termination and dumping of core is the lesser
> of all evils in this case.

Would this "reliable execution" also apply to creating the stack
overflow message and passing it the PHPs error handler?

I was just two days ago that we had, of course, an infinite recursion
bug in our software, but the application just segfaults. The hard part
was to find where it segfaults. If it would just give a tiny hint as to
where the problem might be, this would be so awesome (if, of course,
technical possible/feasable).

- - Markus
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFD/Azr1nS0RcInK9ARAk0KAJ4wnL2pRypacl0dqFjr2Bb5EN81KwCcDmKu
kiMDEVP6pB/jYov1DCB7JSU=
=zxbn
-----END PGP SIGNATURE-----

Nuno Lopes

20 years ago

> Nuno Lopes wrote:
>> In the last days I've exchanged some e-mails with PCRE's author because
>> of one more bug that appeared in our database about segfaults in PCRE
>> (related to stack overflows).
>> PCRE can consume a lot of stack, because of backtracking (thus
>> segfaulting PHP). Yesterday I've discovered that when using the
>> setrlimit() function, most segfaults can be avoided
>> (http://mega.ist.utl.pt/~ncpl/php_pcre_stack_limits.txt)  :)
>
> This sounds like an interesting idea, I think we need to consider it doing
> for PHP in general rather then jut when PCRE is being used. The only thing
> is that rather then setting the stack to infinity, perhaps a small value
> can be used ;-)

Using it in PCRE is expecialy important! The new PCRE release (6.6) include
a protection against stack recursion overflows, but the usual linux stack
runs out of space before the protection code is triggered :) (although this
doesn't apply on windows, I think).
My idea was to do some testing and set the stack size to a reasonable limit
that would allow the pcre's protection code to kick in before the segfault.
But to do this, we must upgrade PCRE (I already did it on my pc).

>> I've done a little program for fun to show myself how to catch the
>> SIGSEGV signals and print a nice message.
>> (http://mega.ist.utl.pt/~ncpl/break-stack.html)
>>
>> So, catching the signal is easy. What about recovery? Doesn't anyone has
>> experience in this area? Can this be done? (and in most
>> SAPIs/architectures?)
>>
>
> After SEGV or any memory problem has happened the situation is undefined.
> Another words you cannot reliably continue the operation of the program.
> Which is why termination and dumping of core is the lesser of all evils in
> this case.

After reading some stuff, I think that recovering is a bit undoable (I
previously though that messing with the stack pointer would be easy...).
But setting the handler to run on a separate stack seems to be a clean and
stable solution. Its all handled by the OS. And then we could produce a nice
error message and log it (if error_log is On).
I'll also try to produce a patch for this :)

Nuno

Wez Furlong

20 years ago

If we do do this, please make it optional; it's bad form when a module
or a library unilaterally changes a process-wide option or setting and
impacts a system that has already been tuned; the sysadmin may already
have set up appropriate limits.

--Wez.

On 2/21/06, Ilia Alshanetsky <ilia@prohost.org> wrote:

Michael Vergoz

20 years ago

Hi,

The problem is that setrlimit() need privilege to expand the process stack.
I don't think it's a good idea to use this system call to fix the
too-high-function-stack bug.
There is no really way to fix correctly this bug.
However I think we should re-examined the execution flow of PECL functions.
It is not normal PECL can eat all the stack.

-michael

----- Original Message -----
From: "Wez Furlong" <kingwez@gmail.com>
To: "Ilia Alshanetsky" <ilia@prohost.org>
Cc: "Nuno Lopes" <nlopess@php.net>; "PHPdev" <internals@lists.php.net>
Sent: Thursday, February 23, 2006 5:29 AM
Subject: Re: [PHP-DEV] recover from a segfault


If we do do this, please make it optional; it's bad form when a module
or a library unilaterally changes a process-wide option or setting and
impacts a system that has already been tuned; the sysadmin may already
have set up appropriate limits.

--Wez.

On 2/21/06, Ilia Alshanetsky <ilia@prohost.org> wrote:

> Nuno Lopes wrote:
> > PCRE can consume a lot of stack, because of backtracking (thus
> > segfaulting PHP). Yesterday I've discovered that when using the
> > setrlimit() function, most segfaults can be avoided
> > (http://mega.ist.utl.pt/~ncpl/php_pcre_stack_limits.txt)  :)
>
> This sounds like an interesting idea, I think we need to consider it
> doing for PHP in general rather then jut when PCRE is being used. The
> only thing is that rather then setting the stack to infinity, perhaps a
> small value can be used ;-)

--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php

Jared Williams

20 years ago

> The problem is that setrlimit() need privilege to expand the
> process stack.
> I don't think it's a good idea to use this system call to fix
> the too-high-function-stack bug.
> There is no really way to fix correctly this bug.
> However I think we should re-examined the execution flow of
> PECL functions.

How about core functions too..   Using array_walk_recursive to undo magic_quotes creates a stack smashing exploit.

Jared

Michael Vergoz

20 years ago

Hi Jared,

It's true, array_walk_recursive() used the php_array_walk() and it can cause
some serious problem within the stack.
The way to fix this bug is to prevent the "out-of stack limit". I'll send a
patch if i can find a way to fix correctly this bug.

-michael

----- Original Message -----
From: "Jared Williams" <jared.williams1@ntlworld.com>
To: "'Michael Vergoz'" <mv-php@binarysec.com>; "'Wez Furlong'"
<kingwez@gmail.com>; "'Ilia Alshanetsky'" <ilia@prohost.org>
Cc: "'Nuno Lopes'" <nlopess@php.net>; "'PHPdev'" <internals@lists.php.net>
Sent: Friday, February 24, 2006 4:55 PM
Subject: RE: [PHP-DEV] recover from a segfault

Michael Vergoz

20 years ago

Hello,

ZDSP is a module to allow the Zend engine to know when it will go too much
far within the stack. In fact this module to allow to prevent a possible
segfault from the script.

If you type (php 5.1) :
php -r 'function foo() { foo(); } foo();'

You will get a segfault because foo() will go to read a too far stack
address and the kernel will send a bus error known under SIGSEGV.

Is it dangerous ? Not really, In fact it should be if web-hosters with a
'bad-client' who he will plays with the above example. The problem could be
more serious if the SAPI filter use in threaded environnement what would
cause the server halt.

The treatment the SIGSEGV signal is a nightmare there is a method to prevent
signal SIGSEGV by system call sigaltstack() which makes it possible to build
another stack.
However this system call is not portable and very dangerous.
"Executing codes after a SIGSEGV could be very dangerous".

ZDSP makes it possible to limit the damage and it's portable. In fact the
address of lower stack known is recorded (sapi_startup()) and at each
execution of function a control and carried out on the difference between
the current starting stack and that.

One considerer that the system will not consume more than 30% of the stack
at the time of the sapi_startup() and that PHP even will not consume more
than 20% of stack.
It remains 50% which would be the maximun load of Zend recursive execution
can take. It is simple, if one jump these 50%, ZDSP stop the script and
create an error.

CURRENT PATCHED FUNCTION
ZEND_API void execute(zend_op_array *op_array TSRMLS_DC)
static int php_array_walk(HashTable *target_hash, zval **userdata, int
recursive TSRMLS_DC)

You see function what need a prevention ? contact me mv-php@binarysec.com.

INSTALLATION
http://badcode.be/~descript/PHP/zend_dsp/

cd php-src
cp /PATH/zend_dsp.c Zend/
cp /PATH/zend_dsp.h Zend/
cd ..
cat PHP_5_1.zend-dsp.patch | patch -p0
cd php-src
./buildconf
./configure --enable-zend-dsp
make
./sapi/cli/php -r 'function foo() { foo(); } foo();'

enjoy

Michael Vergoz

----- Original Message -----
From: "Jared Williams" <jared.williams1@ntlworld.com>
To: "'Michael Vergoz'" <mv-php@binarysec.com>; "'Wez Furlong'"
<kingwez@gmail.com>; "'Ilia Alshanetsky'" <ilia@prohost.org>
Cc: "'Nuno Lopes'" <nlopess@php.net>; "'PHPdev'" <internals@lists.php.net>
Sent: Friday, February 24, 2006 4:55 PM
Subject: RE: [PHP-DEV] recover from a segfault

Stefan Esser

20 years ago

Basicly your patch does just try to automatically detect the maximum
execution depth limit, that can be manually set with xdebug or
hardened-php patch for quite a while now...

This kind of protection was rejected before to get into core and most
probably will be rejected again...

Stefan Esser

--
--------------------------------------------------------------------------
Stefan Esser                                               sesser@php.net
Hardened-PHP Project                         http://www.hardened-php.net/

GPG-Key                gpg --keyserver pgp.mit.edu --recv-key 0x15ABDA78
Key fingerprint       7806 58C8 CFA8 CE4A 1C2C  57DD 4AE1 795E 15AB DA78
--------------------------------------------------------------------------

Nuno Lopes

20 years ago

> Basicly your patch does just try to automatically detect the maximum
> execution depth limit, that can be manually set with xdebug or
> hardened-php patch for quite a while now...

which is not bad for the average user. I personally like the patch (although
it needs some tweaking), but the idea is there.

> This kind of protection was rejected before to get into core and most
> probably will be rejected again...

OK, but we can't keep allowing PHP to segfault because of stack issues..
If you don't want this patch, please considere the others you mention
(Xdebug or Hardened-PHP) to go into the core.

> and that your code will fail on systems where the stack is growing into
> the opposite direction.

thats not a real problem.. a simple configure program can detect such
things, like:

int main(int argc, char **argv) {
if ((void*)&argc > (void*)&argv) return 0; /* stack grows up */
if ((void*)&argc < (void*)&argv) return 1; /* stack grows down */
return 3;
}


Nuno

P.S.: I already have a solution for the PCRE stack-related segfaults..

Ilia A.

20 years ago

Nuno Lopes wrote:

>> Basicly your patch does just try to automatically detect the maximum
>> execution depth limit, that can be manually set with xdebug or
>> hardened-php patch for quite a while now...
>
>
> which is not bad for the average user. I personally like the patch
> (although it needs some tweaking), but the idea is there.

I think we have to be very careful in applying solutions that are only
applicable to "average users".

Ilia

Michael Vergoz

20 years ago

Hello,

I reply to Nuno because i think its post is the more objective i have seen.
The stack grows up/down is easy to know as Nuno said.
What more ? 5 functions, 3 global vars, some checks into recursive function
that all... Nothing impressive and it's a pruposing. It's sure that i need
more time & a cvs account to find all user-function that would to use
recursivity.
It's just a base.
And as Nuno said we can't keep allowing PHP to segfault because of stack
issues. And as i said in threaded environnement the problem is more serious.

-michael

----- Original Message -----
From: "Nuno Lopes" <nlopess@php.net>
To: "Stefan Esser" <sesser@hardened-php.net>; "Michael Vergoz"
<mv-php@binarysec.com>
Cc: <Jared.Williams1@ntlworld.com>; "'Wez Furlong'" <kingwez@gmail.com>;
"'Ilia Alshanetsky'" <ilia@prohost.org>; "'PHPdev'"
<internals@lists.php.net>
Sent: Saturday, February 25, 2006 5:53 PM
Subject: Re: [PHP-DEV] Vote for Zend Deep Stack Prevention (ZDSP)

Ilia A.

20 years ago

Using recursive functions without any constraint is a bad programming
practice, period. You should always implement a pop/push stack in those
situations to avoid getting into infinite or near infinite recursion.

Ilia

Stefan Esser

20 years ago

beside the fact that I think the correct way to stop the script is
zend_bailout() and not php_handle_aborted_connection()
and that your code will fail on systems where the stack is growing into
the opposite direction.

Stefan Esser

Jeff Moore

20 years ago

On Feb 25, 2006, at 4:47 AM, Michael Vergoz wrote:

> You see function what need a prevention ? contact me
> mv-php@binarysec.com.

Hello,

How about one that needs less prevention?  Would this patch (or another
like it) allow for the elimination of the current recursion guard for
__get, etc?

http://groups.google.com/group/mailing.www.php-dev/search?
q=%22Why+not+allow+recursive+calls%22

Jeff

Rasmus Lerdorf

20 years ago

Nuno Lopes wrote:

> Hi,
>
> In the last days I've exchanged some e-mails with PCRE's author because
> of one more bug that appeared in our database about segfaults in PCRE
> (related to stack overflows).
> PCRE can consume a lot of stack, because of backtracking (thus
> segfaulting PHP). Yesterday I've discovered that when using the
> setrlimit() function, most segfaults can be avoided
> (http://mega.ist.utl.pt/~ncpl/php_pcre_stack_limits.txt)  :)
>
> But I was thinking why we can't fail more silently from segfaults. A
> recursive PHP function can easily segfault PHP. Why can't we catch the
> signal and recover from the error? (and print a "stack overflow"
> message, like java does).

Because catching these kinds of signals is a nightmare.  In this
particular instance it may work because your stack hasn't been trashed,
but in many other cases it simply isn't safe to try to execute further
code.  I don't know how many times I have had to go through and get rid
of some programmer's misguided idea that trying to catch and handle a
SEGV nicely was a good idea.  An infinitely chain of segfaults in your
segfault handler is impossible to avoid and is much much worse than just
dropping a core as fast as possible and exiting.  Any such handler also
tends to obscure the actual problem making it much more difficult to
examine the core file later.

-Rasmus