Everything

php

FPM preloading of PHP files

php.internals

Jakub Zelenka

8 years ago

Hi Dmitry,

This is a bit different topic to FFI discussion so creating a new thread.
You wrote this in the FFI extension discussion:

At the same time, we will develop a technology to preload and reuse  PHP

> files across requests.
> And allow FFI there.

Have you been thinking about it in the FPM context? I'm asking as I have
been pondering with a related idea about some restructuring how things work
currently in the FPM. I will try to give a quick overview of the current
way how things work (for those who read this and are not familiar with it),
the problem that it is already causing and finally the idea that I have. :)

The thing is that at the moment there is just a master process that except
other things does the module init and spawns children. Each child is then
configured by the pool configuration which means that it can have a
different effective user id. It's simple and it works in most cases.
However there is an issue if it tries to access shared resources in other
pools running under different user which can happen due to the fact that
the initialization is done during MINIT by master (root user in such
case).  One example is the opcache that tries to kill lockers as described
in https://bugs.php.net/bug.php?id=74709 . I think that it's also not ideal
to do MINIT as a root from security point of view (the last security issue
is actually good example of that - https://bugs.php.net/bug.php?id=75605 ).
However not sure if there are some extension that depends on it.

As you can imagine, it would get even worse if you allow preloading of PHP
script in master so such implementation would be a no-go IMHO.

What I have been thinking about is to have a new process (I will call it a
pool manager) for each pool that would except other things spawn and manage
children and run under the same user. It would also do the MINIT and it
could possibly preload PHP files and load the libraries for FFI
(considering there would be a support in the engine for that - some kind of
partial execution or whatever you think would be best). Master would then
spawn and manage the pool managers as well as doing other things like log
handling. That would prevent the user mix up mentioned above - in the
example case, the opcache would have separate shared memory for each pool.

WDYT?

Cheers

Jakub

Dmitry Stogov

8 years ago

Hi Jakub,

I though not only about FPM, but a technology for any SAPI.

Pre-loaded classes and functions might be kept in CG() tables, in the same way as "internal" ones.

EG() constants and variables may be re-created at start of request, executing pre-loaded PHP code.

Few years ago, I developed something similar for PHP-5.

Thanks. Dmitry.

________________________________
From: jakub.php@gmail.com <jakub.php@gmail.com> on behalf of Jakub Zelenka <bukka@php.net>
Sent: Tuesday, April 17, 2018 5:23:30 PM
To: Dmitry Stogov
Cc: PHP internals list
Subject: FPM preloading of PHP files

Hi Dmitry,

This is a bit different topic to FFI discussion so creating a new thread. You wrote this in the FFI extension discussion:

At the same time, we will develop a technology to preload and reuse  PHP files across requests.
And allow FFI there.

Have you been thinking about it in the FPM context? I'm asking as I have been pondering with a related idea about some restructuring how things work currently in the FPM. I will try to give a quick overview of the current way how things work (for those who read this and are not familiar with it), the problem that it is already causing and finally the idea that I have. :)

The thing is that at the moment there is just a master process that except other things does the module init and spawns children. Each child is then configured by the pool configuration which means that it can have a different effective user id. It's simple and it works in most cases. However there is an issue if it tries to access shared resources in other pools running under different user which can happen due to the fact that the initialization is done during MINIT by master (root user in such case).  One example is the opcache that tries to kill lockers as described in https://bugs.php.net/bug.php?id=74709 . I think that it's also not ideal to do MINIT as a root from security point of view (the last security issue is actually good example of that - https://bugs.php.net/bug.php?id=75605 ). However not sure if there are some extension that depends on it.

As you can imagine, it would get even worse if you allow preloading of PHP script in master so such implementation would be a no-go IMHO.

What I have been thinking about is to have a new process (I will call it a pool manager) for each pool that would except other things spawn and manage children and run under the same user. It would also do the MINIT and it could possibly preload PHP files and load the libraries for FFI (considering there would be a support in the engine for that - some kind of partial execution or whatever you think would be best). Master would then spawn and manage the pool managers as well as doing other things like log handling. That would prevent the user mix up mentioned above - in the example case, the opcache would have separate shared memory for each pool.

WDYT?

Cheers

Jakub

Jakub Zelenka

8 years ago

Hi Dmitry

On Tue, Apr 17, 2018 at 5:44 PM, Dmitry Stogov <dmitry@zend.com> wrote:

> Hi Jakub,
>
>
> I though not only about FPM, but a technology for any SAPI.
>
> Pre-loaded classes and functions might be kept in CG() tables, in the same
> way as "internal" ones.
>
> EG() constants and variables may be re-created at start of request,
> executing pre-loaded PHP code.
>
> Few years ago, I developed something similar for PHP-5.
>

Yeah that's what I kind of meant by the engine support. :) My idea was more
about when it should be preloaded. For example one option would be to
preload it automatically if nothing has been preloaded as part of
php_execute_script / zend_execute_script. However that would mean to do it
for each child (basically use it for all request for the child process
life). It should work and might be a viable options for the initial
implementation but more optimal would be do it once for all children
running in the same pool which could be doable using the pool managers. I
think it would be great to have some kind of API so the SAPI can explicitly
preload the script.

Cheers

Jakub

Dmitry Stogov

8 years ago

Hi Jakub,

I thought about pre-loading explicitly specified files (or directories) on PHP startup, before forking worker processes.

As result all pre-loadd files should be available to all workers. Of course the should be kept in immutable form.

Thanks. Dmitry.

________________________________
From: jakub.php@gmail.com <jakub.php@gmail.com> on behalf of Jakub Zelenka <bukka@php.net>
Sent: Wednesday, April 18, 2018 8:18:33 PM
To: Dmitry Stogov
Cc: PHP internals list
Subject: Re: FPM preloading of PHP files

Hi Dmitry

On Tue, Apr 17, 2018 at 5:44 PM, Dmitry Stogov <dmitry@zend.com<mailto:dmitry@zend.com>> wrote:

Hi Jakub,

I though not only about FPM, but a technology for any SAPI.

Pre-loaded classes and functions might be kept in CG() tables, in the same way as "internal" ones.

EG() constants and variables may be re-created at start of request, executing pre-loaded PHP code.

Few years ago, I developed something similar for PHP-5.

Yeah that's what I kind of meant by the engine support. :) My idea was more about when it should be preloaded. For example one option would be to preload it automatically if nothing has been preloaded as part of php_execute_script / zend_execute_script. However that would mean to do it for each child (basically use it for all request for the child process life). It should work and might be a viable options for the initial implementation but more optimal would be do it once for all children running in the same pool which could be doable using the pool managers. I think it would be great to have some kind of API so the SAPI can explicitly preload the script.

Cheers

Jakub

Jakub Zelenka

8 years ago

Hi Dmitry,

On Fri, Apr 20, 2018 at 8:54 AM, Dmitry Stogov <dmitry@zend.com> wrote:

> Hi Jakub,
>
>
> I thought about pre-loading explicitly specified files (or directories) on
> PHP startup, before forking worker processes.
>
> As result all pre-loadd files should be available to all workers. Of
> course the should be kept in immutable form.
>
>
>

This is actually the reason why I'm thinking about the pool manager process
in this context. The current FPM architecture would make this quite
problematic. Currently the process that would have to preload the file
would be a master process because it's the one that forks worker processes.
The problem is that many (probably most) configurations have master running
as a root and a worker process uses setuid and setgid to set a different
user and group after the fork. I think that would be a source of many bug
reports if the resources created during preloading could not be accessed by
the application (worker executed files) because the permission issue. Also
it wouldn't be probably a good idea to give a root access to the preloading
as it could have some security implications. That's why I think it would be
better to do preloading in a pool manager process that I described before
(basically a process with the pool configured user and group that would
spawn workers).

Cheers

Jakub

Dmitry Stogov

8 years ago

This makes sense.

________________________________
From: jakub.php@gmail.com <jakub.php@gmail.com> on behalf of Jakub Zelenka <bukka@php.net>
Sent: Friday, April 20, 2018 1:04:01 PM
To: Dmitry Stogov
Cc: PHP internals list
Subject: Re: FPM preloading of PHP files

Hi Dmitry,

On Fri, Apr 20, 2018 at 8:54 AM, Dmitry Stogov <dmitry@zend.com<mailto:dmitry@zend.com>> wrote:

Hi Jakub,

I thought about pre-loading explicitly specified files (or directories) on PHP startup, before forking worker processes.

As result all pre-loadd files should be available to all workers. Of course the should be kept in immutable form.

This is actually the reason why I'm thinking about the pool manager process in this context. The current FPM architecture would make this quite problematic. Currently the process that would have to preload the file would be a master process because it's the one that forks worker processes. The problem is that many (probably most) configurations have master running as a root and a worker process uses setuid and setgid to set a different user and group after the fork. I think that would be a source of many bug reports if the resources created during preloading could not be accessed by the application (worker executed files) because the permission issue. Also it wouldn't be probably a good idea to give a root access to the preloading as it could have some security implications. That's why I think it would be better to do preloading in a pool manager process that I described before (basically a process with the pool configured user and group that would spawn workers).

Cheers

Jakub