Getting fd from stream

php.internals

Martijn van Duren

9 years ago
Hello internals@, I have an (exotic) case where I need to be able to get the filedescriptor from a previously opened stream (via stream_socket_pair) to hand over to the child process, which needs to reexecute. I saw that the php:// wrapper supports php://fd/X syntax, so I reckon there should be a way to extract the information as well from a stream as well. Unfortunately I don't seem to be able to find it. Can someone point me in the right direction? Sincerely, Martijn van Duren

Kalle Sommer Nielsen

9 years ago
Hi Martijn 2017-07-06 11:12 GMT+02:00 Martijn van Duren <php@list.imperialat.at>:
> Hello internals@, > > I have an (exotic) case where I need to be able to get the > filedescriptor from a previously opened stream (via stream_socket_pair) > to hand over to the child process, which needs to reexecute. > > I saw that the php:// wrapper supports php://fd/X syntax, so I reckon > there should be a way to extract the information as well from a stream > as well. Unfortunately I don't seem to be able to find it. > > Can someone point me in the right direction?
It seems like this is not exposed at all by PHP. I have CC'ed Sara as one of the old school streams persons, maybe she has some insights why it is not exposed or if it is worth exposing fileno() in 7.2+
-- regards, Kalle Sommer Nielsen kalle@php.net

Sara Golemon

9 years ago
On Thu, Jul 6, 2017 at 10:53 AM, Kalle Sommer Nielsen <kalle@php.net> wrote:
> 2017-07-06 11:12 GMT+02:00 Martijn van Duren <php@list.imperialat.at>: >> I have an (exotic) case where I need to be able to get the >> filedescriptor from a previously opened stream (via stream_socket_pair) >> to hand over to the child process, which needs to reexecute. >> >
"hand over to a child process" Child as in pcntl_fork()? Or a subprocess via exec() et. al.? For the former, the variable should (I think) carry over. For the latter, the stream number is going to be a meaningless value to the subprocess. ((e.g. stdin (fd=0) for the parent is a different source than stdin for the child, despite being the same number)). I'm not sure of your exact use case, but you might be able to use proc_open to spawn the child and supply the socket in the descriptor spec which will (I think, iirc) bind the internal socket to a new pipe between the parent and child. Though it's possible you'll need to maintain a proxy loop in the parent. 40% of the above is guesswork and conjecture, but maybe it'll spawn the right question. -Sara

Niklas Keller

9 years ago
> > On Thu, Jul 6, 2017 at 10:53 AM, Kalle Sommer Nielsen <kalle@php.net> > wrote: > > 2017-07-06 11:12 GMT+02:00 Martijn van Duren <php@list.imperialat.at>: > >> I have an (exotic) case where I need to be able to get the > >> filedescriptor from a previously opened stream (via stream_socket_pair) > >> to hand over to the child process, which needs to reexecute. > >> > > > "hand over to a child process" > Child as in pcntl_fork()? Or a subprocess via exec() et. al.? > > For the former, the variable should (I think) carry over. > For the latter, the stream number is going to be a meaningless value > to the subprocess. ((e.g. stdin (fd=0) for the parent is a different > source than stdin for the child, despite being the same number)). > > I'm not sure of your exact use case, but you might be able to use > proc_open to spawn the child and supply the socket in the descriptor > spec which will (I think, iirc) bind the internal socket to a new pipe > between the parent and child. Though it's possible you'll need to > maintain a proxy loop in the parent. > > 40% of the above is guesswork and conjecture, but maybe it'll spawn > the right question.
There's a (I think undocumented) feature that allow transferring sockets to other processes with the socket extension. https://github.com/amphp/aerys/commit/40fae01e4f5f82570a0bc3cb8ebbf1fee36dbb0b Bob implemented that, so maybe he can help you. But what's your exact use case? Regards, Niklas

Martijn van Duren

9 years ago
On 07/06/17 20:01, Niklas Keller wrote:
>> >> On Thu, Jul 6, 2017 at 10:53 AM, Kalle Sommer Nielsen <kalle@php.net> >> wrote: >>> 2017-07-06 11:12 GMT+02:00 Martijn van Duren <php@list.imperialat.at>: >>>> I have an (exotic) case where I need to be able to get the >>>> filedescriptor from a previously opened stream (via stream_socket_pair) >>>> to hand over to the child process, which needs to reexecute. >>>> >>> >> "hand over to a child process" >> Child as in pcntl_fork()? Or a subprocess via exec() et. al.?
The latter.
>> >> For the former, the variable should (I think) carry over. >> For the latter, the stream number is going to be a meaningless value >> to the subprocess. ((e.g. stdin (fd=0) for the parent is a different >> source than stdin for the child, despite being the same number)).
Not entirely, when I use stream_socket_pair I get 2 filedescriptors and two resources with possible 2 different IDs. E.g. fd 5 and 6 while the resources have id 8 and 9. When I do the fork I close one end in the parent and one end in the child. If I exec into another program I want to be able to say: we can communicate over this fd, so that the child program can hook into that specific file descriptor.
>> >> I'm not sure of your exact use case, but you might be able to use >> proc_open to spawn the child and supply the socket in the descriptor >> spec which will (I think, iirc) bind the internal socket to a new pipe >> between the parent and child. Though it's possible you'll need to >> maintain a proxy loop in the parent.
My use case is to set up a cluster of processes each with their own task and permission sets. These processes need to be able to communicate with each other over socket pairs, not just between the parent and the child, but also between children. Although the proc_open does solve some problems, it doesn't solve them all. E.g. I can't create socket pairs between children via this options, and socketpairs opened to the proc_open would be left dangling. Hence I want to set up all my socket pairs, pcntl_fork the child process, close the socket pair ends that are not owned by the child and execute the new process and specify the fds via commandline arguments. For processes that execute a new PHP process I would also prefer to wrap an existing fd in a stream (similar to fdopen in C), but I can live with the dup-ing of the file-descriptor in fopen(php://fd/N, X). So the end result would (somewhat) look like: +------+ |master| +------+ 3 7 / \ / \ 4 8 +------+ +------+ |child1|5-----------6|child2| +------+ +------+ instead of: 4568 +------+ |master| +------+ 3 7 / \ / \ 4 8 +------+ +------+ |child1|5-----------6|child2| +------+ +------+ 3678 3457
>> >> 40% of the above is guesswork and conjecture, but maybe it'll spawn >> the right question. > > > There's a (I think undocumented) feature that allow transferring sockets to > other processes with the socket extension. > > https://github.com/amphp/aerys/commit/40fae01e4f5f82570a0bc3cb8ebbf1fee36dbb0b
That's a big help on another project I have, so thank you for that hint. I've actually been looking for that option for quite some time.

Sara Golemon

9 years ago
On Fri, Jul 7, 2017 at 10:04 AM, Martijn van Duren <php@list.imperialat.at> wrote:
> My use case is to set up a cluster of processes each with their own task > and permission sets. These processes need to be able to communicate with > each other over socket pairs, not just between the parent and the child, > but also between children. >
Sounds like you're describing unix domain sockets (or NamedPipes if you're on Windows). -Sara

Martijn van Duren

9 years ago
On 07/07/17 16:38, Sara Golemon wrote:
> On Fri, Jul 7, 2017 at 10:04 AM, Martijn van Duren > <php@list.imperialat.at> wrote: >> My use case is to set up a cluster of processes each with their own task >> and permission sets. These processes need to be able to communicate with >> each other over socket pairs, not just between the parent and the child, >> but also between children. >> > Sounds like you're describing unix domain sockets (or NamedPipes if > you're on Windows). > > -Sara >
Yes, although tcp/ip are also possible with stream_socket_pair. The point is, regardless of unix - or tcp sockets, they still have an underlying file descriptor, which I need to specify to the child, so that they can start using it. Ergo, I need to find a way to export this from the stream into PHP. To put it in code: <?php $sp = stream_socket_pair(STREAM_PF_UNIX, STREAM_SOCK_STREAM, 0); echo (int) $sp[0]."\n"; ?> This returns 4 (the object id), instead of 3 (the fd). If I could get the fd, I could turn this into something like: <?php $sp = stream_socket_pair(STREAM_PF_UNIX, STREAM_SOCK_STREAM, 0); switch (pcntl_fork()) { case -1: exit(255); case 0: fclose($sp[1]); pcntl_exec("/usr/local/libexec/whatever", ["-i", (int) $sp[0]]); default: fclose($sp[0]); do_parent_code(); } ?> Where whatever can be anything from a PHP-script to a C-compiled binary.

Sara Golemon

9 years ago
On Fri, Jul 7, 2017 at 11:04 AM, Martijn van Duren <php@list.imperialat.at> wrote:
> On 07/07/17 16:38, Sara Golemon wrote: > Yes, although tcp/ip are also possible with stream_socket_pair. >
nit; socket_pair sockets are not AF_INET, they're just sockets.
> The point is, regardless of unix - or tcp sockets, they still have an > underlying file descriptor, which I need to specify to the child, so > that they can start using it. Ergo, I need to find a way to export this > from the stream into PHP. >
You misunderstand, I'm not saying to create a unix socket and try to pass that open pair to the child. I'm saying listen on a unix domain socket on the file system and tell the children what the path is. They can all individually then open streams to the parent using the path. Psuedo code: // parent $mysocket = 'unix://tmp/parent.sock'; $server = stream_socket_server($mysocket); spawn_child('child_proc --sock='.escapeshellarg($mysocket)); $child = stream_socket_accept($server); var_dump(fgets($child)); // string (8) "Hi mom!\n" // child $parentssocket = parseCliArg('sock'); $stream = stream_socket_client($parentssocket); fwrite($stream, "Hi mom!\n"); Then the children can each report their own listening sockets to the parent and the parent can provide a lookup for siblings to find each other and allow them to open sockets to each other. Trying to pass a meaningless number to the child that it can't do anything with isn't going to work. -Sara

Martijn van Duren

9 years ago
On 07/07/17 18:03, Sara Golemon wrote
>> The point is, regardless of unix - or tcp sockets, they still have an >> underlying file descriptor, which I need to specify to the child, so >> that they can start using it. Ergo, I need to find a way to export this >> from the stream into PHP. >> > You misunderstand, I'm not saying to create a unix socket and try to > pass that open pair to the child. I'm saying listen on a unix domain > socket on the file system and tell the children what the path is. > They can all individually then open streams to the parent using the > path.
That would be an extremely ugly solution. You would create overhead for 1, and 2, you leave open the possibility that an arbitrary application (bound to the filesystem permissions) allows to connect to it during the time the socket is listening. See [0] for what can go wrong with a similar solutions.
> > Psuedo code: > // parent > $mysocket = 'unix://tmp/parent.sock'; > $server = stream_socket_server($mysocket); > spawn_child('child_proc --sock='.escapeshellarg($mysocket)); > $child = stream_socket_accept($server); > var_dump(fgets($child)); // string (8) "Hi mom!\n" > > // child > $parentssocket = parseCliArg('sock'); > $stream = stream_socket_client($parentssocket); > fwrite($stream, "Hi mom!\n"); > > > Then the children can each report their own listening sockets to the > parent and the parent can provide a lookup for siblings to find each > other and allow them to open sockets to each other. > > Trying to pass a meaningless number to the child that it can't do > anything with isn't going to work.
That's the whole point. They're not meaningless numbers, they're the actual references that the kernel exposes to the process for file access: $ cd /tmp $ cat test.php <?php $sp = stream_socket_pair(STREAM_PF_UNIX, STREAM_SOCK_STREAM, 0); switch(pcntl_fork()) { case -1: exit(255); case 0: fclose($sp[1]); // 3 is known because of known open files in default shell pcntl_exec("/tmp/test", ["-i", "3"]); default: fclose($sp[0]); fwrite($sp[1], "hello world"); } ?> $ cat test.c #include <stdio.h> #include <stdlib.h> #include <unistd.h> int main(int argc, char *argv[]) { extern char *optarg; int ch; int fd = -1; int readbytes; char buf[sizeof("hello world")]; while ((ch = getopt(argc, argv, "i:")) != -1) { switch(ch) { case 'i': fd = atoi(optarg); break; default: fprintf(stderr, "wrong parameter\n"); exit(1); } } readbytes = read(fd, buf, sizeof(buf)); buf[readbytes] = '\0'; printf("read: %s\n", buf); } $ gcc ./test.c -o test && php ./test.php read: hello world $ cat test.c #include <errno.h> #include <string.h> #include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <sys/socket.h> int main(int argc, char *argv[]) { int sp[2]; char *fd; extern int errno; socketpair(AF_UNIX, SOCK_STREAM, 0, sp); switch(fork()) { case -1: exit(1); case 0: close(sp[1]); asprintf(&fd, "%d", sp[0]); execl("/usr/bin/env", "/usr/bin/env", "php", "/tmp/test2.php", "-i", fd, NULL); default: close(sp[0]); write(sp[1], "hello world", sizeof("hello world")); } } $ cat test2.php <?php $opts = getopt("i:"); $fd = null; foreach ($opts as $opt => $arg) { switch($opt) { case "i": $fd = fopen("php://fd/" . $arg, "r"); break; default: trigger_error("wrong parameter", E_USER_ERROR); } } $buf = fread($fd, 11); echo "read: " . $buf . "\n"; $ gcc ./test2.c -o ./test2 && ./test2 read: hello world Don't mind the crummy C here, it's for illustration purposes only. As you can see, these "meaningless numbers" allow me to connect two applications written in two completely different languages without setting up any external listening system. The problem here is that for test.php I had to hardcode the value 3 because I don't know how to retrieve it from $sp[0]. This only works in a default shell where only stdin, stdout and stderr (fd0, fd1 and fd2) are set. The kernel allocates the next free fd, which happens to have number 3. As soon as I would execute it like: php ./test.php < ./test.c it would fail, because fd3 would be taken by a handle to test.c and $sp[0] would be allocated on 4.
> -Sara >
[0] https://legalhackers.com/videos/MySQL-MariaDB-PerconaDB-PrivEsc-Race-CVE-2016-6663-5616-6664-5617-Exploits.html

Sara Golemon

9 years ago
On Fri, Jul 7, 2017 at 4:06 PM, Martijn van Duren <php@list.imperialat.at> wrote:
> blah blah blah >
Ignoring the brokenness of your design, the original ask: Exposing the underlying FD from a file stream? Sure. No particular objection to it, and the stream::set_option interface has the right mechanism to surface it in a general way. Find someone to RFC it, and I don't see why it won't show up in PHP 7.3. What you're able to do with it is your problem. -Sara

Ferenc Kovacs

9 years ago
On Thu, Jul 6, 2017 at 11:12 AM, Martijn van Duren <php@list.imperialat.at> wrote:
> Hello internals@, > > I have an (exotic) case where I need to be able to get the > filedescriptor from a previously opened stream (via stream_socket_pair) > to hand over to the child process, which needs to reexecute. > > I saw that the php:// wrapper supports php://fd/X syntax, so I reckon > there should be a way to extract the information as well from a stream > as well. Unfortunately I don't seem to be able to find it. > > Can someone point me in the right direction? > > Sincerely, > > Martijn van Duren > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > >
hi, would you need something like fildes_fileno() from https://github.com/Tyrael/php-fildes ? I never managed to put that up for pecl, not sure if it even compiles anymore with recent php versions
-- Ferenc Kovács @Tyr43l - http://tyrael.hu

Martijn van Duren

9 years ago
Hello Ferenc, On 07/07/17 19:49, Ferenc Kovacs wrote:
> > > On Thu, Jul 6, 2017 at 11:12 AM, Martijn van Duren <php@list.imperialat.at <mailto:php@list.imperialat.at>> wrote: > > Hello internals@, > > I have an (exotic) case where I need to be able to get the > filedescriptor from a previously opened stream (via stream_socket_pair) > to hand over to the child process, which needs to reexecute. > > I saw that the php:// wrapper supports php://fd/X syntax, so I reckon > there should be a way to extract the information as well from a stream > as well. Unfortunately I don't seem to be able to find it. > > Can someone point me in the right direction? > > Sincerely, > > Martijn van Duren > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > > > hi, > > would you need something like fildes_fileno() from https://github.com/Tyrael/php-fildes ? > I never managed to put that up for pecl, not sure if it even compiles anymore with recent php versions >
Thanks for the hint, but I'd like to refrain from external libraries as much as I can, especially for something simple as a simple number inside a data structure. I might also achieve something like this by eio from pecl, through e.g. eio_dup2, but it's just unrealistic overkill to ask for a full extension installation for just 1 one trivial piece of functionality.

Ferenc Kovacs

9 years ago
On Fri, Jul 7, 2017 at 10:14 PM, Martijn van Duren <php@list.imperialat.at> wrote:
> Hello Ferenc, > > On 07/07/17 19:49, Ferenc Kovacs wrote: > > > > > > On Thu, Jul 6, 2017 at 11:12 AM, Martijn van Duren < > php@list.imperialat.at <mailto:php@list.imperialat.at>> wrote: > > > > Hello internals@, > > > > I have an (exotic) case where I need to be able to get the > > filedescriptor from a previously opened stream (via > stream_socket_pair) > > to hand over to the child process, which needs to reexecute. > > > > I saw that the php:// wrapper supports php://fd/X syntax, so I reckon > > there should be a way to extract the information as well from a > stream > > as well. Unfortunately I don't seem to be able to find it. > > > > Can someone point me in the right direction? > > > > Sincerely, > > > > Martijn van Duren > > > > -- > > PHP Internals - PHP Runtime Development Mailing List > > To unsubscribe, visit: http://www.php.net/unsub.php > > > > > > hi, > > > > would you need something like fildes_fileno() from > https://github.com/Tyrael/php-fildes ? > > I never managed to put that up for pecl, not sure if it even compiles > anymore with recent php versions > > > Thanks for the hint, but I'd like to refrain from external > libraries as much as I can, especially for something simple as > a simple number inside a data structure. > > I might also achieve something like this by eio from pecl, through > e.g. eio_dup2, but it's just unrealistic overkill to ask for a full > extension installation for just 1 one trivial piece of functionality. > > > > -- > > Ferenc Kovács > > @Tyr43l - http://tyrael.hu >
sure, I just thought that you are asking for a solution and not proposing a new feature/function to the core.
-- Ferenc Kovács @Tyr43l - http://tyrael.hu