Extending include path to support stream wrappers

php.internals

LAUPRETRE François (P)

20 years ago
Hello, I am currently implementing a packaging tool in PHP. If you know PHAR, it is the same kind of tool, which allows to implement a virtual file tree in a single physical file. In such a tool, every file path becomes a 'stream-wrapped' path and the problem comes from the difference in the way PHP handles these paths relatively to 'plain' paths, especially the fact that a 'stream-wrapped' path cannot be part of the include path. The first reason is that the ':' character is used by the stream wrapper and, also, as the Unix path separator. This one would be easy to solve, but it would be useless because the engine dispatches its 'open' requests to the wrappers before searching through the include paths. And, the include path is searched in the 'plain' wrapper code. I have proposed some ideas to correct this behavior as a 'Feature request' at http://bugs.php.net/bug.php?id=38006. Could you please comment on it ? Even if you think that it is total nonsense or a loss of time, I would really appreciate to have your opinion on this subject before starting to modify anything. Thanks & regards François PS: If we can solve this issue, I have the same for you (maybe more complex) about stream wrapper support in chdir/getcwd, and then the glob/fnmatch issue ;-)

Antony Dovgal

20 years ago
On 10.07.2006 18:40, LAUPRETRE François (P) wrote:
> Hello, > > I am currently implementing a packaging tool in PHP. If you know PHAR, it is the same kind of tool, which allows to implement a virtual file tree in a single physical file.
What's wrong with PHAR ? Why do you need another tool, which does pretty much the same?
-- Wbr, Antony Dovgal

LAUPRETRE François (P)

20 years ago
Antony Dovgal wrote:
> What's wrong with PHAR ? > Why do you need another tool, which does pretty much the same?
Nothing's wrong with PHAR... I am sorry but my question's purpose was not to discuss about why I am building such a tool. And the change I am suggesting would also benefit PHAR (if PHAR could set 'phar://' in the include path, it would become possible to keep relative paths in include statements). I could explain, of course, why phar doesn't fit my needs, but I would prefer to keep this topic on the PHP core aspect. The reason I don't start a topic to talk about the tool I am writing is that the documentation is not ready yet. And I want to write at least the user's guide before asking for your comments. The doc contains a section explaining why I chose not to use or enhance PHAR... But... OK, to resume, the main problem I met with PHAR was that I wanted to package libraries and not a whole application. I wanted to keep the tools and libraries I was calling from my main program as single package files. And PHAR cannot do that. I have several main scripts, which use several libraries, and I don't want to rebuild everything each time I change anything in a library. Think of it as static vs dynamic linking. So, I needed a tool which could 'mount' several package files on the same stream wrapper. It could have been done with phar but I would have had to change the URL syntax to include a 'mount point' as first element of the path. And, if I had chosen this way, I would still be discussing about compatibility problems... So, I created a new tool. If you want to read the documentation (just chapter heads at this time), please have a look at http://www.tekwire.net/redir.php/phk Regards Francois

Marcus Börger

20 years ago
Hello LAUPRETRE, have you ever looked at pecl/phar? It can do that already. best regards marcus Monday, July 10, 2006, 7:26:58 PM, you wrote:
> Antony Dovgal wrote: >> What's wrong with PHAR ? >> Why do you need another tool, which does pretty much the same?
> Nothing's wrong with PHAR... I am sorry but my question's purpose was not > to discuss about why I am building such a tool. And the change I am > suggesting would also benefit PHAR (if PHAR could set 'phar://' in the > include path, it would become possible to keep relative paths in include statements).
> I could explain, of course, why phar doesn't fit my needs, but I would > prefer to keep this topic on the PHP core aspect.
> The reason I don't start a topic to talk about the tool I am writing is > that the documentation is not ready yet. And I want to write at least the > user's guide before asking for your comments. The doc contains a section > explaining why I chose not to use or enhance PHAR...
> But... OK, to resume, the main problem I met with PHAR was that I wanted > to package libraries and not a whole application. I wanted to keep the > tools and libraries I was calling from my main program as single package > files. And PHAR cannot do that. I have several main scripts, which use > several libraries, and I don't want to rebuild everything each time I > change anything in a library. Think of it as static vs dynamic linking. > So, I needed a tool which could 'mount' several package files on the same > stream wrapper. It could have been done with phar but I would have had to > change the URL syntax to include a 'mount point' as first element of the > path. And, if I had chosen this way, I would still be discussing about > compatibility problems... So, I created a new tool.
> If you want to read the documentation (just chapter heads at this time), > please have a look at http://www.tekwire.net/redir.php/phk
> Regards
> Francois
Best regards, Marcus

LAUPRETRE François (P)

20 years ago
Marcus Boerger wrote :
> have you ever looked at pecl/phar? It can do that already.
Maybe. But I couldn't find any documentation about it and its features. And its 'Download latest' link is broken. I had to download the C source code from CVS. But, once again, may I humbly ask you to focus on my primary question, instead of arguing about a potential alternative to PHAR. And I also repeat that my question is of great interest for PHAR too. Thanks Francois

Pierre Joye

20 years ago
On 7/11/06, LAUPRETRE François (P) <francois.laupretre@ratp.fr> wrote:
> Marcus Boerger wrote : > > have you ever looked at pecl/phar? It can do that already. > > Maybe. But I couldn't find any documentation about it and its features. And its > 'Download latest' link is broken. I had to download the C source code from CVS.
It is not broken, there is no release. --Pierre

Robin Ericsson

20 years ago
On 7/11/06, Pierre <pierre.php@gmail.com> wrote:
> On 7/11/06, LAUPRETRE François (P) <francois.laupretre@ratp.fr> wrote: > > Maybe. But I couldn't find any documentation about it and its features. And its > > 'Download latest' link is broken. I had to download the C source code from CVS. > > It is not broken, there is no release.
Having a "Download latest" link that gives me a "File not found" seems broken to me. regards, Robin

Pierre Joye

20 years ago
On 7/11/06, Robin Ericsson <lobbin@gmail.com> wrote:
> On 7/11/06, Pierre <pierre.php@gmail.com> wrote: > > On 7/11/06, LAUPRETRE François (P) <francois.laupretre@ratp.fr> wrote: > > > Maybe. But I couldn't find any documentation about it and its features. And its > > > 'Download latest' link is broken. I had to download the C source code from CVS. > > > > It is not broken, there is no release. > > Having a "Download latest" link that gives me a "File not found" seems > broken to me.
Not if there is no "latest", the msg is clear. But that's not the point of my mail, only an explanation for Francois. --Pierre

Sara Golemon

20 years ago
> But, once again, may I humbly ask you to > focus on my primary question, instead of > arguing about a potential alternative to PHAR. > And I also repeat that my question is of > great interest for PHAR too. >
I think it's an interresting, yet ultimately inefficient idea. I would recommend that if you *really* need this functionality, that you first rethink what you're trying to accomplish. If you still havn't changed your mind there are a couple approaches you could take to handling this within your script rather than modifying core. -1 -Sara

LAUPRETRE François (P)

20 years ago
Thanks for your answer, I have been searching for an alternate way for more than one month now but I still don't see any way to resolve the problem of relative includes in packaged files. The goal, here, is to package an existing software with as few modifications as possible. It is the only way to have software teams adopt the package's format. And, when the software's author is OK to make his source compatible with the package system's requirement, his source files must be included without any modification. Setting a 'phar://' prefix on every file access everywhere does not really fit this constraint. If the code cannot be made compatible, as nobody wants to maintain and support two sets of sources, it is practically impossible to distribute both formats at the same time. I am not sure it is the best way to promote a new package system to the developers. Anybody having replaced every relative URLs in phpMyAdmin knows what I am talking about. I wanted to try it myself: I spent three hours on it yesterday and I still have errors. And, now, what can I do with it ? If I distribute it, who will accept to use it in a production environment ? The only alternative I found was to prefix every relative paths with 'dirname (_FILE_).DIRECTORY_SEPARATOR', but it works only for paths relative to the including file's directory, and once again, I don't think it would be a good argument to give to developers. I am sorry but, be it PHAR or another one, I humbly think we have a potential problem with the packaging process in general. If somebody has an idea to solve it without modifying the code, I'd really like to hear from him. Regards Francois -----Original Message----- From: Sara Golemon [mailto:pollita@php.net] Sent: Tuesday, July 11, 2006 4:29 PM To: LAUPRETRE François (P) Cc: internals@lists.php.net Subject: Re: Extending include path to support stream wrappers
> But, once again, may I humbly ask you to > focus on my primary question, instead of > arguing about a potential alternative to PHAR. > And I also repeat that my question is of > great interest for PHAR too. >
I think it's an interresting, yet ultimately inefficient idea. I would recommend that if you *really* need this functionality, that you first rethink what you're trying to accomplish. If you still havn't changed your mind there are a couple approaches you could take to handling this within your script rather than modifying core. -1 -Sara

Greg Beaver

20 years ago
LAUPRETRE François (P) wrote:
> Hello, > > I am currently implementing a packaging tool in PHP. If you know PHAR, it is the same kind of tool, which allows to implement a virtual file tree in a single physical file. > > In such a tool, every file path becomes a 'stream-wrapped' path and the problem comes from the difference in the way PHP handles these paths relatively to 'plain' paths, especially the fact that a 'stream-wrapped' path cannot be part of the include path. > > The first reason is that the ':' character is used by the stream wrapper and, also, as the Unix path separator. This one would be easy to solve, but it would be useless because the engine dispatches its 'open' requests to the wrappers before searching through the include paths. And, the include path is searched in the 'plain' wrapper code. > > I have proposed some ideas to correct this behavior as a 'Feature request' at http://bugs.php.net/bug.php?id=38006. Could you please comment on it ? Even if you think that it is total nonsense or a loss of time, I would really appreciate to have your opinion on this subject before starting to modify anything. > > Thanks & regards > > François > > PS: If we can solve this issue, I have the same for you (maybe more complex) about stream wrapper support in chdir/getcwd, and then the glob/fnmatch issue ;-)
allowing stream wrappers in include_path will open a potentially gigantic security hole, making sure that it is closed will be a huge challenge, and introduce a bit of overhead. In terms of what choices you have with pecl/phar and libraries, the best way to handle this is to use relative includes (require_once "phar://lib.phar/File.php") and have the user either include all the necessary phars in their php script or define an autoload that does it for them, just as they have to do with normal PHP files, or to bundle it all up in 1 phar. In other words, once you split a phar into multiple files, you've lost the primary benefit. Of course, we could move the discussion to pecl-dev if you'd like to talk about features in phar. I find it rather lame that your first response is "phar can't do this" when it is in alpha stage (meaning anything can be changed or added if needed) and has no releases... Greg

LAUPRETRE François (P)

20 years ago
Greg Beaver wrote :
> Allowing stream wrappers in include_path will open a potentially gigantic > security hole, making sure that it is closed will be a huge challenge, and > introduce a bit of overhead.
I don't understand where the security hole would be. I see the stream wrappers as virtual directory trees and I don't see why adding one of their sub-paths in the include path would be a security hole. Especially with the allow_url_fopen() security. The only difference would be to provide a base for relative URLs when there is none possible today (neither the cwd, nor the include path). I agree that authorizing stream wrapper paths in chdir() would have security implications but, for the include path, I don't understand.
> In terms of what choices you have with pecl/phar and libraries, the best way to > handle this is to use relative includes (require_once phar://lib.phar/File.php") > and have the user either include all the necessary phars in their php script or > define an autoload that does it for them, just as they have to do with normal > PHP files, or to bundle it all up in 1 phar. In other words, once you split a > phar into multiple files, you've lost the primary benefit. Of course, we could > move the discussion to pecl-dev if you'd like to talk about features in phar. > I find it rather lame that your first response is "phar can't do this" when it > is in alpha stage (meaning anything can be changed or added if needed) and has > no releases...
OK. I apoligize if I looked rude to you. But I tried to understand what features (current and future) PHAR provides. And I could not find any documentation. Almost nothing in pecl, nothing returned by google. Nothing on features, not even a small text to explain the pecl/phar's relation to the PEAR package. When I see in the CVS that the project and most files are 4 to 7 months old, and that the only available 'documentation' is the C source code, how can you blame me for not knowing exactly what your software provides ? I never said that I wanted to create a new package system just because it is fun. If I had found the information I needed about PHAR, I would have tried to enhance it. And it is still possible, of course. Talking about documentation, in order to present more in-depth which features are included in the tool I have written, I would be glad if you had a look at http://www.tekwire.net/redir.php/phk_doc, chapter 1: Getting started. For the rest, it is coming... Take it as a prototype or as feature ideas for PHAR...

Greg Beaver

20 years ago
LAUPRETRE François (P) wrote:
> Greg Beaver wrote : > > >>Allowing stream wrappers in include_path will open a potentially gigantic >>security hole, making sure that it is closed will be a huge challenge, and >>introduce a bit of overhead. > > > I don't understand where the security hole would be. I see the stream wrappers as > virtual directory trees and I don't see why adding one of their sub-paths in the > include path would be a security hole. Especially with the allow_url_fopen() > security. The only difference would be to provide a base for relative URLs when > there is none possible today (neither the cwd, nor the include path). I agree that > authorizing stream wrapper paths in chdir() would have security implications but, > for the include path, I don't understand.
Basically, as of right now, a URL must be explicitly included, which means something like: require_once 'PEAR.php'; cannot include anything but a local file. The existence of allow_url_fopen does not mean that people will only disable it in some cases, and if stream wrappers are allowed in include_path, this means that someday a worm will exist that takes advantage of another security hole to set include_path and then hi-jack the code. This is a security risk that I am pretty sure is unacceptable considering the marginal gains it would bring :). To eliminate this risk properly, we would need to disallow any remote streams (is_url = 1) from being used in include_path, regardless of the allow_url_fopen setting, which would slow down parsing of include_path and complicate things. The solution I've found for pharring things has involved clever use of str_replace(). I pharred up phpMyAdmin and it works just great using this technique. This removes the need for an include_path hack.
>>In terms of what choices you have with pecl/phar and libraries, the best way to >>handle this is to use relative includes (require_once phar://lib.phar/File.php") >>and have the user either include all the necessary phars in their php script or >>define an autoload that does it for them, just as they have to do with normal >>PHP files, or to bundle it all up in 1 phar. In other words, once you split a >>phar into multiple files, you've lost the primary benefit. Of course, we could >>move the discussion to pecl-dev if you'd like to talk about features in phar. >>I find it rather lame that your first response is "phar can't do this" when it >>is in alpha stage (meaning anything can be changed or added if needed) and has >>no releases... > > > OK. I apoligize if I looked rude to you. But I tried to understand what features > (current and future) PHAR provides. And I could not find any documentation. Almost > nothing in pecl, nothing returned by google. Nothing on features, not even a small > text to explain the pecl/phar's relation to the PEAR package. When I see in the > CVS that the project and most files are 4 to 7 months old, and that the only > available 'documentation' is the C source code, how can you blame me for not > knowing exactly what your software provides ? I never said that I wanted to create
There may be a language barrier issue here, as I see you are in France and I am not in France and we are speaking English. I don't blame you (or anyone) for not knowing things about phar that are not available (documentation, etc.), all I was saying is that the best way to find out this information is to ask. Good places to start are the links to the authors, available at http://pecl.php.net/phar, and conveniently clickable. Also good is pecl-dev@lists.php.net, where all things pecl are discussed.
> a new package system just because it is fun. If I had found the information I > needed about PHAR, I would have tried to enhance it. And it is still possible, of > course.
I created pecl/phar because it was fun as well as useful, that's not a bad reason :). Re-inventing the wheel, however, does not seem nearly as fun. Or as useful.
> Talking about documentation, in order to present more in-depth which features are > included in the tool I have written, I would be glad if you had a look at > http://www.tekwire.net/redir.php/phk_doc, chapter 1: Getting started. For the > rest, it is coming... Take it as a prototype or as feature ideas for PHAR...
I will take a look at your information. Initial glance says you've put lots of effort into this. As you may have guessed from the age of the files that you pointed out, neither Marcus nor I have prioritized pecl/phar over our other pressing needs in the past 4 to 7 months. If you wish to help with development, we welcome the input, especially if it ends up improving the code beyond where it is now. Greg

bertrand Gugger

20 years ago
Greg Beaver wrote:
> There may be a language barrier issue here, as I see you are in France > and I am not in France and we are speaking English.
Don't use that
> I don't blame you > (or anyone) for not knowing things about phar that are not available > (documentation, etc.), all I was saying is that the best way to find out > this information is to ask.
and to test it.
> Good places to start are the links to the > authors, available at http://pecl.php.net/phar, and conveniently > clickable. Also good is pecl-dev@lists.php.net, where all things pecl > are discussed.
I agree with .phar. à bientôt :)
-- toggg

LAUPRETRE François (P)

20 years ago
Greg Beaver wrote:
> The solution I've found for pharring things has involved clever use of > str_replace(). I pharred up phpMyAdmin and it works just great using this > technique. This removes the need for an include_path hack.
phpMyAdmin is an interesting test case because every file access are relative to the current directory, and some global variables are used both for relative file access, and as HTTP URLs base directories. In such a case, prefixing every path with a stream wrapper prefix cannot work because, as a side effect, it generates some '<img src=phar://....' strings in the HTML pages. I just had a look to the way you 'pharred' phpMyAdmin. I wanted to know how you solved the problem of the $pmaThemeImage global variable, which is used as a base for HTTP URLs and for file access. From what I understand, you replace it with an 'http:' string, which forces every file access through the http wrapper (which assumes at least that url_fopen is allowed). Here, it is not just a couple of str_replace(), it is a real change in the software structure. And it is not the only one. I understand that it is a prototype but I must say that I am skeptic about generalizing such an approach, even if, with the current PHP engine, there seems to be no btter one : - With changes like this in the source code, you cannot guarantee that the software works the same. And I don't see who will support it when somebody finds a difference. - If the original developers decide to support the phar packages and to distribute this format, they will want to distribute their software, at least for a given time, in both formats. Which means that their source code must be compatible. And filtering the files through str_replace() won't help. - If you have to make such changes in the software code, it implies that you know the software structure very well. And the work has to be repeated for every new version. Just try to build a phar package from the current phpMyAdmin version (2.8.2) and you will see that your changes cannot work exactly as in your previous build. For these reasons, I still consider that any PHP packaging system, PHAR or another one, will be greatly handicapped as long as we don't find a solution to the 'current directory' and 'include path' problems. Francois