Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » realpath() caching »

realpath() caching

php.internals

Andi Gutmans

21 years ago
Hi, Attached is the promised patch to do realpath() caching. You can set cache size and TTL via the php.ini (default is 2 minutes TTL and 16K memory). It doesn't use the SAPI time() but uses the syscall time() because it seemed to be slightly faster but we can adjust it to use the SAPI one if anyone measures a difference in performance (might depend on the OS). Feedback welcome. Also any feedback on the VM patch? I'd like to commit it soon but I haven't heard anything from this list (people were saying they wanted to review it). Thanks, Andi

Andi Gutmans

21 years ago
Thanks for pointing out that the attachment didn't make it through. You can find it at: http://snaps.php.net/~andi/realpath_cache2.diff Andi At 06:28 PM 9/20/2004 -0700, Andi Gutmans wrote:

Alan Knowles

21 years ago
Just a thought, should clearstatcache() force the reset of the cache? I cant think of many situations where you would re-build directory tree's on the fly, but you never know what to expect from PHP users :) Regards Alan Andi Gutmans wrote:

Sara Golemon

21 years ago
clearstatcache() is the wrong place to do this, but I certainly agree that the capability to flush the cache should exist. -Sara "Alan Knowles" <alan@akbkhome.com> wrote in message news:414F9CB7.8020904@akbkhome.com...
> Just a thought, should clearstatcache() force the reset of the cache? > I cant think of many situations where you would re-build directory > tree's on the fly, but you never know what to expect from PHP users :) > > Regards > Alan > > Andi Gutmans wrote: > > > Thanks for pointing out that the attachment didn't make it through. > > You can find it at: > > http://snaps.php.net/~andi/realpath_cache2.diff > > > > Andi > > > > At 06:28 PM 9/20/2004 -0700, Andi Gutmans wrote: > > > >> Hi, > >> > >> Attached is the promised patch to do realpath() caching. > >> You can set cache size and TTL via the php.ini (default is 2 minutes > >> TTL and 16K memory). > >> It doesn't use the SAPI time() but uses the syscall time() because it > >> seemed to be slightly faster but we can adjust it to use the SAPI one > >> if anyone measures a difference in performance (might depend on the
OS).

Andi Gutmans

21 years ago
It wouldn't be hard to add. I'm indifferent on how much it's really needed because it doesn't happen too often and the server could be restarted (then again some people might not like doing that). Andi At 11:15 AM 9/21/2004 +0800, Alan Knowles wrote:

Andi Gutmans

21 years ago
I was asked to provide some more info about this patch. In general, PHP does a realpath() on files (most importantly included files) so that when include_once() and other PHP functionality needs to know if two file paths are identical it will give a definite answer. realpath() itself is quite a slow function (especially on some specific OS) and therefore should be saved when possible. As it is rare that a realpath() for a certain file path + cwd combination ever changes, it's a perfect candidate for caching (actually a realpath() can only really change if you are adding/changing symlinks). Therefore, this patch implements caching of realpath() calls. There are two INI settings which can be changed: realpath_cache_size - Size of cache (default's to 16K; if set to 0 it turns of caching). realpath_cache_ttl - Time to live for cache entry (default 120 seconds; if set to 0 then doesn't refresh). On a loaded side even a small TTL should do the trick, although it might make sense to raise the TTL to something much higher than 120 seconds because as I said, it doesn't usually change. Hope that's enough info. Andi At 07:24 PM 9/20/2004 -0700, Andi Gutmans wrote:

Sara Golemon

21 years ago
> realpath_cache_ttl - Time to live for cache entry (default 120 seconds; if > set to 0 then doesn't refresh). >
Ah, didn't grok the TTL at first glance.... that fairly well negates the need for an explicit flush.... If the FS layout needs to change all one would have to do is turn down the TTL, make the change, and turn it back up after the cache is sure to be cleared... -Sara

Christian Schneider

21 years ago
Sara Golemon wrote:
>>realpath_cache_ttl - Time to live for cache entry (default 120 seconds; if >>set to 0 then doesn't refresh). >> > Ah, didn't grok the TTL at first glance.... that fairly well negates the > need for an explicit flush.... If the FS layout needs to change all one > would have to do is turn down the TTL, make the change, and turn it back up > after the cache is sure to be cleared...
Hmm.. if I understand correctly then one can only turn it down to 1 second as 0 means unlimited. So if an application changes the tree it would have to sleep(1) for it to happen. I'd strongly suggest that we have an explicit flush option. And no, I don't think restarting the server is an opton as this can't be done from inside an application. BTW: This needs to be well documented as running into caching problems can cause a very annoying debugging sessions due to the Heisenbug nature. Happened to more than one person I know with filesize() with missing clearstatcache() calls. One of PHP's main feature is programmer efficiency so we should be careful to add optimizations which make things harder for programmers. - Chris

Elfyn McBratney

21 years ago
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, Btw, cool patch. :-) I'll see if we can start testing this in an experimental ebuild in Gentoo land. On Wednesday 22 Sep 2004 10:52, Christian Schneider wrote:
> Sara Golemon wrote: > >>realpath_cache_ttl - Time to live for cache entry (default 120 seconds; > >> if set to 0 then doesn't refresh). > > > > Ah, didn't grok the TTL at first glance.... that fairly well negates the > > need for an explicit flush.... If the FS layout needs to change all one > > would have to do is turn down the TTL, make the change, and turn it back > > up after the cache is sure to be cleared... > > Hmm.. if I understand correctly then one can only turn it down to 1 > second as 0 means unlimited. So if an application changes the tree it > would have to sleep(1) for it to happen.
In real life, that would only mean a 'broken' path for 1 second. And changing path's on a production system doesn't happen all that regularly, in my experience. On busy sites, a lot of users could get that broken file (not exist, whatever), but if you don't code realpath() file i/o related correctly i.e., checking the file open'ed ok at least, (and in the stat()/is_*() case) checking the file size, you're toast anyway. ;-)
> I'd strongly suggest that we have an explicit flush option. And no, I > don't think restarting the server is an opton as this can't be done from > inside an application.
Same here, if it counts. ;-) For those short-by-1-second cases, and during development, t'would be handy. Elfyn - -- Elfyn McBratney beu on irc.freenode.net/savannah.[non]gnu.org PGP Key ID: 0x456548B4 PGP Key Fingerprint: 29D5 91BB 8748 7CC9 650F 31FE 6888 0C2A 4565 48B4 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFBUVuDaIgMKkVlSLQRAvhiAJ9IGQtNBi1T+XNIddkEbxWblgrfAACfXeGr sbhCUYI5tascMi95g9Is+ug= =z+yd -----END PGP SIGNATURE-----

Andi Gutmans

21 years ago
I don't know if I'd like to make a PHP user-land function for this, especially as it'll get slightly sticky due to this being a TSRM mechanism. Also, as mentioned by Elfyn, it's very unlikely to affect anyone. How about allowing ini_set("realpath_catch_size", ?); during execution. It would change the cache size and while doing that would reset the cache? I think that might be a good solution. Andi At 11:52 AM 9/22/2004 +0200, Christian Schneider wrote:

Vlad Bosinceanu

21 years ago
Sounds a bit hackish - as opposed to most things in php that have a special function. V Andi Gutmans wrote:

Marcus Börger

21 years ago
Hello Andi, if we have ini_set("realpath_catch_size") then obviously ini_set("realpath_catch_size", 0) === clearstatcache() without any additional line of code. regards marcus Friday, September 24, 2004, 1:08:14 AM, you wrote:
> I don't know if I'd like to make a PHP user-land function for this, > especially as it'll get slightly sticky due to this being a TSRM mechanism. > Also, as mentioned by Elfyn, it's very unlikely to affect anyone. > How about allowing ini_set("realpath_catch_size", ?); during execution. It > would change the cache size and while doing that would reset the cache? > I think that might be a good solution.
> Andi
> At 11:52 AM 9/22/2004 +0200, Christian Schneider wrote: >>Sara Golemon wrote: >>>>realpath_cache_ttl - Time to live for cache entry (default 120 seconds; if >>>>set to 0 then doesn't refresh). >>>Ah, didn't grok the TTL at first glance.... that fairly well negates the >>>need for an explicit flush.... If the FS layout needs to change all one >>>would have to do is turn down the TTL, make the change, and turn it back up >>>after the cache is sure to be cleared... >> >>Hmm.. if I understand correctly then one can only turn it down to 1 second >>as 0 means unlimited. So if an application changes the tree it would have >>to sleep(1) for it to happen. >> >>I'd strongly suggest that we have an explicit flush option. And no, I >>don't think restarting the server is an opton as this can't be done from >>inside an application. >> >>BTW: This needs to be well documented as running into caching problems can >>cause a very annoying debugging sessions due to the Heisenbug nature. >>Happened to more than one person I know with filesize() with missing >>clearstatcache() calls. One of PHP's main feature is programmer efficiency >>so we should be careful to add optimizations which make things harder for >>programmers. >> >>- Chris >> >>-- >>PHP Internals - PHP Runtime Development Mailing List >>To unsubscribe, visit: http://www.php.net/unsub.php
-- Best regards, Marcus mailto:helly@php.net

Andi Gutmans

21 years ago
Yep, that was my thinking. Andi At 08:29 PM 9/24/2004 +0200, Marcus Boerger wrote:

Christian Schneider

21 years ago
Andi Gutmans wrote:
> Yep, that was my thinking. > > At 08:29 PM 9/24/2004 +0200, Marcus Boerger wrote: > >> if we have ini_set("realpath_catch_size") then obviously >> ini_set("realpath_catch_size", 0) === clearstatcache() without any >> additional line of code.
I wouldn't introduce a new function but just have clearstatcache() do this. And I don't think we should use ini_set("realpath_cache_size", 0) for this as one then has to do ini_set() twice just to flush but still use the cache. The caches (stat and realpath) introduce enough magic to PHP so I'd at least keep the solution (clearstatcache) as simple as possible. Just my $.02, - Chris

Andi Gutmans

21 years ago
At 02:54 PM 9/27/2004 +0200, Christian Schneider wrote:
>Andi Gutmans wrote: >>Yep, that was my thinking. >>At 08:29 PM 9/24/2004 +0200, Marcus Boerger wrote: >> >>> if we have ini_set("realpath_catch_size") then obviously >>>ini_set("realpath_catch_size", 0) === clearstatcache() without any >>>additional line of code. > >I wouldn't introduce a new function but just have clearstatcache() do >this. And I don't think we should use ini_set("realpath_cache_size", >0) for this as one then has to do ini_set() twice just to flush but still >use the cache. > >The caches (stat and realpath) introduce enough magic to PHP so I'd at >least keep the solution (clearstatcache) as simple as possible.
What I meant was that any re-setting of this INI parameter (even if the cache size is the same) would flush the cache. I think this is fine because in real life, I don't think anyone will need it anyway. It's VERY rare to change symlinks in a way which will affect this. Andi

Sara Golemon

21 years ago
> >I wouldn't introduce a new function but just have clearstatcache() do > >this. And I don't think we should use ini_set("realpath_cache_size", > >0) for this as one then has to do ini_set() twice just to flush but
still
> >use the cache. > > > >The caches (stat and realpath) introduce enough magic to PHP so I'd at > >least keep the solution (clearstatcache) as simple as possible. > > What I meant was that any re-setting of this INI parameter (even if the > cache size is the same) would flush the cache. I think this is fine
because
> in real life, I don't think anyone will need it anyway. It's VERY rare to > change symlinks in a way which will affect this. >
Right, and people *do* make changes to files which necessitate the need to invalidate the stat cache. Given that difference and the fact that they're really just plain two different things: I don't want to see clearstatcache() tied to the realpath() caching in any way. On a side topic, I've been meaning to expand the statecache beyond a single item, something along the lines of what Andi's done with realpath() caching. I'd also like to introduce an additional configuration option (please don't cringe too much) to reflect the fact that stat() family calls now support protocol wrappers (many of which are network based). I think it'd make sense to allow clearstatcache() to selectively clear all cache entries, or only entries for wrappers which do not have the is_url bit set. I briefly considered making the stat cache global (to share the speed up with all), but that has the potential to introduce a BC break (a script which relies on a stale cache?), and opens the door to some possible safe_mode circumventions (User A stats a file he's allowed to, then User B hits the cache). We could throw in some additional safe mode checks at that point, but then we're just adding complexity in to cover up the stat calls taken out. Kind of a zero-sum (or even negative sum) balance there.... Thoughts? -Sara

Christian Schneider

21 years ago
Sara Golemon wrote:
> protocol wrappers (many of which are network based). I think it'd make sense > to allow clearstatcache() to selectively clear all cache entries, or only > entries for wrappers which do not have the is_url bit set.
Maybe this should be generalized to a cache_flush() function with e.g. two optional parameters: Cache type to flush and object to flush from cache. Something like cache_flush(); # Same as flushcache(CACHE_ALL) cache_flush(CACHE_STAT) # Same as clearstatcache() cache_flush(CACHE_REALPATH); cache_flush(CACHE_REALPATH, $path); Caches can help, they can also confuse you and be in the way and you need control over them. With a reasonable interface. I'd also propose to disable caching in the default ini file as according to my observations most users are not pushing their servers to the limit but they do get nailed by strange bugs due to caching artefacts. - Chris

Andi Gutmans

21 years ago
I don't think STAT cache and realpath have to necessarily be merged because there's a big difference. stat()'s do change periodically (i.e. it's very common) as opposed to realpath() which don't. That said, I think if we add cache_flush() for the stat() cache, I see no problem with it also flushing the realpath() cache so your API suggestion looks fine to me. Andi At 12:39 AM 9/28/2004 +0200, Christian Schneider wrote: