Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » Ability to lower PHP memory usage »

Ability to lower PHP memory usage

php.internals

Andi Gutmans

22 years ago
Hey, I just compiled PHP using --disable-all and realized that there's a shit load of relatively useless stuff still being compiled into it. The first thing I realized is that regex is still being compiled in by default. Are any parts of PHP's core relying on regex, thus, requiring it to be in the minimal build? I think most people today are using pcre and it would be great if we could support a version without regex. The second problem is in ext/standard with a zillion of functions. A lot of them are in my opinion not used by the majority of users and it would be great to be able not to compile them into PHP. Things that come to my mind: a) crc32 b) cyr_convert c) image d) lcg e) metaphone f) soundex g) levenshtein h) sha1 i) uuencode and probably some other stuff, In order for PHP users to get the most out of their servers, being able to reduce memory footprint as much as possible is quite important so that they can increase MaxClients to as much as possible. As we're in a code freeze for RC1 it's a bit late to do something about it, but I think it'd be nice to brainstorm about this to see how we can solve this, maybe for 5.1? If regex isn't required by the core that's one of the first things I'd like to see disabled when building with --disable-all. I think ext/standard should maybe be split up into two, the first ext/core and the second ext/standard where ext/core is the stuff which *really* needs to be part of PHP such as strlen(), file functions and pretty much stuff I didn't list in my list :) Any thoughts? Ideas? Flames? :) Andi

Stanislav Malyshev

22 years ago
AG>> but I think it'd be nice to brainstorm about this to see how we can AG>> solve this, maybe for 5.1? If regex isn't required by the core that's AG>> one of the first things I'd like to see disabled when building with AG>> --disable-all. I think ext/standard should maybe be split up into AG>> two, the first ext/core and the second ext/standard where ext/core is AG>> the stuff which *really* needs to be part of PHP such as strlen(), AG>> file functions and pretty much stuff I didn't list in my list :) OK, the problem I see here is that such thing would increase PHP platform diversity. And what I mean by that is: Suppose I am writing some PHP application for distribution. And I use some 'weird' functions - meaning, something beyond strlen() and other obviously 'standard' ones. As for now, it is enough for me to say to the user "this application uses MySQL extension" or "this application uses BlaFooBar extension" or "this application uses only standard functions" and it would be completely sufficient for the user to create an environment where the application would run. If we enable the user to individually disable the functions by standard means, the PHP developer would have now to check if the application uses image() and alert the user in the documentation about it - since the user - or his provider or IT stuff - could decide it is "unneeded" and drop it from compiled version. This means more or less everybody needs now to remember which functions are "core" and which are "non-core". Now we have more or less module-level granularity with clear naming so that there's no doubt which module pgsql_connect belongs to. But enabling the user to drop standard functions would create function-level granularity without a good mean for the developer to know even if given function is "removable" or not (except for memorizing the whole list, of course). Certainly, it is possible even now to break the platform by disabling some functions manually, but then it is messing with the code - which is much higher barrier than altering one config option. Most users have no problem writing some configure line but would never touch the code. I'm concerned that this problem of breaking common platform might be more dangerous than the performance benefit. Which, BTW, I estmate as pretty minimal - code space is shared on all modern OSes anyway, so a little extra code doesn't significantly impact scalability, and unused functions usually don't claim data memory - except for, probably, symbol table entries, but this is a small change. With all this said, if some functions like levenstein(), soundex() etc. might be moved to separate extension and this move would be announced, clearly marked, etc. - it might be feasible.
-- Stanislav Malyshev, Zend Products Engineer stas@zend.com http://www.zend.com/ +972-3-6139665 ext.109

Christian Schneider

22 years ago
Stanislav Malyshev wrote:
> I'm concerned that this problem of breaking common platform might be more > dangerous than the performance benefit. Which, BTW, I estmate as pretty > minimal - code space is shared on all modern OSes anyway, so a little
I think that's a good point for leaving it the way it is: Minimal benefit while opening a can of worms of possible problems. Another reason not to do it is the amount of work to decide which function should go where. Let's keep it simple and focus on IMHO more pressing problems. - Chris

Marcus Börger

22 years ago
Hello Christian, Thursday, January 8, 2004, 2:24:33 PM, you wrote:
> Stanislav Malyshev wrote: >> I'm concerned that this problem of breaking common platform might be more >> dangerous than the performance benefit. Which, BTW, I estmate as pretty >> minimal - code space is shared on all modern OSes anyway, so a little
> I think that's a good point for leaving it the way it is: Minimal > benefit while opening a can of worms of possible problems.
> Another reason not to do it is the amount of work to decide which > function should go where. Let's keep it simple and focus on IMHO more > pressing problems.
Same thoughts here. The benefit is far to low. There are a lot of places we could put our efforts into better: bug-fixing instead of creating new bug fixes and increasing the WFT factor. The only real reason is it php would have been meant to serve cellphones or other small devices. But hey those run java. So we shouldn't care about a few kilobytes (or hundreds of them). Best regards, Marcus mailto:helly@php.net

George Schlossnagle

22 years ago
On Jan 8, 2004, at 12:22 PM, Marcus Boerger wrote:
> Hello Christian, > > Thursday, January 8, 2004, 2:24:33 PM, you wrote: > >> Stanislav Malyshev wrote: >>> I'm concerned that this problem of breaking common platform might be >>> more >>> dangerous than the performance benefit. Which, BTW, I estmate as >>> pretty >>> minimal - code space is shared on all modern OSes anyway, so a little > >> I think that's a good point for leaving it the way it is: Minimal >> benefit while opening a can of worms of possible problems. > >> Another reason not to do it is the amount of work to decide which >> function should go where. Let's keep it simple and focus on IMHO more >> pressing problems. > > Same thoughts here. The benefit is far to low. There are a lot of > places > we could put our efforts into better: bug-fixing instead of creating > new > bug fixes and increasing the WFT factor. > > The only real reason is it php would have been meant to serve > cellphones or > other small devices. But hey those run java. So we shouldn't care > about a > few kilobytes (or hundreds of them).
Just to chip in my feeling that this is a low-gain proposal. George

Andi Gutmans

22 years ago
At 02:24 PM 1/8/2004 +0100, Christian Schneider wrote:
>Stanislav Malyshev wrote: >>I'm concerned that this problem of breaking common platform might be more >>dangerous than the performance benefit. Which, BTW, I estmate as pretty >>minimal - code space is shared on all modern OSes anyway, so a little > >I think that's a good point for leaving it the way it is: Minimal benefit >while opening a can of worms of possible problems. > >Another reason not to do it is the amount of work to decide which function >should go where. Let's keep it simple and focus on IMHO more pressing problems.
Guys, You are all missing the point. All the ext/standard would still be enabled by default, but it would allow people with very high traffic sites who need to save every bit of memory they can to build a lean-and-mean version of PHP. These kinds of users are looking for optimizing PHP for their application and do it with a sane mind. I don't even mind if --disable-all doesn't disable ext/standard but it'd be nice that if we do a split to core/ and standard/ (I wouldn't go into more granularity than that) that we could have a --disable-standard. It wouldn't hurt anyone and it *is* a pressing problem from talks I've had with all sorts of ppl that have high traffic machines. No one answered me about regex btw. Is it only being compiled because of the regex functions in ext/standard or does anyone know if other places in PHP's source tree use it. Andi

Ilia A.

22 years ago
On January 08, 2004 12:02 pm, Andi Gutmans wrote:
> You are all missing the point. All the ext/standard would still be enabled > by default, but it would allow people with very high traffic sites who need > to save every bit of memory they can to build a lean-and-mean version of > PHP. These kinds of users are looking for optimizing PHP for their > application and do it with a sane mind.
There is a cost vs benefit here how much do you expect save? Suppose we save 100k by moving some code from ext/standard/ with even as many as 200 processes running you'll still only save about 19 megs. Even that may be hard to do, since A LOT of code will need to be 'moved off' from ext/standard/ to save those 100k (after stripping). Given current memory prices 19 megs is about ($2-$3 USD). On a related note, how do we determine which functions are 'safe' to move and which ones are not. I may not use 1/2 of the array_* functions we have, but someone else may use them and not use metaphone() which I use. What is the criteria by which we determine what goes out and what stays in?
> I don't even mind if --disable-all doesn't disable ext/standard but it'd be > nice that if we do a split to core/ and standard/ (I wouldn't go into more > granularity than that) that we could have a --disable-standard. > It wouldn't hurt anyone and it *is* a pressing problem from talks I've had > with all sorts of ppl that have high traffic machines.
I suspect the memory usage is due to other things like forgetting to strip binaries & libraries, inefficient scripts and PHP enabled web servers being used to serve static requests.
> No one answered me about regex btw. Is it only being compiled because of > the regex functions in ext/standard or does anyone know if other places in > PHP's source tree use it.
browscap code uses regular regex. mbstring does not use regex, but it's regex syntax is compatible to that of ereg_* and not pcre. Ilia

Christian Schneider

22 years ago
Andi Gutmans wrote:
> I don't even mind if --disable-all doesn't disable ext/standard but it'd > be nice that if we do a split to core/ and standard/ (I wouldn't go into > more granularity than that) that we could have a --disable-standard.
I see no problem if it's not included in --disable-all. It is then left to the system administrator to shoot himself in the foot if he decides to --disable-standard. On the other hand I don't think any system administrator running someone elses scripts uses disable-all anyway. So I guess either --disable-all or --disable-all-and-more are ok :-)
> It wouldn't hurt anyone and it *is* a pressing problem from talks I've > had with all sorts of ppl that have high traffic machines.
Seems weird to me but I have to take your word for it. We have a caching loadbalancer which frees up apache processes more quickly. And we disabled keepalive at some points. But providing a low-footprint PHP for people who can't use these methods could be a good thing.
> No one answered me about regex btw. Is it only being compiled because of > the regex functions in ext/standard or does anyone know if other places
I had a quick look and browscap seems to use it. And some internal split functions are in ext/standard/reg.c, not sure where they are used. - Chris

Marcus Börger

22 years ago
Hello Andi, Thursday, January 8, 2004, 6:02:42 PM, you wrote:
> At 02:24 PM 1/8/2004 +0100, Christian Schneider wrote: >>Stanislav Malyshev wrote: >>>I'm concerned that this problem of breaking common platform might be more >>>dangerous than the performance benefit. Which, BTW, I estmate as pretty >>>minimal - code space is shared on all modern OSes anyway, so a little >> >>I think that's a good point for leaving it the way it is: Minimal benefit >>while opening a can of worms of possible problems. >> >>Another reason not to do it is the amount of work to decide which function >>should go where. Let's keep it simple and focus on IMHO more pressing problems.
> Guys,
> You are all missing the point. All the ext/standard would still be enabled > by default,
Seems like everyone sees builtin/enabling by default as a minimum requirement for such an approach.
> but it would allow people with very high traffic sites who need > to save every bit of memory they can to build a lean-and-mean version of PHP. > These kinds of users are looking for optimizing PHP for their application > and do it with a sane mind.
In highspeed scenarios you wouldn't use CGI but a module. There the memory consumption by php core doesn't change after the worker threads/processes are started. The benefit here would be reduced function tables and hence faster function lookup. But if i would deal with such a site i would first disable some function calling overhead in the engine. That should bring much more. Maybe we should add some #ifdef's there combined with a (--very-fast) option that on the other hand would disable features like the tick operator.
-- Best regards, Marcus mailto:helly@php.net

Andrei Zmievski

22 years ago
On Thu, 08 Jan 2004, Andi Gutmans wrote:
> You are all missing the point. All the ext/standard would still be enabled > by default, but it would allow people with very high traffic sites who need > to save every bit of memory they can to build a lean-and-mean version of > PHP.
If they want lean-and-mean version, why can't they compile an .so one and have that loaded up by Apache so that it's shared by the processes?
> No one answered me about regex btw. Is it only being compiled because of > the regex functions in ext/standard or does anyone know if other places in > PHP's source tree use it.
Simple recipe: 1. Install cscope (RPM, apt-get, whatever) 2. cd into php-src 3. cscope -R 4. Type in 'regcomp' and see for yourself. - Andrei

Rasmus Lerdorf

22 years ago
On Thu, 8 Jan 2004, Andi Gutmans wrote:
> It wouldn't hurt anyone and it *is* a pressing problem from talks I've had > with all sorts of ppl that have high traffic machines.
You must be talking to different people than I am. 100 or so functions all sitting in shared pages and thus shared across each httpd is a trivial amount of memory. I suspect these people who are having problems don't know how to optimize their httpd very well. You get a lot more bang for the buck from making each request process faster and lowering the number of httpd children you need to run. System-call optimization and overall optimization of the various critical sections would give you a lot more return on your effort than disabling a few functions. I would be very surprised if you were to comment out 100 functions somewhere, that you could even measure a performance difference on a high-traffic site. -Rasmus

Zeev Suraski

22 years ago
At 19:53 08/01/2004, Rasmus Lerdorf wrote:
>On Thu, 8 Jan 2004, Andi Gutmans wrote: > > It wouldn't hurt anyone and it *is* a pressing problem from talks I've had > > with all sorts of ppl that have high traffic machines. > >You must be talking to different people than I am.
We are :)
> 100 or so functions >all sitting in shared pages and thus shared across each httpd is a trivial >amount of memory. I suspect these people who are having problems don't >know how to optimize their httpd very well. You get a lot more bang for >the buck from making each request process faster and lowering the number >of httpd children you need to run. System-call optimization and overall >optimization of the various critical sections would give you a lot more >return on your effort than disabling a few functions. I would be very >surprised if you were to comment out 100 functions somewhere, that you >could even measure a performance difference on a high-traffic site.
I think it very much depends on how many such entries we find that make sense to disable. From Andi's list, I'd say that only the image extension sounds as if it may give us some noticeable gain. I don't think this has anything to do with overall optimization. That particular thing is trivial to perform, optimization isn't, so in terms of return-on-investment, it's likely to be better. But overall, again, it very much depends if we there's a significant chunk in PHP that can be disabled without breaking the average application. Personally, I'm not convinced this is that case, even if the people we're dealing with run thousands of Apache processes per server (which they do). Zeev

George Schlossnagle

22 years ago
On Jan 8, 2004, at 1:39 PM, Zeev Suraski wrote:
> > Personally, I'm not convinced this is that case, even if the people > we're dealing with run thousands of Apache processes per server (which > they do).
Unless they're running thousands of apache server instances (not just children), shouldn't the memory sit in shared pages and thus be completely insignificant? And if they are running 1000s of instances per server, perhaps they have an architectural problem that this band-aid won't fix? :) George

Rasmus Lerdorf

22 years ago
On Thu, 8 Jan 2004, Zeev Suraski wrote:
> Personally, I'm not convinced this is that case, even if the people we're > dealing with run thousands of Apache processes per server (which they do).
But could you explain where you see the significant incremental memory usage going from 20 to 2000 httpd processes asscociated with a couple of functions? I assume we are not talking about Windows or CGI here. -Rasmus

Zeev Suraski

22 years ago
At 20:57 08/01/2004, George Schlossnagle wrote:
>On Jan 8, 2004, at 1:39 PM, Zeev Suraski wrote: >> >>Personally, I'm not convinced this is that case, even if the people we're >>dealing with run thousands of Apache processes per server (which they do). > >Unless they're running thousands of apache server instances (not just >children), shouldn't the memory sit in shared pages and thus be completely >insignificant? > >And if they are running 1000s of instances per server, perhaps they have >an architectural problem that this band-aid won't fix? :)
Obviously we're talking about httpd children and not 1,000 roots... Anyway, depending on the module, if there's any sort of RINIT initialization, the CoW trick doesn't work very well and it consumes some per-process memory. There's also the function table which is not shared and will grow by a few dozen kilobytes thanks to the extra functions, which is not shared across children. Note, my position at this point is that there's probably very little gain in this as well, unless we can find a large number of modules with a significant amount of functions, some of them maybe doing their own per-request initialization, thus conserving a significant amount of per-process memory. The list that Andi posted does not convince me that this is the case. Zeev

Rasmus Lerdorf

22 years ago
On Thu, 8 Jan 2004, Zeev Suraski wrote:
> Obviously we're talking about httpd children and not 1,000 > roots... Anyway, depending on the module, if there's any sort of RINIT > initialization, the CoW trick doesn't work very well and it consumes some > per-process memory. There's also the function table which is not shared > and will grow by a few dozen kilobytes thanks to the extra functions, which > is not shared across children. > > Note, my position at this point is that there's probably very little gain > in this as well, unless we can find a large number of modules with a > significant amount of functions, some of them maybe doing their own > per-request initialization, thus conserving a significant amount of > per-process memory. The list that Andi posted does not convince me that > this is the case.
Yeah, I am all for getting rid of or at least seriously reducing what is done in any RINITs. That is a clear win. I would even go as far as to say that anything that has an RINIT should not be compiled in by default. -Rasmus

George Schlossnagle

22 years ago
On Jan 8, 2004, at 2:12 PM, Zeev Suraski wrote:
> At 20:57 08/01/2004, George Schlossnagle wrote: > >> On Jan 8, 2004, at 1:39 PM, Zeev Suraski wrote: >>> >>> Personally, I'm not convinced this is that case, even if the people >>> we're dealing with run thousands of Apache processes per server >>> (which they do). >> >> Unless they're running thousands of apache server instances (not just >> children), shouldn't the memory sit in shared pages and thus be >> completely insignificant? >> >> And if they are running 1000s of instances per server, perhaps they >> have an architectural problem that this band-aid won't fix? :) > > Obviously we're talking about httpd children and not 1,000 roots... > Anyway, depending on the module, if there's any sort of RINIT > initialization, the CoW trick doesn't work very well and it consumes > some per-process memory. There's also the function table which is not > shared and will grow by a few dozen kilobytes thanks to the extra > functions, which is not shared across children.
The RINIT issue makes good sense. In regards to the couple of kilobytes for the function table - that seems really inconsequential to me. couple k * couple thousand children is a couple megs. Hardly much to fret about in an installation as large as you are saying. george p.s. Is there a technical reason the function table could be shareable across children? I can't think of one of the top of my head.

Zeev Suraski

22 years ago
At 21:59 08/01/2004, George Schlossnagle wrote:
>On Jan 8, 2004, at 2:12 PM, Zeev Suraski wrote: > >>At 20:57 08/01/2004, George Schlossnagle wrote: >> >>>On Jan 8, 2004, at 1:39 PM, Zeev Suraski wrote: >>>> >>>>Personally, I'm not convinced this is that case, even if the people >>>>we're dealing with run thousands of Apache processes per server (which >>>>they do). >>> >>>Unless they're running thousands of apache server instances (not just >>>children), shouldn't the memory sit in shared pages and thus be >>>completely insignificant? >>> >>>And if they are running 1000s of instances per server, perhaps they have >>>an architectural problem that this band-aid won't fix? :) >> >>Obviously we're talking about httpd children and not 1,000 roots... >>Anyway, depending on the module, if there's any sort of RINIT >>initialization, the CoW trick doesn't work very well and it consumes some >>per-process memory. There's also the function table which is not shared >>and will grow by a few dozen kilobytes thanks to the extra functions, >>which is not shared across children. > >The RINIT issue makes good sense. In regards to the couple of kilobytes >for the function table - that seems really inconsequential to me. couple >k * couple thousand children is a couple megs. Hardly much to fret about >in an installation as large as you are saying.
Well, if we really get to save 100K as Ilia imagined, for a thousand children, it's 100MB, and there are those with more. I doubt we can easily get 100K though.
>george > >p.s. Is there a technical reason the function table could be shareable >across children? I can't think of one of the top of my head.
I guess you're missing a 'not' in this question? :) Anyway, the reason it cannot be shared is that it also contains user-defined functions. It starts up as shared (the builtin functions are loaded in MINIT, before the fork), but very quickly this hash table becomes local to the process because it's updated with user defined functions. Zeev

Ilia A.

22 years ago
On January 08, 2004 04:11 pm, Zeev Suraski wrote:
> Well, if we really get to save 100K as Ilia imagined, for a thousand > children, it's 100MB, and there are those with more. I doubt we can easily > get 100K though.
'Freeing' 100k is not as difficult as it sounds in PHP 5 when you consider that 4 object files (filters.o 14460, ftp_fopen_wrapper.o 13616, info.o 13868, streamsfuncs.o 13408) make up for more then 1/2. Add (url_scanner_ex.o 12304, var.o 14616 image.o 11432,var_unserializer.o 5984) and you've reduced your php library/binary by 100k. Even so you gain only a 100mb saving (1000 simultaneous apache processes, WOW) per server. Given that 100mb of ram is only about $10 (or less) it seems to me this would be a cheaper solution then having to pay a programmer ($50/hour+) to recompile PHP with obscure options (to reduce memory footprint) on a variety of servers. I won't even go into the loss of functionality that maybe needed later. Ilia

Sascha Schumann

22 years ago
On Thu, 8 Jan 2004, Ilia Alshanetsky wrote:
> On January 08, 2004 04:11 pm, Zeev Suraski wrote: > > Well, if we really get to save 100K as Ilia imagined, for a thousand > > children, it's 100MB, and there are those with more. I doubt we can easily > > get 100K though. > > 'Freeing' 100k is not as difficult as it sounds in PHP 5 when you consider > that 4 object files (filters.o 14460, ftp_fopen_wrapper.o 13616, info.o > 13868, streamsfuncs.o 13408) make up for more then 1/2. Add (url_scanner_ex.o > 12304, var.o 14616 image.o 11432,var_unserializer.o 5984) and you've reduced > your php library/binary by 100k.
Great, so you have freed 100K in *total*, not per process. Remember that object code resides in read-only/shared pages. The code is loaded to memory only once and shared among all running processes. Thus, removing object code cannot possibly reduce the memory footprint of a large installation in any significant way.
> Even so you gain only a 100mb saving (1000 simultaneous apache processes, WOW)
According to a poster on new-httpd a few days ago, some people even use 20K Apache children. Although they are unlikely using Apache for serving only dynamic requests. So, if you want to optimize the memory footprint, reduce the size of run-time structures. - Sascha

Sterling Hughes

22 years ago
> >p.s. Is there a technical reason the function table could be shareable > >across children? I can't think of one of the top of my head. > > I guess you're missing a 'not' in this question? :) Anyway, the reason it > cannot be shared is that it also contains user-defined functions. It > starts up as shared (the builtin functions are loaded in MINIT, before the > fork), but very quickly this hash table becomes local to the process > because it's updated with user defined functions. >
Right - but then only the hash isn't shared, the allocated functions still are. I can't believe that the hash array is 100k of memory. It brings up an interesting point though. If these tables could be separated, having a separate user and builtin function table, then you could save memory on builtin functions, and more importantly, you could further make the symbol table implementation quite a bit lighter. As well as eliminating the hash lookup for builtin functions. During compile time, builtin functions could be resolved to pointers (avoiding a lookup), whereas user functions would still incur the lookup. This should work fine with compiler cache's as well, so long as they have a mechanism for allocating the builtins into shm. This would decrease performance of dynamic function calls (incuring a two phase lookup), but if you use those functions, you've already lost. Just a thought. :) -Sterling

George Schlossnagle

22 years ago
On Jan 8, 2004, at 4:24 PM, Sterling Hughes wrote:
>>> p.s. Is there a technical reason the function table could be >>> shareable >>> across children? I can't think of one of the top of my head. >> >> I guess you're missing a 'not' in this question? :) Anyway, the >> reason it >> cannot be shared is that it also contains user-defined functions. It >> starts up as shared (the builtin functions are loaded in MINIT, >> before the >> fork), but very quickly this hash table becomes local to the process >> because it's updated with user defined functions. >> > > Right - but then only the hash isn't shared, the allocated functions > still > are. I can't believe that the hash array is 100k of memory. > > It brings up an interesting point though. If these tables could be > separated, having a separate user and builtin function table, then you > could save memory on builtin functions, and more importantly, you could > further make the symbol table implementation quite a bit lighter. As > well as eliminating the hash lookup for builtin functions.
I think this idea (using two hash tables) was rejected during the namespace discussions around here: http://www.zend.com/lists/engine2/200304/msg00190.html The purpose is different, but the idea very similar, and the double-lookup problem the same. I don't see the real problem though - the function table is what 32 bytes * numBuckets + some other stuff? numBuckets needs to be pretty big for that to amount to anything. The compile-time pointer resolution is a nice optimization of course, without without double tables. George

Zeev Suraski

22 years ago
At 23:24 08/01/2004, Sterling Hughes wrote:
> > >p.s. Is there a technical reason the function table could be shareable > > >across children? I can't think of one of the top of my head. > > > > I guess you're missing a 'not' in this question? :) Anyway, the reason it > > cannot be shared is that it also contains user-defined functions. It > > starts up as shared (the builtin functions are loaded in MINIT, before the > > fork), but very quickly this hash table becomes local to the process > > because it's updated with user defined functions. > > > >Right - but then only the hash isn't shared, the allocated functions still >are. I can't believe that the hash array is 100k of memory.
They probably aren't, because of the way the hash is and the allocation pattern. First, the bucket and the function container itself probably sit in the same page because they're allocated next to each other. Secondly, the buckets contain list pointers that are updated as the hash is updated. Hence, the functions are also not shared (in other words, the hash buckets actually take many more pages than just their accumulated size). People, READ, I'm saying that I *DON'T* think that we'd be saving 100K from reducing the size of this hash. Read a couple of more seconds before you say I do :)
>It brings up an interesting point though. If these tables could be >separated, having a separate user and builtin function table, then you >could save memory on builtin functions, and more importantly, you could >further make the symbol table implementation quite a bit lighter. As >well as eliminating the hash lookup for builtin functions.
I don't see how it will make the symbol table implementation any lighter...
>During compile time, builtin functions could be resolved to pointers >(avoiding a lookup), whereas user functions would still incur the >lookup. This should work fine with compiler cache's as well, so long as >they have a mechanism for allocating the builtins into shm.
I don't think they have to be allocated into shm at all in any way. There's really no need for that. You could simply save their pointers as they never change.
>This would decrease performance of dynamic function calls (incuring a >two phase lookup), but if you use those functions, you've already lost. > >Just a thought. :)
Interesting thought. Something for 5.1 :) Zeev

Sterling Hughes

22 years ago
> At 23:24 08/01/2004, Sterling Hughes wrote: > >> >p.s. Is there a technical reason the function table could be shareable > >> >across children? I can't think of one of the top of my head. > >> > >> I guess you're missing a 'not' in this question? :) Anyway, the reason > >it > >> cannot be shared is that it also contains user-defined functions. It > >> starts up as shared (the builtin functions are loaded in MINIT, before > >the > >> fork), but very quickly this hash table becomes local to the process > >> because it's updated with user defined functions. > >> > > > >Right - but then only the hash isn't shared, the allocated functions still > >are. I can't believe that the hash array is 100k of memory. > > They probably aren't, because of the way the hash is and the allocation > pattern. First, the bucket and the function container itself probably sit > in the same page because they're allocated next to each other. Secondly, > the buckets contain list pointers that are updated as the hash is > updated. Hence, the functions are also not shared (in other words, the > hash buckets actually take many more pages than just their accumulated > size). >
yep.. forgot about that.
> People, READ, I'm saying that I *DON'T* think that we'd be saving 100K from > reducing the size of this hash. Read a couple of more seconds before you > say I do :)
I was agreeing with you. :)
> > >It brings up an interesting point though. If these tables could be > >separated, having a separate user and builtin function table, then you > >could save memory on builtin functions, and more importantly, you could > >further make the symbol table implementation quite a bit lighter. As > >well as eliminating the hash lookup for builtin functions. > > I don't see how it will make the symbol table implementation any lighter... >
the reason that a symbol table needs to be a hasharray as opposed to a simple hashtable is the distinction between builtin functions and user functions as a reverse_apply() is required at request shutdown. if you separate the tables, you no longer need to reverse apply over the symbol table.
> >During compile time, builtin functions could be resolved to pointers > >(avoiding a lookup), whereas user functions would still incur the > >lookup. This should work fine with compiler cache's as well, so long as > >they have a mechanism for allocating the builtins into shm. > > I don't think they have to be allocated into shm at all in any > way. There's really no need for that. You could simply save their > pointers as they never change. >
that's what i thought too, but i remember that thies and i had a problem with apc and builtins + vtable caching (storing it in the ops). Would be even better if I was proved wrong, of course.
> >This would decrease performance of dynamic function calls (incuring a > >two phase lookup), but if you use those functions, you've already lost. > > > >Just a thought. :) > > Interesting thought. Something for 5.1 :)
yeah. i wasn't saying tommorow, but i am saying it should be done before we rip out internal functions for memory size. :) -Sterling

Andi Gutmans

22 years ago
Hey, Let's cut this thread because it's not going anywhere. Once 5.0 is out I'll try and play with a few things and see if memory consumption can be reduced. It's not something which I meant to happen for 5.0 anyway which is now in bug fix only mode. Whether it's by supporting a --lean-and-mean switch to disable some rinits and functions (maybe aliases as well), or changing some other aspects of PHP, it can be a good thing. I understand that merrily nuking a few functions from the PHP build won't do the trick, but not everyone is Y! and wants to run on a zillion 100 process httpd servers. Some actually prefer running a tenth of a zillion servers with 1000 httpd processes :) Anyway, I'm pretty sure some things can be done and I'll look into them after 5.0. It's bad timing now anyway before the RC. Not to do with memory, I still think ext/standard has become a junk yard and it would be cool to split that up. For those who are hard of hearing, I still mean for all that crap to be enabled by default (I hate breaking BC or having different default builds behave differently), but there is no reason not to enable a --lean-and-mean switch at some point to remove all the crap some ppl don't need. If you want all the crap and aliases, *just don't use that switch* :) Andi

Zeev Suraski

22 years ago
At 23:34 08/01/2004, Ilia Alshanetsky wrote:
>On January 08, 2004 04:11 pm, Zeev Suraski wrote: > > Well, if we really get to save 100K as Ilia imagined, for a thousand > > children, it's 100MB, and there are those with more. I doubt we can easily > > get 100K though. > >'Freeing' 100k is not as difficult as it sounds in PHP 5 when you consider >that 4 object files (filters.o 14460, ftp_fopen_wrapper.o 13616, info.o >13868, streamsfuncs.o 13408) make up for more then 1/2. Add (url_scanner_ex.o >12304, var.o 14616 image.o 11432,var_unserializer.o 5984) and you've reduced >your php library/binary by 100k.
Hrm, but that's not interesting at all, since that's server-wide memory that's shared across the children processes. Saving memory is only really interesting if we save per-process memory.
>Even so you gain only a 100mb saving (1000 simultaneous apache processes, >WOW) >per server. Given that 100mb of ram is only about $10 (or less) it seems to >me this would be a cheaper solution then having to pay a programmer >($50/hour+) to recompile PHP with obscure options (to reduce memory >footprint) on a variety of servers. I won't even go into the loss of >functionality that maybe needed later.
We're talking about servers where you can add no further memory because they're already maxed out. And they also use way more than 1,000 simultaneous processes, so those *THEORETIC* 100K could translate into half a gig, plenty of room to run additional processes. Zeev

George Schlossnagle

22 years ago
On Jan 8, 2004, at 4:40 PM, Zeev Suraski wrote:
> At 23:34 08/01/2004, Ilia Alshanetsky wrote: >> On January 08, 2004 04:11 pm, Zeev Suraski wrote: >> > Well, if we really get to save 100K as Ilia imagined, for a thousand >> > children, it's 100MB, and there are those with more. I doubt we >> can easily >> > get 100K though. >> >> 'Freeing' 100k is not as difficult as it sounds in PHP 5 when you >> consider >> that 4 object files (filters.o 14460, ftp_fopen_wrapper.o 13616, >> info.o >> 13868, streamsfuncs.o 13408) make up for more then 1/2. Add >> (url_scanner_ex.o >> 12304, var.o 14616 image.o 11432,var_unserializer.o 5984) and you've >> reduced >> your php library/binary by 100k. > > Hrm, but that's not interesting at all, since that's server-wide > memory that's shared across the children processes. Saving memory is > only really interesting if we save per-process memory. > >> Even so you gain only a 100mb saving (1000 simultaneous apache >> processes, WOW) >> per server. Given that 100mb of ram is only about $10 (or less) it >> seems to >> me this would be a cheaper solution then having to pay a programmer >> ($50/hour+) to recompile PHP with obscure options (to reduce memory >> footprint) on a variety of servers. I won't even go into the loss of >> functionality that maybe needed later. > > We're talking about servers where you can add no further memory > because they're already maxed out. And they also use way more than > 1,000 simultaneous processes, so those *THEORETIC* 100K could > translate into half a gig, plenty of room to run additional processes.
My inclination is that this is a pretty fringe usage of the software. I feel like I've been around the block a few times and haven't seen too may installations that large who had such tight per-process memory requirements, and even in applications where this is a serious concern (large Oracle installs being the first example that jumps to mind), the burden was usually on user-created data and not internal memory usage. This change (which I heard you don't support :) seems to have only a tiny possible benefit for a very small number of users and negatively affect script portability. As a complete aside - I've always found it to be really hard to avoid context-switching myself to death running near that many processes. I believe the Y! folks had similar experience. I'm having trouble envisioning an app were that runs well. Are you running on serious enterprise hw (os390/e10k/etc)? George

Rasmus Lerdorf

22 years ago
On Thu, 8 Jan 2004, George Schlossnagle wrote:
> As a complete aside - I've always found it to be really hard to avoid > context-switching myself to death running near that many processes. I > believe the Y! folks had similar experience. I'm having trouble > envisioning an app were that runs well. Are you running on serious > enterprise hw (os390/e10k/etc)?
Yup, thousands of little servers with less than 100 httpd children on each is how the biggest web load in the world is handled. -Rasmus

Christian Schneider

22 years ago
Rasmus Lerdorf wrote:
> Yup, thousands of little servers with less than 100 httpd children on each > is how the biggest web load in the world is handled.
I completely agree and think this is the way to do it. The only point to consider might be KeepAlive which binds processes without using any CPU/IO resources really. Maybe they could just switch off KeepAlive and have only 100 processes running :-) Or long static files which take forever to download without using CPU but that could easily be off-loaded to a light-weight http daemon. To me it sounds like they could use some consulting, maybe someone on this list could earn some money there :-) - Chris

Marcus Börger

22 years ago
Hello Zeev, Thursday, January 8, 2004, 10:40:19 PM, you wrote:
> At 23:34 08/01/2004, Ilia Alshanetsky wrote: >>On January 08, 2004 04:11 pm, Zeev Suraski wrote: >> > Well, if we really get to save 100K as Ilia imagined, for a thousand >> > children, it's 100MB, and there are those with more. I doubt we can easily >> > get 100K though. >> >>'Freeing' 100k is not as difficult as it sounds in PHP 5 when you consider >>that 4 object files (filters.o 14460, ftp_fopen_wrapper.o 13616, info.o >>13868, streamsfuncs.o 13408) make up for more then 1/2. Add (url_scanner_ex.o >>12304, var.o 14616 image.o 11432,var_unserializer.o 5984) and you've reduced >>your php library/binary by 100k.
> Hrm, but that's not interesting at all, since that's server-wide memory > that's shared across the children processes. Saving memory is only really > interesting if we save per-process memory.
>>Even so you gain only a 100mb saving (1000 simultaneous apache processes, >>WOW) >>per server. Given that 100mb of ram is only about $10 (or less) it seems to >>me this would be a cheaper solution then having to pay a programmer >>($50/hour+) to recompile PHP with obscure options (to reduce memory >>footprint) on a variety of servers. I won't even go into the loss of >>functionality that maybe needed later.
I couldn't agree more to what Ilia said. Also how do you count the apache eleohants in here? Wouldn't those people first drop some modules there? Or recompile their databases and drop some trigger languages or additional unused table handlers or other unused features?
> We're talking about servers where you can add no further memory because > they're already maxed out. And they also use way more than 1,000 > simultaneous processes, so those *THEORETIC* 100K could translate into half > a gig, plenty of room to run additional processes.
As far as i know the times where you want one big server for the inet are gone for years. Nowadays i'd go with some blades or 1U 19" chassis with slow cpus (lots of ram in case of java) and save lots of resources like power for example. Did i miss a revolution, are we back to big machines? Or are we speaking of the windows world where you cannot do anything else but working with big machines? But well the approach to not share the function table of buildin functions is something we should consider for php 5.1. Zeev do you think there's an easy way to write a wrapper around the hash functions so that we could hide the two hash table approach from the compiler/executor for example? marcus

Sterling Hughes

22 years ago
> At 20:57 08/01/2004, George Schlossnagle wrote: > > >On Jan 8, 2004, at 1:39 PM, Zeev Suraski wrote: > >> > >>Personally, I'm not convinced this is that case, even if the people we're > >>dealing with run thousands of Apache processes per server (which they do). > > > >Unless they're running thousands of apache server instances (not just > >children), shouldn't the memory sit in shared pages and thus be completely > >insignificant? > > > >And if they are running 1000s of instances per server, perhaps they have > >an architectural problem that this band-aid won't fix? :) > > Obviously we're talking about httpd children and not 1,000 > roots... Anyway, depending on the module, if there's any sort of RINIT > initialization, the CoW trick doesn't work very well and it consumes some > per-process memory. There's also the function table which is not shared > and will grow by a few dozen kilobytes thanks to the extra functions, which > is not shared across children. > > Note, my position at this point is that there's probably very little gain > in this as well, unless we can find a large number of modules with a > significant amount of functions, some of them maybe doing their own > per-request initialization, thus conserving a significant amount of > per-process memory. The list that Andi posted does not convince me that > this is the case. >
Yep. It also seems silly to reduce functionality based on the memory footprint of a few functions. I sorta agree with the RINIT argument (*), but if its just a function entry - there are many other ways to reduce PHP's memory usage before we get into removing builtin functions. -Sterling (*) I don't think it even has something to do with removing functions, looking at RINIT(basic) there are quite a few places that can be optimized without removing builtins.

Wez Furlong

22 years ago
As Stas mentioned, I'm not sure that this is a good idea, unless we split some of these things out of ext/standard and into their own extensions; ext/std_regex, ext/std_string (for levenstein, soundex, metaphone etc.), ext/std_hash (crc32, sha1) and have those extensions compiled in by default. IIRC, Jani was hoping for something like this a while back. Meanwhile, the best way to reduce footprint is to disable libxml extensions ;-) --Wez. ----- Original Message ----- From: "Andi Gutmans" <andi@zend.com> To: <internals@lists.php.net> Sent: Thursday, January 08, 2004 8:10 AM Subject: [PHP-DEV] Ability to lower PHP memory usage
> Hey, > > I just compiled PHP using --disable-all and realized that there's a shit > load of relatively useless stuff still being compiled into it. > The first thing I realized is that regex is still being compiled in by > default. Are any parts of PHP's core relying on regex, thus, requiring it > to be in the minimal build? I think most people today are using pcre and
it
> would be great if we could support a version without regex. > The second problem is in ext/standard with a zillion of functions. A lot
of
> them are in my opinion not used by the majority of users and it would be > great to be able not to compile them into PHP. Things that come to my
mind:
> a) crc32 > b) cyr_convert > c) image > d) lcg > e) metaphone > f) soundex > g) levenshtein > h) sha1 > i) uuencode > > and probably some other stuff, > > In order for PHP users to get the most out of their servers, being able to > reduce memory footprint as much as possible is quite important so that
they
> can increase MaxClients to as much as possible. > > As we're in a code freeze for RC1 it's a bit late to do something about
it,

Derick Rethans

22 years ago
On Thu, 8 Jan 2004, Wez Furlong wrote:
> As Stas mentioned, I'm not sure that this is a good idea, unless > we split some of these things out of ext/standard and into their > own extensions; ext/std_regex, ext/std_string (for levenstein, soundex, > metaphone etc.), ext/std_hash (crc32, sha1) and have those extensions > compiled in by default.
I also don't think it's a good idea, not even if we split out some of the functions to new 'extensions'. The power of PHP is that it had a lot of useful and powerful functions in the *core* and why would you want to change that? The problem with moving all those core functions to their own extensions is that they *can* be turned off too easily y hosters and then just the most simple scripts can break because of some ISP stupidity (and from what I know most of them are like that). Derick

Jani Taskinen

22 years ago
On Thu, 8 Jan 2004, Wez Furlong wrote:
>As Stas mentioned, I'm not sure that this is a good idea, unless >we split some of these things out of ext/standard and into their >own extensions; ext/std_regex, ext/std_string (for levenstein, soundex, >metaphone etc.), ext/std_hash (crc32, sha1) and have those extensions >compiled in by default. > >IIRC, Jani was hoping for something like this a while back.
I was thinking about it for increasing the maintainability of ext/standard, not to reduce the size of the binary. --Jani

Andi Gutmans

22 years ago
At 06:43 PM 1/8/2004 +0200, Jani Taskinen wrote:
>On Thu, 8 Jan 2004, Wez Furlong wrote: > > >As Stas mentioned, I'm not sure that this is a good idea, unless > >we split some of these things out of ext/standard and into their > >own extensions; ext/std_regex, ext/std_string (for levenstein, soundex, > >metaphone etc.), ext/std_hash (crc32, sha1) and have those extensions > >compiled in by default. > > > >IIRC, Jani was hoping for something like this a while back. > > I was thinking about it for increasing the maintainability > of ext/standard, not to reduce the size of the binary. >
Again, I am talking about having them compiled in by default. I wouldn't want to change that. However, for people who host their own application on a dedicated server, I want them to have the ability to use a stripped down version of PHP. And although code might be changed, the RINIT's of all sorts of functions to lead to copy-on-writes in all sorts of situations. I see no reason not to implement this because it doesn't change the default build one bit. Andi

Andrey Hristov

22 years ago
Andi Gutmans wrote:
> At 06:43 PM 1/8/2004 +0200, Jani Taskinen wrote: > >> On Thu, 8 Jan 2004, Wez Furlong wrote: >> >> >As Stas mentioned, I'm not sure that this is a good idea, unless >> >we split some of these things out of ext/standard and into their >> >own extensions; ext/std_regex, ext/std_string (for levenstein, soundex, >> >metaphone etc.), ext/std_hash (crc32, sha1) and have those extensions >> >compiled in by default. >> > >> >IIRC, Jani was hoping for something like this a while back. >> >> I was thinking about it for increasing the maintainability >> of ext/standard, not to reduce the size of the binary. >> > > Again, I am talking about having them compiled in by default. I wouldn't > want to change that. However, for people who host their own application > on a dedicated server, I want them to have the ability to use a stripped > down version of PHP. > And although code might be changed, the RINIT's of all sorts of > functions to lead to copy-on-writes in all sorts of situations. > I see no reason not to implement this because it doesn't change the > default build one bit. > > Andi >
Hi Andi, what's the problem of these ppl just to remove the functions they don't have directly from the source and after that to build. Removing a function from ext/standard is not that hard, and consists mostly of 3 steps. Maybe, for the ppl that want that hacking the source won't be a problem. You say that these core exts will be enabled by default but every ISP out there can use --disable-all and after that not to enable some parts. I think this is the point of Derick. And all this lead to a big number of bug reports why PHP fails with unexisting function for example array_map(). So my solution is a text file into the repository which explains how to strip functions. Ppl who want that are on their own. Andrey

Ilia A.

22 years ago
Removing few functions from standard will not save more then a few kilobytes from the final binary (stripped). However, it'll create problems for people who try to write portable scripts relying on those extensions. For example ext/ctype is pretty basic functionality and is even enabled by default. However, many servers do not have it requiring a PHP wrapper to do the same. While it's relatively simple to re-implement ctype, that is not the case for some of the functions you want to remove. Unless PHP is used for most basic tasks it is pretty useless with --disable-all. Removing regex is a fine idea, BUT it used by code like browscap internally. Cooperatively speaking it would be better (binary size wise) to use regex rather then PCRE, which while better in many respects is by no means more compact. -1 on the removal idea Ilia P.S. Here are the sizes of the striped object files for some of the functions you want to remove: metaphone.o (6156) soundex.o (840) uuencode.o (2308) crc32.o (1660) sha1.o (6696) levenshtein.o (3440) lcg.o (832) image.o (11432) cyr_convert.o (4104) Overall saving <40kb, which is about 2.5% of the size of a PHP binary On January 08, 2004 03:10 am, Andi Gutmans wrote: