Everything

php

The way the engine works?

php.internals

Brian Moon

19 years ago

Ok, I am no C coder and I don't claim to be very smart about low level
parts of PHP.  But, IMO, this is bug.

http://bugs.php.net/bug.php?id=36924

I have created a test script that shows the insanity of this bug:

<?php

$str = "This is a medium length string";

$start = memory_get_usage();
for($x=1;$x<=100;$x++){
echo $str.": (3.5): ".time()."\n";
}
$first_growth = number_format(memory_get_usage() - $start);

$start = memory_get_usage();
for($x=1;$x<=100;$x++){
echo "$str: (3.5): ".time()."\n";
}
$growth = number_format(memory_get_usage() - $start);


echo "first growth: $first_growth\nsecond growth: $growth\n";

?>

Now, you turn that into a script that is going to loop millions of
times, building strings to build data and run for hours.  The next thing
you know, your cron job is using 450MB of memory and you try to figure
out what YOU did wrong.  Hours later you find this little jewel waiting
for you and realize you did not do anything wrong.  You just used the
language.

Everyone ignored my last email about memory waste and large arrays.  I
can only assume that you will ignore this one too.  I am starting to see
why Stefan became so bitter.

I have always tried to squash the "myth" that putting variables inside
strings in PHP was bad, but now, I think I will flip on that.

I tested as far back as PHP 4.3.9 and as new as 5.2.1.

--

Brian Moon
-------------
http://dealnews.com/
It's good to be cheap =)

Antony Dovgal

19 years ago

On 03/06/2007 12:34 AM, Brian Moon wrote:

> Ok, I am no C coder and I don't claim to be very smart about low level
> parts of PHP.  But, IMO, this is bug.
>
> http://bugs.php.net/bug.php?id=36924
>
> I have created a test script that shows the insanity of this bug:
>
> <?php
>
> $str = "This is a medium length string";
>
> $start = memory_get_usage();
> for($x=1;$x<=100;$x++){
>      echo $str.": (3.5): ".time()."\n";
> }
> $first_growth = number_format(memory_get_usage() - $start);
>
> $start = memory_get_usage();
> for($x=1;$x<=100;$x++){
>      echo "$str: (3.5): ".time()."\n";
> }
> $growth = number_format(memory_get_usage() - $start);
>
>
> echo "first growth: $first_growth\nsecond growth: $growth\n";
>
> ?>

Number of iterations | First growth | Second growth | Peak usage
100                      480 bytes      ~5Kb          ~73Kb
100 000                  556 bytes      ~32Kb         ~100Kb
1 000 000                556 bytes      ~32Kb         ~100Kb

Tested with PHP 5.2.1 and 5.2.2-dev on Linux (with --disable-debug).

> Now, you turn that into a script that is going to loop millions of
> times, building strings to build data and run for hours.  The next thing
> you know, your cron job is using 450MB of memory and you try to figure
> out what YOU did wrong.  Hours later you find this little jewel waiting
> for you and realize you did not do anything wrong.  You just used the
> language.

Yup, the language stole ~32Kb of your memory and used it to speedup the allocation of small chunks.

> Everyone ignored my last email about memory waste and large arrays.  I
> can only assume that you will ignore this one too.
> I am starting to see why Stefan became so bitter.

> I have always tried to squash the "myth" that putting variables inside
> strings in PHP was bad, but now, I think I will flip on that.

> I tested as far back as PHP 4.3.9 and as new as 5.2.1.

--
Wbr,
Antony Dovgal

Brian Moon

19 years ago

Antony Dovgal wrote:

> Yup, the language stole ~32Kb of your memory and used it to speedup the
> allocation of small chunks.

Ok, in my attempt to send a sane looking example, I cleaned up the problem.

<?php

$str = "This is a medium length string";

$start = memory_get_usage();
for($x=1;$x<=1000000;$x++){
$var = $str." ".$x.": (3.5): ".time()."\n";
}
$first_growth = number_format(memory_get_usage() - $start);

$start = memory_get_usage();
for($x=1;$x<=1000000;$x++){
$var = "$str $x: (3.5): ".time()."\n";
}
$growth = number_format(memory_get_usage() - $start);


echo "first growth: $first_growth\nsecond growth: $growth\n";

?>

The introduction of $x into the string makes the difference.

first growth: 892
second growth: 3,955,068

--

Brian Moon
-------------
http://dealnews.com/
It's good to be cheap =)

Stanislav Malyshev

19 years ago

> The introduction of $x into the string makes the difference.
>
> first growth: 892
> second growth: 3,955,068

Strange, I get:

first growth: 820
second growth: 56

Which build did you test?

--
Stanislav Malyshev, Zend Products Engineer
stas@zend.com  http://www.zend.com/

Rasmus Lerdorf

19 years ago

Brian Moon wrote:

> Antony Dovgal wrote:
>> Yup, the language stole ~32Kb of your memory and used it to speedup
>> the allocation of small chunks.
>
> Ok, in my attempt to send a sane looking example, I cleaned up the problem.
>
> <?php
>
> $str = "This is a medium length string";
>
> $start = memory_get_usage();
> for($x=1;$x<=1000000;$x++){
>      $var = $str." ".$x.": (3.5): ".time()."\n";
> }
> $first_growth = number_format(memory_get_usage() - $start);
>
> $start = memory_get_usage();
> for($x=1;$x<=1000000;$x++){
>      $var = "$str $x: (3.5): ".time()."\n";
> }
> $growth = number_format(memory_get_usage() - $start);
>
>
> echo "first growth: $first_growth\nsecond growth: $growth\n";
>
> ?>
>
> The introduction of $x into the string makes the difference.
>
> first growth: 892
> second growth: 3,955,068

I get:

first growth: 704
second growth: 32,264

with current PHP_5_2 checkout.  I don't have a 5.1.x handy with memory
limits compiled in.

-Rasmus

Brian Moon

19 years ago

Rasmus Lerdorf wrote:

>
> I get:
>
> first growth: 704
> second growth: 32,264
>
> with current PHP_5_2 checkout.  I don't have a 5.1.x handy with memory
> limits compiled in.

Starting to think this is more prevelant on Mac OS X.  My huge numbers
are coming from 5.2.0, on Mac OS X.  On my Linux boxes I am getting
smaller numbers, more like what you guys are reporting so I assume you
are using Linux.

Then there is this confusing result:

$ php test.php
first growth: 860
second growth: -12,145,628

That is on 5.2.0 on Linux.  to get a -12MB, it had to be that big or
bigger to start with at some point.  Anyone know what that is about?
Passing true to memory_get_usage only clouds things further as I get all
0 for memory growth on all platforms making me think that
memory_get_usage() without the true param (>=5.2) is useless.

However, despite all those tests (which you can never trust to show real
world perfectly) I changed the script I was writing to use NO vars in
strings.  It is a script that processes queued items.  As long as there
are items in the queue, it keeps running.  The strings I was building
were mostly SQL queries.  It is now holding at 11MB and not growing.
Before, I was having to kill it after an hour as it would grow to over
100MB of memory and growing.  That was all determined by looking at ps.
Also, what was taking hours, now only took 15 minutes while I was
writing this email.  All of this was on Linux.  My fear is that Mac OS X
is making an issue very obvious that is more subtle on Linux, but does
exist.  I will see if I can come up with a good example on Linux that
makes this more prevelant.  It may have to run for an hour, which I
understand is not the primary target of PHP.  I am a big proponent of
that philosophy.  But, there is a CLI and I am gonna use it.  I would
like it to be sane. =)

--

Brian Moon
-------------
http://dealnews.com/
It's good to be cheap =)

Rasmus Lerdorf

19 years ago

Brian Moon wrote:

> Rasmus Lerdorf wrote:
>>
>> I get:
>>
>> first growth: 704
>> second growth: 32,264
>>
>> with current PHP_5_2 checkout.  I don't have a 5.1.x handy with memory
>> limits compiled in.
>
> Starting to think this is more prevelant on Mac OS X.  My huge numbers
> are coming from 5.2.0, on Mac OS X.  On my Linux boxes I am getting
> smaller numbers, more like what you guys are reporting so I assume you
> are using Linux.

Nope, mine was from OSX:

3:45pm shiny:~> php uu
first growth: 704
second growth: 32,264

3:45pm shiny:~> php -v
PHP 5.2.2-dev (cli) (built: Feb 18 2007 00:11:29)
Copyright (c) 1997-2007 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies
3:45pm shiny:~> uname -a
Darwin shiny.lerdorf.com 8.8.0 Darwin Kernel Version 8.8.0: Fri Sep  8
17:18:57 PDT 2006; root:xnu-792.12.6.obj~1/RELEASE_PPC Power Macintosh
powerpc


-Rasmus

Stanislav Malyshev

19 years ago

> Starting to think this is more prevelant on Mac OS X.  My huge numbers
> are coming from 5.2.0, on Mac OS X.  On my Linux boxes I am getting
> smaller numbers, more like what you guys are reporting so I assume you
> are using Linux.

I vaguely remember there was some problem with calculating exact memory
usage sometime ago. Not sure it's 5.2.0 but it might be. I would first
verify what happens in 5.2.1.

> Passing true to memory_get_usage only clouds things further as I get all
> 0 for memory growth on all platforms making me think that
> memory_get_usage() without the true param (>=5.2) is useless.

Why it's useless? 0 means your script didn't change real memory usage -
but it wasn't supposed to - since PHP pre-allocated memory blocks
anyway, so to change real memory usage you have to allocate something
big and hold it.

--
Stanislav Malyshev, Zend Products Engineer
stas@zend.com  http://www.zend.com/

Antony Dovgal

19 years ago

On 03/06/2007 02:33 AM, Brian Moon wrote:

> Rasmus Lerdorf wrote:
>>
>> I get:
>>
>> first growth: 704
>> second growth: 32,264
>>
>> with current PHP_5_2 checkout.  I don't have a 5.1.x handy with memory
>> limits compiled in.
>
> Starting to think this is more prevelant on Mac OS X.  My huge numbers
> are coming from 5.2.0, on Mac OS X.

You should really upgrade to 5.2.1 first.

> On my Linux boxes I am getting
> smaller numbers, more like what you guys are reporting so I assume you
> are using Linux.

--
Wbr,
Antony Dovgal

Stanislav Malyshev

19 years ago

I think what is happening is that the Engine caches small-sized memory
blocks and does not really free them when they are deallocated, even
when they are not referenced anymore. The cache size is limited, so I
don't think you need to be concerned. If you still think it is a big
problem for you, you could set USE_ZEND_ALLOC environment variable to 0
and have the engine user regular malloc/free. Note that this would
probably harm your performance.

--
Stanislav Malyshev, Zend Products Engineer
stas@zend.com  http://www.zend.com/