Large file support for PHP

sean finney

18 years ago

hi,

some non-php-dev comments in case they're helpful...

On Monday 15 October 2007 12:13:50 am Wez Furlong wrote:

> This bug has been open for a while:
> http://bugs.php.net/bug.php?id=27792

> Having run into this issue recently, here's a patch (hopefully
> attached, mail.app and list filters willing) against PHP 5.3 to
> address it.

using size_t/off_t is always good

> This patch will promote to double the file sizes that overflow
> LONG_MAX, which works transparently for my script.
> Note that this only touches the obvious functions in the core; there
> may be other extensions that need to behave similarly.

why not just use SIZE_MAX, and not have to worry about anything bigger?  I
can't profess to fully understand what's going on, but it looks like the
functions are being converted from returning a long to returning either a
long or a double, and then the result is parsed as a string???

> The defines that I jam into CFLAGS are known to work in our other
> projects on Linux, Solaris, FreeBSD and OSX.  The other (weirder)
> platforms might need some other adjustments to work correctly.

"getconf LFS_CFLAGS", assuming "getconf" exists...

sean

Stefan Esser

18 years ago

Hi,

please keep in mind that compiling PHP with large file support breaks
binary compatibility...
One of the globals contain a "stat" struct that has different size for
LFS or no LFS.

Stefan Esser

Wez Furlong schrieb:

> This bug has been open for a while:
> http://bugs.php.net/bug.php?id=27792
>
> Having run into this issue recently, here's a patch (hopefully
> attached, mail.app and list filters willing) against PHP 5.3 to
> address it.
> This patch will promote to double the file sizes that overflow
> LONG_MAX, which works transparently for my script.
> Note that this only touches the obvious functions in the core; there
> may be other extensions that need to behave similarly.
>
> The defines that I jam into CFLAGS are known to work in our other
> projects on Linux, Solaris, FreeBSD and OSX.  The other (weirder)
> platforms might need some other adjustments to work correctly.
>
> --Wez.
>

--
Stefan Esser                 SektionEins GmbH
Tel. xxxxxxxxxxxxxxx         Ober Buschweg 9a
50999 Köln
stefan.esser@sektioneins.de  www.sektioneins.de

SektionEins GmbH, Standort Köln
Firmensitz: Ober Buschweg 9a, 50999 Köln
Registergericht: Amtsgericht Köln, HRB 59920
Geschäftsführer: Johann-Peter Hartmann

Joe Orton

18 years ago

On Mon, Oct 15, 2007 at 09:26:30AM +0200, Stefan Esser wrote:

> Hi,
>
> please keep in mind that compiling PHP with large file support breaks
> binary compatibility...
> One of the globals contain a "stat" struct that has different size for
> LFS or no LFS.

More harmfully, it also changes the ABI of any library/application
linked against PHP where that ABI uses off_t.  So PHP would see all
off_t's in the libraries API as 64-bit, whereas they were built as
32-bit.  This breaks structure offsets etc very badly.

The way this is done for e.g APR avoids the issue; using only
-D_LARGEFILE64_SOURCE, off64_t in place of off_t, and fstat64,
open64/O_LARGEFILE, etc.

joe

sean finney

18 years ago

hi,

On Monday 15 October 2007 09:26:30 am Stefan Esser wrote:

> please keep in mind that compiling PHP with large file support breaks
> binary compatibility...
> One of the globals contain a "stat" struct that has different size for
> LFS or no LFS.

yes, this is of course a big deal for some people, esp if you're using
proprietary software that's built against the "original" abi.  of course if
you're only using OSS extensions, you can simply recompile them against the
new api/abi and there's no drawback.

btw - in our (debian's) case, the benefits outweighed the consequences, so
we've been shipping with LFS turned on for somewhere around a year now.


sean

Stanislav Malyshev

18 years ago

> yes, this is of course a big deal for some people, esp if you're using
> proprietary software that's built against the "original" abi.  of course if
> you're only using OSS extensions, you can simply recompile them against the
> new api/abi and there's no drawback.

That's not only about OSS vs. proprietary - even if you use only OSS
extensions, but you do not compile everything from the source on each
install (and have many platforms to install on - such as big
organization with dozens of servers), you might have trouble.

--
Stanislav Malyshev, Zend Software Architect
stas@zend.com   http://www.zend.com/
(408)253-8829   MSN: stas@zend.com

sean finney

18 years ago

hi stanislav,

On Wednesday 17 October 2007 02:08:06 am Stanislav Malyshev wrote:

> > yes, this is of course a big deal for some people, esp if you're using
> > proprietary software that's built against the "original" abi.  of course
> > if you're only using OSS extensions, you can simply recompile them
> > against the new api/abi and there's no drawback.
>
> That's not only about OSS vs. proprietary - even if you use only OSS
> extensions, but you do not compile everything from the source on each
> install (and have many platforms to install on - such as big
> organization with dozens of servers), you might have trouble.

but this wouldn't be any different from any other api/abi bump.  furthermore,
in the context of running debian (and by extension ubuntu) servers, you
wouldn't need to do anything if you were only using the pre-packaged modules,
as they'd be compiled with the right options for you, ideally.  so you'd just
need to do an "apt-get upgrade", etc.  and if you were using other pecl
extensions, i think the command to update that isn't so complicated either :)


sean

Stanislav Malyshev

18 years ago

> but this wouldn't be any different from any other api/abi bump.  furthermore,

Of course. If there are two different PHP versions with different API
numbers, it's OK. What's less OK is when there's two PHP builds with
same API numbers which are binary incompatible.

> in the context of running debian (and by extension ubuntu) servers, you
> wouldn't need to do anything if you were only using the pre-packaged modules,

You mean - pre-packaged by Debian?

> need to do an "apt-get upgrade", etc.  and if you were using other pecl
> extensions, i think the command to update that isn't so complicated either :)

Compatibility is not complicated if you deal with one machine where you
control everything and can rebuild everything from source. It starts to
get worse when you have a lot of machines on which you want to achieve
some kind of common environment. Having indistinguishable binary
incompatible PHP versions complicates things.

--
Stanislav Malyshev, Zend Software Architect
stas@zend.com   http://www.zend.com/
(408)253-8829   MSN: stas@zend.com

sean finney

18 years ago

On Wednesday 17 October 2007 09:13:03 am Stanislav Malyshev wrote:

> Of course. If there are two different PHP versions with different API
> numbers, it's OK. What's less OK is when there's two PHP builds with
> same API numbers which are binary incompatible.

right.

> > in the context of running debian (and by extension ubuntu) servers, you
> > wouldn't need to do anything if you were only using the pre-packaged
> > modules,
>
> You mean - pre-packaged by Debian?

yeah.  there are a number of precompiled modules provided by the debian php
package as well as other precompiled pecl packages available in the debian
archive.  all of them are compiled against the same api, which is provided by
a php{4,5}-dev package, which you can install on your system if you want to
build your own modules as well.

> Compatibility is not complicated if you deal with one machine where you
> control everything and can rebuild everything from source. It starts to
> get worse when you have a lot of machines on which you want to achieve
> some kind of common environment. Having indistinguishable binary
> incompatible PHP versions complicates things.

i'd say this has more to do with the administration habits of the individual
sysadmin.  i for one would do my utmost to avoid ever finding myself in a
situation where  i was compiling php  (or any other moderately complex
software for that matter) from source on more than a couple different
OS/architecuture/distribution combinations, and instead try to consolidate
things.  for example, using the distributions' precompiled packages,
backported packages, 3rd party packages, compiling once and nfs-mounting,
rsyncing, etc. but anyway, this is maybe getting a little side tracked....


sean

Stanislav Malyshev

18 years ago

I didn't dive yet too deep into the patch, but shouldn't it be fixed on
stream level and not function level? I.e. there are a lot of functions
using streams (including files) - would they support bigger files too?

I also think that while using size_t is good, changing binary structures
might be rather dangerous, unless we can ensure all PHPs built on the
certain platform would use the same setting.

> Having run into this issue recently, here's a patch (hopefully attached,
> mail.app and list filters willing) against PHP 5.3 to address it.
> This patch will promote to double the file sizes that overflow LONG_MAX,
> which works transparently for my script.
> Note that this only touches the obvious functions in the core; there may
> be other extensions that need to behave similarly.
>
> The defines that I jam into CFLAGS are known to work in our other
> projects on Linux, Solaris, FreeBSD and OSX.  The other (weirder)
> platforms might need some other adjustments to work correctly.
>
> --Wez.
>

--
Stanislav Malyshev, Zend Software Architect
stas@zend.com   http://www.zend.com/
(408)253-8829   MSN: stas@zend.com

Wez Furlong

18 years ago

On Oct 15, 2007, at 1:41 PM, Stanislav Malyshev wrote:

> I didn't dive yet too deep into the patch, but shouldn't it be
> fixed on stream level and not function level? I.e. there are a lot
> of functions using streams (including files) - would they support
> bigger files too?

Yes, the patch does that; it turns on LFS in the headers, which
promotes the off_t and size_t types that are used by streams to the
64-bit versions.  This is the one liner in configure.in.
The other larger part of the patch is to make PHP functions capable
of returning and accepting numbers that are too big to fit into a long.

> I also think that while using size_t is good, changing binary
> structures might be rather dangerous, unless we can ensure all PHPs
> built on the certain platform would use the same setting.

That's why 5.3 is a good point to apply this patch, rather than
sneaking it into 5.2.x

--Wez.

Stanislav Malyshev

18 years ago

> Yes, the patch does that; it turns on LFS in the headers, which promotes
> the off_t and size_t types that are used by streams to the 64-bit
> versions.  This is the one liner in configure.in.
> The other larger part of the patch is to make PHP functions capable of
> returning and accepting numbers that are too big to fit into a long.

I think it's better to do it in .h file and not CFLAGS (or both?) so
that if somebody includes PHP headers they would always get the same
result. This would prevent one from building incompatible module.

Is it guaranteed that once you have this define you would always get the
same result on all builds (regardless of libc difference, etc.)?

--
Stanislav Malyshev, Zend Software Architect
stas@zend.com   http://www.zend.com/
(408)253-8829   MSN: stas@zend.com

Wez Furlong

18 years ago

You might have a point there; I'd assumed that CFLAGS made it through
to php-config, but it doesn't look like they do.
It should be a simple matter to define them in php_config.h instead.

--Wez.

On Oct 16, 2007, at 6:16 PM, Stanislav Malyshev wrote:

sean finney

18 years ago

hi,

On Monday 15 October 2007 07:41:11 pm Stanislav Malyshev wrote:

> I didn't dive yet too deep into the patch, but shouldn't it be fixed on
> stream level and not function level? I.e. there are a lot of functions
> using streams (including files) - would they support bigger files too?

i would suggest that anywhere where one is doing something with a size or
offset and not using the posix size_t/off_t types should get such changes.
and like i said, i don't see the motivation behind this extra step of
returning the size in double form if it's bigger than LONG_MAX.

> I also think that while using size_t is good, changing binary structures
> might be rather dangerous, unless we can ensure all PHPs built on the
> certain platform would use the same setting.

i don't think switching from long/int -> size_t is a problem in the scope of
function internal variables.  the only place where you need to worry about
this is in headers/structs/function declarations that are exported to the
API/ABI.


sean

Wez Furlong

18 years ago

On Oct 16, 2007, at 2:44 AM, sean finney wrote:

>
> i would suggest that anywhere where one is doing something with a
> size or
> offset and not using the posix size_t/off_t types should get such
> changes.
> and like i said, i don't see the motivation behind this extra step of
> returning the size in double form if it's bigger than LONG_MAX.

PHP's native integer type is long, how else are you going to relay
numbers longer than a long back to the script without rewriting the
engine to add additional integer types, which is a massive changeset?

> i don't think switching from long/int -> size_t is a problem in the
> scope of
> function internal variables.  the only place where you need to
> worry about
> this is in headers/structs/function declarations that are exported
> to the
> API/ABI.

Or just bump our API number(s) and "not worry about it", since the
module loading code will refuse to load an incompatible module.

PHP 5.3 is an ideal point to make this kind of change, as I've
already stated.

--Wez.

sean finney

18 years ago

On Tuesday 16 October 2007 04:37:57 pm Wez Furlong wrote:

> PHP's native integer type is long, how else are you going to relay
> numbers longer than a long back to the script without rewriting the
> engine to add additional integer types, which is a massive changeset?

okay, chalk this up to my IANAPHPD ignorance.

>
> Or just bump our API number(s) and "not worry about it", since the
> module loading code will refuse to load an incompatible module.
>
> PHP 5.3 is an ideal point to make this kind of change, as I've
> already stated.

the general impression that i got was that zend/php did not want to break
their abi because then they'd have to support an additional set of platforms
for the zend optimiser.  but maybe that was just hearsay.


sean

Stanislav Malyshev

18 years ago

> Having run into this issue recently, here's a patch (hopefully attached,
> mail.app and list filters willing) against PHP 5.3 to address it.
> This patch will promote to double the file sizes that overflow LONG_MAX,
> which works transparently for my script.
> Note that this only touches the obvious functions in the core; there may
> be other extensions that need to behave similarly.
>
> The defines that I jam into CFLAGS are known to work in our other
> projects on Linux, Solaris, FreeBSD and OSX.  The other (weirder)
> platforms might need some other adjustments to work correctly.

I wonder - if we enable this patch, and there's some library that PHP
modules use, which is not compiled with 64-bit files support, will it
work OK (including using PHP file handles, calling file functions,
etc.)? I am worried about something like changing the structure of FILE*
and then both modules using different settings would access it. So do
you know how LF-enabled code interoperates with non-lf-enabled code?

Also I'm not sure MAKE_DOUBLE_ZVAL_INCREF is necessary - as I see, it's
used only in 2 places.

--
Stanislav Malyshev, Zend Software Architect
stas@zend.com   http://www.zend.com/
(408)253-8829   MSN: stas@zend.com