Everything

php

Home » Project » Newsgroups » php.internals » [RFC] 64 bit improvements, open questions »

[RFC] 64 bit improvements, open questions

php.internals

Anatoliy Belsky

12 years ago

Hi,

as the discussion phase for this RFC nears to the finish and the patch
itself is huge, I'd like to use the last chance to discuss the open
questions and concerns. Here are once more the links to the RFC (with
several updates) and the porting guide

https://wiki.php.net/rfc/size_t_and_int64
http://git.php.net/?p=php-src.git;a=blob;f=compat/PECL_PORTING;hb=refs/heads/str_size_and_int64

The big open question from the previous discussion is how to handle
changed ZPP formats. The way I've suggested in the porting doc is using
ternary operator like (COMP ? "l" : "i"). Another way were to put the old
ZPP formats "lLsp" back and make them redundant to the new ones "iISP".
Both ways have their pro and contra, the second variant isn't done, but
can be done quickly.

Also one big question from the RFC which haven't been addressed is the
handling of the stale SAPIs. While porting it turned out, that more than
50% of SAPIs reference no more actively supported web servers, some of
them are even not available anymore, no packages in the current
distributions seem to exist, no chance to check if they even complaint
with PHP mainstream. The SAPIs ported so far - apache2handler, CGI, CLI,
embed, FPM, phpdbg.

Please respond also if you have any other concerns about this RFC.

The voting phase is likely to be started on Sunday, January 22.

Thanks

Anatol

Anatoliy Belsky

12 years ago

On Wed, January 22, 2014 09:20, Anatol Belski wrote:

> Hi,
>
>
> as the discussion phase for this RFC nears to the finish and the patch
> itself is huge, I'd like to use the last chance to discuss the open
> questions and concerns. Here are once more the links to the RFC (with
> several updates) and the porting guide
>
> https://wiki.php.net/rfc/size_t_and_int64
> http://git.php.net/?p=php-src.git;a=blob;f=compat/PECL_PORTING;hb=refs/hea
> ds/str_size_and_int64
>
> The big open question from the previous discussion is how to handle
> changed ZPP formats. The way I've suggested in the porting doc is using
> ternary operator like (COMP ? "l" : "i"). Another way were to put the old
>  ZPP formats "lLsp" back and make them redundant to the new ones "iISP".
> Both ways have their pro and contra, the second variant isn't done, but
> can be done quickly.
>
> Also one big question from the RFC which haven't been addressed is the
> handling of the stale SAPIs. While porting it turned out, that more than
> 50% of SAPIs reference no more actively supported web servers, some of
> them are even not available anymore, no packages in the current
> distributions seem to exist, no chance to check if they even complaint
> with PHP mainstream. The SAPIs ported so far - apache2handler, CGI, CLI,
> embed, FPM, phpdbg.
>
> Please respond also if you have any other concerns about this RFC.
>
>
> The voting phase is likely to be started on Sunday, January 22.
>

Sunday, January 26 - that's correct :)

Regards

Anatol

Christopher Jones

12 years ago

On 01/22/2014 01:09 AM, Anatol Belski wrote:

> On Wed, January 22, 2014 09:20, Anatol Belski wrote:
>> Hi,
>>
>>
>> as the discussion phase for this RFC nears to the finish and the patch
>> itself is huge, I'd like to use the last chance to discuss the open
>> questions and concerns. Here are once more the links to the RFC (with
>> several updates) and the porting guide
>>
>> https://wiki.php.net/rfc/size_t_and_int64
>> http://git.php.net/?p=php-src.git;a=blob;f=compat/PECL_PORTING;hb=refs/hea
>> ds/str_size_and_int64
>>
>> The big open question from the previous discussion is how to handle
>> changed ZPP formats. The way I've suggested in the porting doc is using
>> ternary operator like (COMP ? "l" : "i"). Another way were to put the old
>>   ZPP formats "lLsp" back and make them redundant to the new ones "iISP".
>> Both ways have their pro and contra, the second variant isn't done, but
>> can be done quickly.
>>
>> Also one big question from the RFC which haven't been addressed is the
>> handling of the stale SAPIs. While porting it turned out, that more than
>> 50% of SAPIs reference no more actively supported web servers, some of
>> them are even not available anymore, no packages in the current
>> distributions seem to exist, no chance to check if they even complaint
>> with PHP mainstream. The SAPIs ported so far - apache2handler, CGI, CLI,
>> embed, FPM, phpdbg.
>>
>> Please respond also if you have any other concerns about this RFC.
>>
>>
>> The voting phase is likely to be started on Sunday, January 22.
>>
>
> Sunday, January 26 - that's correct :)
>
> Regards
>
> Anatol
>
>

Hi Anatol,

This is a huge patch - kudos.  However it will cause some short term
instability.  It's the kind of deep change that typically would be
merged at the beginning, not the end, of a release process.

The new code will require time to stabilize.  For example, there are
some new OCI8 diffs that will require debugging.  There is one OCI8
code block in PHP-5.6 that isn't in the str_size_and_int64 branch.
(I'll send details separately).  It's likely other extensions will be
similarly subtly impacted.

The open issues (e.g. SAPI support) need to be resolved before
starting the vote, so the RFC direction is obvious.  Also the RFC
needs to discuss proposed voting options, prior to starting the vote.
Since that only gives a couple of days for discussion, I think you
should delay the vote.

The porting doc compat/PECL_PORTING should be merged into the RFC (and
removed from the tree) to capture as much information in the one place
as possible.

Chris

--
christopher.jones@oracle.com  http://twitter.com/ghrd
Free PHP & Oracle book:
http://www.oracle.com/technetwork/topics/php/underground-php-oracle-manual-098250.html

Pierre Joye

12 years ago

Hi Chris,

On Jan 22, 2014 8:37 PM, "Christopher Jones" <christopher.jones@oracle.com>
wrote:

> This is a huge patch - kudos.  However it will cause some short term
> instability.

If any instability it will be very quickly found.

> It's the kind of deep change that typically would be
> merged at the beginning, not the end, of a release process.

We are at the beginning.

> The new code will require time to stabilize.

It is actually very good, compare the tests results with other branches:

http://windows.php.net/downloads/snaps/ostc/pftt/STR_SIZE_AND_INT64/rfcb28ea/

> For example, there are
> some new OCI8 diffs that will require debugging.  There is one OCI8
> code block in PHP-5.6 that isn't in the str_size_and_int64 branch.

As Anatol asked many times, help is welcome. Which part is missing? In case
you cannot commit yourself.

> (I'll send details separately).  It's likely other extensions will be
> similarly subtly impacted.
>
> The open issues (e.g. SAPI support) need to be resolved before
> starting the vote, so the RFC direction is obvious.

All active and maintained SAPIs are supported. Porting dead cows is not
worth the effort.

>  Also the RFC
> needs to discuss proposed voting options, prior to starting the vote.

> Since that only gives a couple of days for discussion, I think you
> should delay the vote.

Well, it has been under discussions for quite some time already. The open
questions (to be chosen) are known. Did we forget something?

Thanks for your feedback!

Cheers,
Pierre

Anatoliy Belsky

12 years ago

Hi Chris,

On Wed, January 22, 2014 20:36, Christopher Jones wrote:

> The open issues (e.g. SAPI support) need to be resolved before
> starting the vote, so the RFC direction is obvious.  Also the RFC needs to
> discuss proposed voting options, prior to starting the vote. Since that
> only gives a couple of days for discussion, I think you should delay the
> vote.

I'm not sure taking SAPI and ZPP topics out of the scope apart into new
RFCs would make any sense? But we probably can do multple votings in the
same RFC. That will be probably the most plausible solution to decide, I
gonna to update it :)

> The porting doc compat/PECL_PORTING should be merged into the RFC (and
> removed from the tree) to capture as much information in the one place as
> possible.
>

I thought more like pick it out into a separate wiki page, as it's porting
specific. Still it will be linked from/to RFC, but the structure were
cleaner IMHO.

Regards

Anatol

Christopher Jones

12 years ago

On 01/23/2014 12:47 AM, Anatol Belski wrote:

> Hi Chris,
>
> On Wed, January 22, 2014 20:36, Christopher Jones wrote:
>> The open issues (e.g. SAPI support) need to be resolved before
>> starting the vote, so the RFC direction is obvious.  Also the RFC needs to
>> discuss proposed voting options, prior to starting the vote. Since that
>> only gives a couple of days for discussion, I think you should delay the
>> vote.
> I'm not sure taking SAPI and ZPP topics out of the scope apart into new
> RFCs would make any sense? But we probably can do multple votings in the
> same RFC. That will be probably the most plausible solution to decide, I
> gonna to update it :)

I'm not a huge fan of multiple votes.  I hope a consensus can be reached via
the mail list first.

>
>> The porting doc compat/PECL_PORTING should be merged into the RFC (and
>> removed from the tree) to capture as much information in the one place as
>> possible.
>>
> I thought more like pick it out into a separate wiki page, as it's porting
> specific. Still it will be linked from/to RFC, but the structure were
> cleaner IMHO.

It can be a separate section of the RFC.  This will make the project's
scope clear when it comes to voting on the RFC.

Chris

--
christopher.jones@oracle.com  http://twitter.com/ghrd
Free PHP & Oracle book:
http://www.oracle.com/technetwork/topics/php/underground-php-oracle-manual-098250.html

Anatoliy Belsky

12 years ago

Hi Chris,

On Thu, January 23, 2014 21:01, Christopher Jones wrote:

>

>
> On 01/23/2014 12:47 AM, Anatol Belski wrote:
>
>> Hi Chris,
>>
>>
>> On Wed, January 22, 2014 20:36, Christopher Jones wrote:
>>
>>> The open issues (e.g. SAPI support) need to be resolved before
>>> starting the vote, so the RFC direction is obvious.  Also the RFC
>>> needs to discuss proposed voting options, prior to starting the vote.
>>> Since that
>>> only gives a couple of days for discussion, I think you should delay
>>> the vote.
>> I'm not sure taking SAPI and ZPP topics out of the scope apart into new
>>  RFCs would make any sense? But we probably can do multple votings in
>> the same RFC. That will be probably the most plausible solution to
>> decide, I gonna to update it :)
>
> I'm not a huge fan of multiple votes.  I hope a consensus can be reached
> via the mail list first.
>>
>>> The porting doc compat/PECL_PORTING should be merged into the RFC
>>> (and
>>> removed from the tree) to capture as much information in the one place
>>> as possible.
>>>
>> I thought more like pick it out into a separate wiki page, as it's
>> porting specific. Still it will be linked from/to RFC, but the structure
>> were cleaner IMHO.
>
> It can be a separate section of the RFC.  This will make the project's
> scope clear when it comes to voting on the RFC.
>

I've moved the SAPIs question into the new RFC
https://wiki.php.net/rfc/removal_of_dead_sapis . I've got your point about
the multiple vote, however would let separate votes stay.

It's clear anyway that keeping old zpp specs makes the migration much more
easier. Another question there put to vote is "no macro renamings", so
that might change as well in the current patch. There are much more
potential voters than currently participate on this discussion, so any
consensus reached on the list were incomplete. Of course we see the
"trend".

For that reason I'd also let it stay as it is with the porting guide, as
it's written specifically for the current patch state. The vote can change
the patch, so rewriting all the docs again would make sense after it's
known how.

I just assume that the majority of the potential voters do follow this
discussions and could forge the opinion. That's why, I think it's better
to rely on the clear vote results with no room for ambiguity.

Regards

Anatol

Julien Pauli

12 years ago

On Wed, Jan 22, 2014 at 9:20 AM, Anatol Belski <ab@php.net> wrote:

> Hi,
>
> as the discussion phase for this RFC nears to the finish and the patch
> itself is huge, I'd like to use the last chance to discuss the open
> questions and concerns. Here are once more the links to the RFC (with
> several updates) and the porting guide
>
> https://wiki.php.net/rfc/size_t_and_int64
> http://git.php.net/?p=php-src.git;a=blob;f=compat/PECL_PORTING;hb=refs/heads/str_size_and_int64
>
> The big open question from the previous discussion is how to handle
> changed ZPP formats. The way I've suggested in the porting doc is using
> ternary operator like (COMP ? "l" : "i"). Another way were to put the old
> ZPP formats "lLsp" back and make them redundant to the new ones "iISP".
> Both ways have their pro and contra, the second variant isn't done, but
> can be done quickly.

That's a good question.
Having to check for a macro and adapt parameters at every zpp() call
would be painful, however we already do such things in other scenarios
to take care of compatibility, so ...
I would however rename the macro to something more meaningfull, same
for PHP_NEED_COMPAT which should be named something like
PHP_NEED_STRSIZE_COMPAT

> Also one big question from the RFC which haven't been addressed is the
> handling of the stale SAPIs. While porting it turned out, that more than
> 50% of SAPIs reference no more actively supported web servers, some of
> them are even not available anymore, no packages in the current
> distributions seem to exist, no chance to check if they even complaint
> with PHP mainstream. The SAPIs ported so far - apache2handler, CGI, CLI,
> embed, FPM, phpdbg.

I would suggest getting rid of those old SAPIs nobody use anymore.
Late 90's SAPI that are not used/maintained anymore can die , IMO.


I also have a memory usage question.
Have we measured the impact of extending all 32 ints to 64 ? Memory
usage should be higher, but how many higher ?

Then is all that compatible with OPCache ?
I guess yes and see nothing which would be blocking, anyway better ask.


Anyway I'm all OK to merge this to 5.6 (should the RFC be accepted),
this is a great push in PHP crossplatform handling, thanks for the
huge work done so far !


Julien

Pierre Joye

12 years ago

On Wed, Jan 22, 2014 at 11:56 AM, Julien Pauli <jpauli@php.net> wrote:

> I would suggest getting rid of those old SAPIs nobody use anymore.
> Late 90's SAPI that are not used/maintained anymore can die , IMO.

agreed.

> I also have a memory usage question.
> Have we measured the impact of extending all 32 ints to 64 ? Memory
> usage should be higher, but how many higher ?

For buffers (string), 4 bytes more as the length is now an integer
(always 32 bits, even on linux 64bit) and it will be size_t, which
will be 64bit from now on. No change for integers as we use long now,
which is 64bit already on 64bit systems (but windows, always 32bits
there).

Cheers,

--
Pierre

@pierrejoye | http://www.libgd.org

Anatoliy Belsky

12 years ago

Hi Julien,

On Wed, January 22, 2014 11:56, Julien Pauli wrote:

> On Wed, Jan 22, 2014 at 9:20 AM, Anatol Belski <ab@php.net> wrote:
>
>> Hi,
>>
>>
>> as the discussion phase for this RFC nears to the finish and the patch
>> itself is huge, I'd like to use the last chance to discuss the open
>> questions and concerns. Here are once more the links to the RFC (with
>> several updates) and the porting guide
>>
>> https://wiki.php.net/rfc/size_t_and_int64
>> http://git.php.net/?p=php-src.git;a=blob;f=compat/PECL_PORTING;hb=refs/h
>> eads/str_size_and_int64
>>
>> The big open question from the previous discussion is how to handle
>> changed ZPP formats. The way I've suggested in the porting doc is using
>> ternary operator like (COMP ? "l" : "i"). Another way were to put the
>> old ZPP formats "lLsp" back and make them redundant to the new ones
>> "iISP".
>> Both ways have their pro and contra, the second variant isn't done, but
>> can be done quickly.
>
> That's a good question.
> Having to check for a macro and adapt parameters at every zpp() call
> would be painful, however we already do such things in other scenarios to
> take care of compatibility, so ... I would however rename the macro to
> something more meaningfull, same for PHP_NEED_COMPAT which should be named
> something like PHP_NEED_STRSIZE_COMPAT
>

Lets rename that. As the header (coupled with the replace tool) might work
out of the box, one could even make a wrapper to use like
ZEND_ZPP_COMPAT("l", "i"). In complicated cases that header will need to
be customized.

>> Also one big question from the RFC which haven't been addressed is the
>> handling of the stale SAPIs. While porting it turned out, that more than
>>  50% of SAPIs reference no more actively supported web servers, some of
>>  them are even not available anymore, no packages in the current
>> distributions seem to exist, no chance to check if they even complaint
>> with PHP mainstream. The SAPIs ported so far - apache2handler, CGI,
>> CLI,
>> embed, FPM, phpdbg.
>
> I would suggest getting rid of those old SAPIs nobody use anymore.
> Late 90's SAPI that are not used/maintained anymore can die , IMO.
>

That's what I was tempted to express too :)

>
> I also have a memory usage question.
> Have we measured the impact of extending all 32 ints to 64 ? Memory
> usage should be higher, but how many higher ?

That's not tested, whereby it is expected to be some higher. One for zval,
as Pierre explained. Even for zval it might stay the same depending on
default/used field alignment (like on mashines with word size of 64 bit).
Two - for the places where datatypes in function signatures or stack
variables has to be adapted, like for string size. That's the matter of 4
bytes per item as well, whereby things on the stack don't really matter.
Lets see what snippets can be invented for such a test :)

> Then is all that compatible with OPCache ?
> I guess yes and see nothing which would be blocking, anyway better ask.
>

All the core extensions are ported so far, so opcache as well.

>
> Anyway I'm all OK to merge this to 5.6 (should the RFC be accepted),
> this is a great push in PHP crossplatform handling, thanks for the huge
> work done so far !
>

That's a great thing, and thanks for the constructive feedback :)

Regards

Anatol

Christopher Jones

12 years ago

On 01/22/2014 12:20 AM, Anatol Belski wrote:

> Hi,
>
> as the discussion phase for this RFC nears to the finish and the patch
> itself is huge, I'd like to use the last chance to discuss the open
> questions and concerns. Here are once more the links to the RFC (with
> several updates) and the porting guide
>
> https://wiki.php.net/rfc/size_t_and_int64
> http://git.php.net/?p=php-src.git;a=blob;f=compat/PECL_PORTING;hb=refs/heads/str_size_and_int64
>
> The big open question from the previous discussion is how to handle
> changed ZPP formats. The way I've suggested in the porting doc is using
> ternary operator like (COMP ? "l" : "i"). Another way were to put the old
> ZPP formats "lLsp" back and make them redundant to the new ones "iISP".
> Both ways have their pro and contra, the second variant isn't done, but
> can be done quickly.

With 22 lines of compatibility macros borrowed from compat/compat.h, I
can make the "size_t_and_int64" branch of OCI8 compile with PHP
5.5. Making ZPP call changes to fix the formats would give a very much
fuglier code base.  There are 86 calls to ZPP in OCI8 (not all would
need changing).

Chris

--
christopher.jones@oracle.com  http://twitter.com/ghrd
Free PHP & Oracle book:
http://www.oracle.com/technetwork/topics/php/underground-php-oracle-manual-098250.html

Derick Rethans

12 years ago

On Wed, 22 Jan 2014, Christopher Jones wrote:

> On 01/22/2014 12:20 AM, Anatol Belski wrote:
> >
> > as the discussion phase for this RFC nears to the finish and the
> > patch itself is huge, I'd like to use the last chance to discuss the
> > open questions and concerns. Here are once more the links to the RFC
> > (with several updates) and the porting guide
> >
> > https://wiki.php.net/rfc/size_t_and_int64
> > http://git.php.net/?p=php-src.git;a=blob;f=compat/PECL_PORTING;hb=refs/heads/str_size_and_int64
> >
> > The big open question from the previous discussion is how to handle
> > changed ZPP formats. The way I've suggested in the porting doc is
> > using ternary operator like (COMP ? "l" : "i"). Another way were to
> > put the old ZPP formats "lLsp" back and make them redundant to the
> > new ones "iISP". Both ways have their pro and contra, the second
> > variant isn't done, but can be done quickly.
>
> With 22 lines of compatibility macros borrowed from compat/compat.h, I
> can make the "size_t_and_int64" branch of OCI8 compile with PHP
> 5.5. Making ZPP call changes to fix the formats would give a very much
> fuglier code base.  There are 86 calls to ZPP in OCI8 (not all would
> need changing).

That would make me a sad panda, I mean, having to #ifdef lots of ZPP
calls. Please try to avoid this necessity. As the 64-bit branch is
really most important for the Windows port—most Unix-like users run on
64-bit already anyway—the difficulty of supporting it should not be
pushed on already existing extensions.

cheers,
Derick

--
http://derickrethans.nl | http://xdebug.org
Like Xdebug? Consider a donation: http://xdebug.org/donate.php
twitter: @derickr and @xdebug
Posted with an email client that doesn't mangle email: alpine

Pierre Joye

12 years ago

On Thu, Jan 23, 2014 at 12:38 PM, Derick Rethans <derick@php.net> wrote:

> On Wed, 22 Jan 2014, Christopher Jones wrote:
>
>> On 01/22/2014 12:20 AM, Anatol Belski wrote:
>> >
>> > as the discussion phase for this RFC nears to the finish and the
>> > patch itself is huge, I'd like to use the last chance to discuss the
>> > open questions and concerns. Here are once more the links to the RFC
>> > (with several updates) and the porting guide
>> >
>> > https://wiki.php.net/rfc/size_t_and_int64
>> > http://git.php.net/?p=php-src.git;a=blob;f=compat/PECL_PORTING;hb=refs/heads/str_size_and_int64
>> >
>> > The big open question from the previous discussion is how to handle
>> > changed ZPP formats. The way I've suggested in the porting doc is
>> > using ternary operator like (COMP ? "l" : "i"). Another way were to
>> > put the old ZPP formats "lLsp" back and make them redundant to the
>> > new ones "iISP". Both ways have their pro and contra, the second
>> > variant isn't done, but can be done quickly.
>>
>> With 22 lines of compatibility macros borrowed from compat/compat.h, I
>> can make the "size_t_and_int64" branch of OCI8 compile with PHP
>> 5.5. Making ZPP call changes to fix the formats would give a very much
>> fuglier code base.  There are 86 calls to ZPP in OCI8 (not all would
>> need changing).
>
> That would make me a sad panda, I mean, having to #ifdef lots of ZPP
> calls. Please try to avoid this necessity. As the 64-bit branch is
> really most important for the Windows port—most Unix-like users run on
> 64-bit already anyway—the difficulty of supporting it should not be
> pushed on already existing extensions.

Yes, that's one of the open questions.

I think we can solve the zpp issue by using the same. However some
changes will be required and should be BC, especially for size_t, as
we really have to use it. I am not sure yet about the int64 usage vs
long without too much impact. It is especially important as long usage
is really not the way to go.

It is also very important to keep in mind that this patch is not only
about improving windows 64bits support, there is much more behind
that. Get rid of int for buffer size, right usage of 64bit integers,
it will drastically reduce the 64bit bugs (security one included).

Also about the PECL extensions problem, no matter which solution is
chosen, it is really easy to port them. The "people won't be able to
use pecl exts because they are not available" argument is somehow
pointless. If maintainers do not port their extension to the latest
PHP version within a couple of months after its release, or even
during the release phases, they won't do it for the next version
either. We can't delay a change forever because of them. Also my
experience is that most used and maintained extensions are ported very
quickly, during the release phases. Dead cows remain dead.

Cheers,

--
Pierre

@pierrejoye | http://www.libgd.org

Hannes Magnusson

12 years ago

On Wed, Jan 22, 2014 at 12:20 AM, Anatol Belski <ab@php.net> wrote:

> Also one big question from the RFC which haven't been addressed is the
> handling of the stale SAPIs. While porting it turned out, that more than
> 50% of SAPIs reference no more actively supported web servers, some of

That sounds like should be a separate RFC "killing old sapis" since
thats a different subject completely, even though that decision has
consequences for this rfc.

-Hannes

Christopher Jones

12 years ago

On 01/23/2014 10:43 AM, Hannes Magnusson wrote:

> On Wed, Jan 22, 2014 at 12:20 AM, Anatol Belski <ab@php.net> wrote:
>
>> Also one big question from the RFC which haven't been addressed is the
>> handling of the stale SAPIs. While porting it turned out, that more than
>> 50% of SAPIs reference no more actively supported web servers, some of
>
>
> That sounds like should be a separate RFC "killing old sapis" since
> thats a different subject completely, even though that decision has
> consequences for this rfc.
>
> -Hannes
>

Anatol,

A separate RFC would certainly help future searchability.
The RFC doesn't need to be long.

Chris

--
christopher.jones@oracle.com  http://twitter.com/ghrd
Free PHP & Oracle book:
http://www.oracle.com/technetwork/topics/php/underground-php-oracle-manual-098250.html