What is the use of "unicode.semantics" in PHP 6?

And you are trying to make sure that new features that you like are turned on by default even when they break things for others. Some people are proposing changes that will enforce your preferred options without leaving any options to others. If I don't complain, PHP developers might do what you are proposing just like they switched unicode_semantics to PHP_INI_SYSTEM. I can't understand performance reasons without numbers that prove that, but I can see when somebody is trying to break things in my code and leaves me without any good options for fixing the code. It is possible that I am not correct. Writing functions in native language is one of key points in Andrei's presentation. http://www.gravitonic.com/talks/, "Unicoding With PHP 6" php|tek 2007 Chicago. Slides 49, 50 Other key points are about evaluating string length (slide 25) and offsets (slide 47), collation (slide 60,61) and strtoupper/strtolower/strcasecmp (slide 66). String length and offsets can be implemented with PHP5 mbstring extension. If I use PHP5 strtolower/strtoupper/strcasecmp, I must assume that they are locale aware. These functions don't follow LC_CTYPE=C rules, when locale is not C. It is possible that I am not correct and I will be able to update code and make it work in PHP6, but in order to do that I will have to use language constructs that are not backwards compatible with older PHP versions (slide 26, 30). I won't be able to mix PHP6 code with PHP5 code and will have to maintain two different library versions for lots of string functions and lots of stream operations. I will have to spend my time in order to make sure that code works, when I am not the one who broke it. I have other bugs to fix and they have higher priority than fixing code broken by others. I have already tested code in PHP6 unicode_semantics=on. Thing broke on password encryption and fix was to do something with binary typecasting. I need more than information currently available in http://www.php.net/manual in order to fix it. Then fputs calls freak out with notices about downcoded buffers and I can't leave those notices unfixed due to error_reporting = 2047 + display_errors = on coding requirements. And I still haven't reached the point when code does 8bit string decoding. We are working on different code. You have code with some specific character set and you can control all strings. My code works with different character sets, different sources of 8bit data and I don't controls those 8bit strings. My experience with PHP4/5 shows that I can work with 8bit strings better than PHP interpreter. Interpreter wins only when I have to work with large mapping tables and even then it is not stable (iconv), not enabled by default (recode), limited (mbstring) or very limited (utf8_decode). Some day I will take some standalone library and will try to make it work in PHP6 unicode_semantics=on. Maybe then I'll stop spreading the FUD. But for now don't expect that I will remain silent, if you propose changes that break things in PHP5 - PHP6 backwards compatibility. I know that unicode_semantics=on breaks things in drastic ways or my experience is based only on unstable PHP6 development code and RC versions will be better.

Ok. International environment. Do you have strtoupper|strtolower|strcasecmp functions operating in LC_CTYPE=C without switching locale? If I remember correctly, PHP does not use those even internally and developers are constantly triggering same Turkish|Kurdish|Azerbaijani bug in different functions. If I want case conversion or case insensitive comparison functions to follow C rules and not LC_CTYPE=some_translation, I am forced to use own functions, because strtoupper|strtolower are definitely locale aware in PHP. PHP6 unicode.semantics=on reduces my options and forces me to recode all 8bit string operations. After recoding functions are not backwards compatible with anything lower than 5.2.1. Your slides show that unicode characters are defined with \u, yet you mess with octals (\300) and hexadecimals (\xC0). It is possible that I am not right and I will be able to do everything more efficiently in PHP6. But for now I have broken password encryption handling, broken work with binary strings and over noisy stream functions. And I still haven't checked how it will handle streams with data encoded in different character sets. I will be forced to recode the code if PHP6 forces me to work in unicode.semantics=on. Don't expect that I will praise PHP6 for that. You are helping same people, who ask others to turn on mbstring.func_overload in php.ini in order to get unicode support. You are not helping people who already have code working with 8bit strings in different character sets.

Sorry to interject, but just a quick slightly off topic note. In your earlier e-mail you've said that ----------------- I actually don't have a problem with 95% of PHP 6 installations turning off Unicode support and this being the default setting for ISP's. Full Unicode support in an application is a big commitment and it will take quite a bit of work. I just don't think that many people will invest the time and effort into doing this, but at the same time there will be large applications and services that have full control over their server settings that will make use of it. Think Flickr, Yahoo, Facebook, etc. ----------------- Since 95% of installations will not be using PHP6 (php6 without unicode is pretty much a slower version php5) for whatever reason, we need a common version for the other 95%. I think it is inevitable that there will be 2 continually developed versions of PHP out there, one for people who need unicode support in the way that is envisioned by PHP6 and one for people who don't need it. On 19-Jun-07, at 3:44 PM, Rasmus Lerdorf wrote:

Ilia Alshanetsky

Rasmus Lerdorf wrote:

I think I've pretty clearly stated why I don't like the idea (anymore).

What has changed is the fact that HEAD is becoming a mess (or already is). Separate tests for different modes, code duplication, etc. Maintaining all this is gonna be hell..and I'm not gonna volunteer doing it especially if it's totally unnecessary (IMO).

To be totally honest, as long as nobody pays us to do something the way they want, we should do it exactly how WE want. After all, the group doing most of the work is Y! (or former Y!) people? (not to forget 1 or 2 Zend folks) The rest of us are not getting paid for this, like for example Ilia. We have an oportunity to finally get rid of the burden of past "mistakes" but lets not add any replacements for them. ;) --Jani

On Tue, June 19, 2007 2:44 pm, Rasmus Lerdorf wrote:

If they switch to 6 with unicode off, and never ever get around to turning unicode on, will it really be any better? They'll just be running some weird-o setup that causes all kinds of bugs and issues and you'll have users with php 6 apps that won't work in php 6 and who submit bogus bug reports about it, because of the setting. A clean break is probably better, especially if it makes php 6 much more maintainable. Large-scale hosts won't switch to 6 any faster than they switched to 5, unless there are ZERO BC breaks. And nobody can guarantee zero breaks, because there are always buglets. The effort to have unicode off in 6 is probably larger than the effort to document what needs to be done to a PHP 5 app to make it be 6-friendly, or even write tools to auto-convert the buik of a script. If unicode semantics are "on" what exactly is borked in PHP 5? Can that be fixed to be BC without resorting to this toggle?

Derick Rethans wrote:

IMHO, how many hosts that run PHP6 with unicode support is moot - as long as those who need it get it and those that don't aren't penalized by having their PHP6 install "crippled" turning off unicode. Just the notion that disabling unicode somehow "cripples" PHP6 is quite startling. That aside, it seems to me that PHP5 support will be around for a while. People can take that time to learn. Some of the energy saved supporting this setting and it's underlying needs can be better devoted to some documentation for teaching people how to migrate to a unicode-aware programming world. FWIW then, do it now. Cut the cord. Remove the setting and be done with it. Frankly, I think the PHP team has done far too much hand-holding and coddling in the last few years (an admirable, very under-appreciated job I might add). I think the core team will have its hands full getting PHP6 out the door and keeping PHP5 reasonably maintained. Humbly, I believe adding unicode.semantics to the equation may be a heavy, heavy piece of straw that breaks a lot of backs. Joe User included. Best Regards Mike Robinson

Andi Gutmans wrote:

I think the arguement was rather that we put so much effort into maintaining BC, yet still there were some minor issues, but more importantly users were still reluctant to jump to PHP5. Partly because they could indeed attain PHP4/PHP5 compatibility with the same code. Therefore they could hold out longer before making the jump. regards, Lukas

Hi Andi, On 6/21/07, Andi Gutmans <andi@zend.com> wrote:

That's true, and this 90% doesn't care either about php6.

I was one who said it and I will said it again. I made many migrations. I confirmed it was not 100% BC and it required work. The problem (even worst after 5.x than between 4.x and 5.x) was all changes after 5.x. The new pedantic errors added in minor versions and other warnings, it was actually more painfull after 5.x than between two 5.x. For a simple reason, 4.x to 5.x issues were mostly known, it was wisely planed and decided.

I agree but we don't have enough resources (as far as I understand.) As you know, I don't consider maintaining 4.x is a lot of work. By maintaining I mean only security fixes. It can be done independently from any other releases and by completely different people.

It was not well done. A 3 commits release required months to be released, that's simply unreliable. If we like to keep every releases active for years, we have to change a couple of things in our model (see the archives to get some tips).

With all respects, how many extensions do you maintain? How many commits do you actually do? Please don't tell me that what I said about the extra work required about the two modes is BS. Well, you don't say directly but you include every argument in the BS :)

Most of us are against it for two reasons: - php6 without unicode makes no sense. The impact will be much more damageable than magic gpc on/off or any other flags we had until now - the extra work required I think we all agree than the slowness is not yet an argument. Even if it was, it is a non issue (as you well explained).

That's the worst thing to do. Breaking things between 6.0 and 6.x is a no-no here. In the same manner, if we keep the flag, we will have to keep it. Or why php7 will be a better moment than php6? That simply makes no sense. --Pierre

I agree totally on giving options. The users have the option to either use PHP 5 (non-unicode) or PHP 6 (PHP 5 + unicode).. So essentially we're just forking here, nothing special about that. (just that it happens inside the project, between two major versions) --Jani p.s. Andi, the outsiders like myself value people on what they actually commit, everything you do behind the curtains means nothing to us. On Wed, 2007-06-20 at 20:52 -0700, Andi Gutmans wrote: