Unicode conversion exceptions and memory leaks

php.internals

Andrei Zmievski

20 years ago
If you run this code in PHP 6 right now, you will get a nice memory leak notice. <?php unicode_set_error_mode(FROM_UNICODE, U_CONV_ERROR_STOP | U_CONV_ERROR_EXCEPTION); $u = "< \u3844 >"; try { $s = (binary)$u; } catch (UnicodeConversionException $e) { } ?> /homes/andrei/dev/php-src/Zend/zend_unicode.c(461) : Freeing 0x013FC4E4 (1 bytes), script=e.php From what Dmitry tells me, it is impossible to fix this with the way our exceptions are implemented currently. I trust him on this, but at the same time I think we need to do something. The conversions will be happening in a lot of place and being able to catch an exception, if one results, is an important part of the workflow. I think Dmitry proposed that we could re-write exceptions implementation, but also said that it would be a lot of work. I think we should seriously consider it, though. Thoughts? -Andrei

Andi Gutmans

20 years ago
As you know, exceptions were designed to be user-land and not for internal functionality. I warned about that more exceptions were being integrated into the extensions; and I especially warned of ideas of integrating them into language constructs such as type hints and conversions. I don't have a good solution for this right now, except for adding a lot of bulk to C extensions and the core and making them overly complicated. Anyway, I'll discuss with Dmitry and see if he has any ideas I didn't think of. re: this specific case. Is it really a good idea for a type conversion to throw an exception? People won't be expecting that from reading the code. They'd most likely only expect methods to throw exceptions... Andi At 03:44 PM 4/13/2006, Andrei Zmievski wrote:

Andrei Zmievski

20 years ago
By default type conversions will just output a warning. You will get exceptions if you set the U_CONV_ERROR_EXCEPTION flag. -Andrei On Apr 13, 2006, at 3:52 PM, Andi Gutmans wrote:

Andi Gutmans

20 years ago
Yeah but we can't only tailor to the default. If you cast "abc" to an integer today PHP will do the conversion (e.g. 0). I think we should stick to that paradigm and provide users with validation methods if they want to strictly validate... Andi At 03:56 PM 4/13/2006, Andrei Zmievski wrote:

Andrei Zmievski

20 years ago
I've had some time to think about this and Derick and I also kicked around some ideas in a private conversation. The situation I am talking about is really about exceptional circumstances, such as ISO-8859-1 string being treated as a UTF-8 one or some other condition that results in illegal sequences. This is very different from an unassigned character condition, which is handled by SUBST, SKIP, etc callbacks. I disagree with the notion that this is similar to (int)"foo" example. There, we have a well defined semantics that say "strings not starting with a number get converted to 0". Treating ISO-8859-1 data as UTF-8 is simply invalid and bad behavior and should not be encouraged by silently ignoring the conversion error. Now, I understand that there is resistance to the use of exceptions in this case and I see the point of those who are against them. My problem is this: if we do not throw exceptions, then all we are left with is a warning, which is not helpful if you want to determine in a programmatic fashion whether there was a conversion error. Sure, you can check the return value of unicode_decode(), or maybe even fread() and such, but it does not help with casting, concatenation, and other similar operations. So, we do need a mechanism for this and it has to be a fairly flexible one because libraries may want to do one thing on failure, and application itself -- another. The best Derick and I could come up with is a user-specified conversion error handler. It would be invoked only when the converter encounters an illegal sequence or other serious error. The existing subst, skip, etc error modes would still apply. The error handler signature would be something like: function my_handler($direction, $encoding, $string, $char_byte, $offset) { .. } Where $direction is the direction of conversion (FROM_UNICODE or TO_UNICODE), $encoding is the name of the encoding in use during the attempted conversion, $string is the source string that converter tried to process, $char_byte is either failed Unicode character or byte sequence (depending on direction), and $offset is the offset of that character/byte sequence in the source string. The user error handler then is free to silence the warning, throw an exception (throw UnicodeConversionException($message, $direction, $char_byte, $offset), or do something else. I have no yet decided whether it's a good idea to allow user handler to continue the conversion or not. I'd rather the conversion always stopped. -Andrei On Apr 13, 2006, at 4:02 PM, Andi Gutmans wrote:

Markus Fischer

20 years ago
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andrei Zmievski wrote:
> The best Derick and I could come up with is a user-specified conversion > error handler. It would be invoked only when the converter encounters an > illegal sequence or other serious error. The existing subst, skip, etc > error modes would still apply. The error handler signature would be > something like: > > function my_handler($direction, $encoding, $string, $char_byte, > $offset) { .. } > > Where $direction is the direction of conversion (FROM_UNICODE or > TO_UNICODE), $encoding is the name of the encoding in use during the > attempted conversion, $string is the source string that converter tried > to process, $char_byte is either failed Unicode character or byte > sequence (depending on direction), and $offset is the offset of that > character/byte sequence in the source string. The user error handler > then is free to silence the warning, throw an exception (throw > UnicodeConversionException($message, $direction, $char_byte, $offset), > or do something else. I have no yet decided whether it's a good idea to > allow user handler to continue the conversion or not. I'd rather the > conversion always stopped.
My problem with those handlers is always that my control flow suddenly gets completely interrupted, i.e. right in the middle the handler is called and I've no information which class, object, function, file, source, line, etc. This is, at best, annoying. That is why I love those exception. I'm in control. Not PHP. Not someone else. I decide whether I want to interrupt my control flow or not. Now, I also understand the "but average joe doesn't know exception" argument and also the other one from this thread that it may be just technical too complex/not worth it. On the other side I fail to see the real advantage of the handler, especially when it's yet to be decided whether continuing will be allowed or not. That's a "horror" (comes from a saying in german, don't know if it makes sense here in english) when you develop in or with frameworks which have strict control flows and semantics on how to handle cases. Maybe I exaggerate in this case, feel free to ignore it. I mean hell, I would even love it when a failed require would throw an exception ... thanks, - - Markus -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFERw4b1nS0RcInK9ARAj6fAKC+cJwqRAc3uWaN1nJQ25Wg+oFJSQCdHWBJ 2oHLQs3JLHf2T26ub08H4dc= =ADVn -----END PGP SIGNATURE-----

Derick Rethans

20 years ago
On Thu, 20 Apr 2006, Markus Fischer wrote:
> Andrei Zmievski wrote: > > The best Derick and I could come up with is a user-specified conversion > > error handler. It would be invoked only when the converter encounters an > > illegal sequence or other serious error. The existing subst, skip, etc > > error modes would still apply. The error handler signature would be > > something like: > > > > function my_handler($direction, $encoding, $string, $char_byte, > > $offset) { .. } > > > > Where $direction is the direction of conversion (FROM_UNICODE or > > TO_UNICODE), $encoding is the name of the encoding in use during the > > attempted conversion, $string is the source string that converter tried > > to process, $char_byte is either failed Unicode character or byte > > sequence (depending on direction), and $offset is the offset of that > > character/byte sequence in the source string. The user error handler > > then is free to silence the warning, throw an exception (throw > > UnicodeConversionException($message, $direction, $char_byte, $offset), > > or do something else. I have no yet decided whether it's a good idea to > > allow user handler to continue the conversion or not. I'd rather the > > conversion always stopped. > > My problem with those handlers is always that my control flow suddenly > gets completely interrupted, i.e. right in the middle the handler is > called and I've no information which class, object, function, file, > source, line, etc. This is, at best, annoying. > > That is why I love those exception. I'm in control. Not PHP. Not someone > else. I decide whether I want to interrupt my control flow or not.
But you can still do that, just throw the exception in the handler yourself like andrei mentioned.
> Now, I also understand the "but average joe doesn't know exception" > argument and also the other one from this thread that it may be just > technical too complex/not worth it.
But that was not the argument that Andrei was making it all here. Besides the point you just raised there is also the following issue which is best illustrated with an example: http://files.derickrethans.nl/pseudoandrei In the example with exceptions you see that the application actually aborts large parts (the "// lots of other shit 2" after the error), where in the error_handler case the application can just continue.
> On the other side I fail to see the real advantage of the handler, > especially when it's yet to be decided whether continuing will be > allowed or not.
Imperative in Andrei's text was "to continue the *conversion* or not.", and I think it should be up the the handler whether the *application* should continue. Illegal sequences are about the same as including a file which has a parse error. With this handler you can handle this situation which is mostly a problem for normal string *operations*. If your code needs to do something more special to handle possible broken input there are plenty of functions to do so (like checking the return value).
> That's a "horror" (comes from a saying in german, don't > know if it makes sense here in english) when you develop in or with > frameworks which have strict control flows and semantics on how to > handle cases. > > Maybe I exaggerate in this case, feel free to ignore it. I mean hell, I > would even love it when a failed require would throw an exception ...
But that would set a bad predecent. Derick
-- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org

Markus Fischer

20 years ago
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Derick Rethans wrote:
> http://files.derickrethans.nl/pseudoandrei
Ah great, it was just not getting "the point". Sorry for the spam. - - Markus -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.1 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFER4sy1nS0RcInK9ARAg8kAJ0ez95BjYDcuWmwc6JiupbvN6+0rgCgiC0C YWZEznPw1ds1DmhU14SDSdY= =+JwW -----END PGP SIGNATURE-----

Andrei Zmievski

20 years ago
So, no particular opinions on this, aside from Markus's? I hoped this proposal would mollify both camps.. -Andrei On Apr 19, 2006, at 2:32 PM, Andrei Zmievski wrote:

Andi Gutmans

20 years ago
I'm wondering whether it's technically feasible that any places where such a conversion could fail would be allowed to throw an exception (i.e. internal functions, stream handlers, INI reader, etc...) At 02:36 PM 4/24/2006, Andrei Zmievski wrote:

Andrei Zmievski

20 years ago
Right. Throwing exceptions from output handler or INI reader may not be optimal. What did you think about the user defined error handler though? -Andrei On Apr 24, 2006, at 10:16 PM, Andi Gutmans wrote:

Andrei Zmievski

20 years ago
I hope that my latest commits makes both camps happy. The handler signature is actually: function my_handler($direction, $encoding, $char_or_byte, $offset, $message) { .. } -Andrei On Apr 19, 2006, at 2:32 PM, Andrei Zmievski wrote:

Jochem Maas

20 years ago
Andi Gutmans wrote:
> As you know, exceptions were designed to be user-land and not for > internal functionality. I warned about that more exceptions were being > integrated into the extensions; and I especially warned of ideas of > integrating them into language constructs such as type hints and > conversions. > I don't have a good solution for this right now, except for adding a lot > of bulk to C extensions and the core and making them overly complicated. > Anyway, I'll discuss with Dmitry and see if he has any ideas I didn't > think of. > > re: this specific case. Is it really a good idea for a type conversion > to throw an exception? People won't be expecting that from reading the > code. They'd most likely only expect methods to throw exceptions...
I really enjoy using exceptions since their inclusion in php and the first thing I thought was "ow, that going to make going 'unicode' _alot_ harder than the transparent transition that was originally pitched to endusers", my second thought was "didn't the big cheeses decide that the engine wouldn't be throwing any exceptions?" without being disrepectful to the tons of great work going into making php(6?) unicode native I would suggest that from the point of view of the 'average' php coder Andi's concerns are spot on. rgds, Jochem.

Dmitry Stogov

20 years ago
In general, exceptions as result of type conversion are good. But they does NOT work in PHP, because of exception mechanism design/implementation. Half year ago we disabled exceptions in __toString() especially because of this reason. I prefer disable exceptions for unicode conversions too. Thanks. Dmitry.

Derick Rethans

20 years ago
On Fri, 14 Apr 2006, Dmitry Stogov wrote:
> In general, exceptions as result of type conversion are good. > But they does NOT work in PHP, because of exception mechanism > design/implementation. > > Half year ago we disabled exceptions in __toString() especially because of > this reason. > I prefer disable exceptions for unicode conversions too.
Yeah, I don't like it too much either. But remember that throwing exceptions is *not* the default here. The default is a warning which I even like less as you'll have to shut it up with the @ operator. Andrei, I think we discuss that at some point too, but I don't remember the outcome on that. Do you have the specific conversion error modes documented somewhere? regards, Derick
-- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org

Andi Gutmans

20 years ago
At 03:51 AM 4/14/2006, Derick Rethans wrote:
>On Fri, 14 Apr 2006, Dmitry Stogov wrote: > > > In general, exceptions as result of type conversion are good. > > But they does NOT work in PHP, because of exception mechanism > > design/implementation. > > > > Half year ago we disabled exceptions in __toString() especially because of > > this reason. > > I prefer disable exceptions for unicode conversions too. > >Yeah, I don't like it too much either. But remember that throwing >exceptions is *not* the default here. The default is a warning which I >even like less as you'll have to shut it up with the @ operator. > >Andrei, I think we discuss that at some point too, but I don't remember >the outcome on that. Do you have the specific conversion error modes >documented somewhere?
I don't think it matters whether it's default or not. I think we shouldn't enable it. As I mentioned in my previous email there are a few reasons for that. I think this should be rolled back from CVS and not supported. Andi

Pierre Joye

20 years ago
On 4/14/06, Dmitry Stogov <dmitry@zend.com> wrote:
> In general, exceptions as result of type conversion are good.
I disagree here, exceptions are... exceptions. That means they should be used in rare cases. Unicode (and string manipulation in gerenal) should not use exceptions. I see no cases which justify an exception.
> Half year ago we disabled exceptions in __toString() especially because of > this reason. > I prefer disable exceptions for unicode conversions too.
Same here, and even if PHP will allow it. String manipulations (unicode included) must remain a quiet and easy operation. This should be a goal for each extensions, give a way to make PHP silent (by default or not does not matter but it must be settable). --Pierre

Andi Gutmans

20 years ago
Amen. At 09:40 AM 4/14/2006, Pierre wrote:

Andrei Zmievski

20 years ago
We had a .. bit of discussion on IRC about this. I will post my reply soon, once I've had sufficient time to cool down. -Andrei On Apr 14, 2006, at 9:50 AM, Andi Gutmans wrote:

Michael Wallner

20 years ago
Pierre wrote:
> Same here, and even if PHP will allow it. String manipulations > (unicode included) must remain a quiet and easy operation. This should > be a goal for each extensions, give a way to make PHP silent (by > default or not does not matter but it must be settable).
So let's go ahead and get rid of those silly picky errors in ext/gd too? Regards,
-- Michael - <mike(@)php.net> http://dev.iworks.at/ext-http/http-functions.html.gz

Ilia A.

20 years ago
On 14-Apr-06, at 1:55 PM, Michael Wallner wrote:
> Pierre wrote: > >> Same here, and even if PHP will allow it. String manipulations >> (unicode included) must remain a quiet and easy operation. This >> should >> be a goal for each extensions, give a way to make PHP silent (by >> default or not does not matter but it must be settable). > > So let's go ahead and get rid of those silly picky errors in ext/gd > too? >
Picky errors? Ilia

Pierre Joye

20 years ago
On 4/14/06, Michael Wallner <mike@php.net> wrote:
> Pierre wrote: > > > Same here, and even if PHP will allow it. String manipulations > > (unicode included) must remain a quiet and easy operation. This should > > be a goal for each extensions, give a way to make PHP silent (by > > default or not does not matter but it must be settable). > > So let's go ahead and get rid of those silly picky errors in ext/gd too?
It is in progress, as I already said many times in the past weeks :) --Pierre