Unicode and fetch class

php.internals

Marcus Börger

20 years ago
Hello internals, Andrei asked me whether we could do something about the double lowercasing when calling zend_is_callable_check_func(). The attached patch adds the new fetch flag ZEND_FETCH_CLASS_NO_NORMALIZE which prevent lowercasing. Also zend_u_lookup_class_ex() got a new parameter which allows to specify whether the function needs to lowercase or not. I am quite sure Andrei needs to touch it a bit further but i think we should discuss this step first. Best regards, Marcus

Andrei Zmievski

20 years ago
I fixed some bugs and reworked the patch. Please look it over, as I intend to commit it in the next couple of days. The next step would be fixing zend_u_str_case_fold() to do only case folding and having all normalization go through zend_normalize_identifier(). -Andrei On Jul 15, 2006, at 12:52 PM, Marcus Boerger wrote:

Andrei Zmievski

20 years ago
Of course it would help if I attached the patch.. -Andrei

Marcus Börger

20 years ago
Hello Andrei, looks pretty fine to me, after your treatment, go ahead commit - Andi, Zeev? best regards marcus Monday, July 17, 2006, 9:50:44 PM, you wrote:
> Index: Zend/zend_API.c > =================================================================== > RCS file: /repository/ZendEngine2/zend_API.c,v > retrieving revision 1.378 > diff -p -u -r1.378 zend_API.c > --- Zend/zend_API.c 13 Jul 2006 21:27:48 -0000 1.378 > +++ Zend/zend_API.c 17 Jul 2006 19:47:30 -0000 > @@ -2578,10 +2578,9 @@ ZEND_API int zend_disable_class(char *cl > static int zend_is_callable_check_func(int check_flags, zval > ***zobj_ptr_ptr, zend_class_entry *ce_org, zval *callable, > zend_class_entry **ce_ptr, zend_function **fptr_ptr TSRMLS_DC) > { > int retval; > - zstr lcname, lmname, mname, colon; > + zstr lmname, mname, colon; > unsigned int clen, mlen; > zend_function *fptr; > - zend_class_entry **pce; > HashTable *ftable; > > *ce_ptr = NULL; > @@ -2601,18 +2600,7 @@ static int zend_is_callable_check_func(i > } > } > if (colon.v != NULL) { > - lcname = zend_u_str_case_fold(Z_TYPE_P(callable), > Z_UNIVAL_P(callable), clen, 0, &clen); > - /* caution: lcname is not '\0' terminated */ > - if (clen == sizeof("self") - 1 && > - ZEND_U_EQUAL(Z_TYPE_P(callable), lcname, clen, "self", sizeof("self")-1)) { > - *ce_ptr = EG(scope); > - } else if (clen == sizeof("parent") - 1 && > - ZEND_U_EQUAL(Z_TYPE_P(callable), lcname, clen, "parent", sizeof("parent")-1)) { > - *ce_ptr = EG(scope) ? EG(scope)->parent : NULL; > - } else if (zend_u_lookup_class(Z_TYPE_P(callable), > Z_UNIVAL_P(callable), clen, &pce TSRMLS_CC) == SUCCESS) { > - *ce_ptr = *pce; > - } > - efree(lcname.v); > + *ce_ptr = zend_u_fetch_class(Z_TYPE_P(callable), > Z_UNIVAL_P(callable), clen, ZEND_FETCH_CLASS_AUTO TSRMLS_CC); > if (!*ce_ptr) { > return 0; > } > @@ -2620,9 +2608,9 @@ static int zend_is_callable_check_func(i > if (ce_org && !instanceof_function(ce_org, *ce_ptr TSRMLS_CC)) { > return 0; > } > - lmname = zend_u_str_case_fold(Z_TYPE_P(callable), mname, mlen, 0, &mlen); > + lmname = zend_u_str_case_fold(Z_TYPE_P(callable), mname, mlen, 1, &mlen); > } else { > - lmname = zend_u_str_case_fold(Z_TYPE_P(callable), > Z_UNIVAL_P(callable), Z_UNILEN_P(callable), 0, &mlen); > + lmname = zend_u_str_case_fold(Z_TYPE_P(callable), > Z_UNIVAL_P(callable), Z_UNILEN_P(callable), 1, &mlen); > if (ce_org) { > ftable = &ce_org->function_table; > *ce_ptr = ce_org; > Index: Zend/zend_builtin_functions.c > =================================================================== > RCS file: /repository/ZendEngine2/zend_builtin_functions.c,v > retrieving revision 1.317 > diff -p -u -r1.317 zend_builtin_functions.c > --- Zend/zend_builtin_functions.c 9 Jul 2006 22:40:10 -0000 1.317 > +++ Zend/zend_builtin_functions.c 17 Jul 2006 19:47:30 -0000 > @@ -702,7 +702,7 @@ static void is_a_impl(INTERNAL_FUNCTION_ > > convert_to_text_ex(class_name); > > - if (zend_u_lookup_class_ex(Z_TYPE_PP(class_name), > Z_UNIVAL_PP(class_name), Z_UNILEN_PP(class_name), (instance_ce != NULL), &ce TSRMLS_CC) == FAILURE) { > + if (zend_u_lookup_class_ex(Z_TYPE_PP(class_name), > Z_UNIVAL_PP(class_name), Z_UNILEN_PP(class_name), (instance_ce != NULL) ? > 1 : 0, 1, &ce TSRMLS_CC) == FAILURE) { > retval = 0; > } else { > if (only_subclass) { > Index: Zend/zend_compile.h > =================================================================== > RCS file: /repository/ZendEngine2/zend_compile.h,v > retrieving revision 1.343 > diff -p -u -r1.343 zend_compile.h > --- Zend/zend_compile.h 13 Jun 2006 12:56:20 -0000 1.343 > +++ Zend/zend_compile.h 17 Jul 2006 19:47:30 -0000 > @@ -615,7 +615,9 @@ int zendlex(znode *zendlval TSRMLS_DC); > #define ZEND_FETCH_CLASS_GLOBAL 4 > #define ZEND_FETCH_CLASS_AUTO 5 > #define ZEND_FETCH_CLASS_INTERFACE 6 > -#define ZEND_FETCH_CLASS_NO_AUTOLOAD 0x80 > +#define ZEND_FETCH_CLASS_FLAGS 0xF0 > +#define ZEND_FETCH_CLASS_NO_NORMALIZE 0x10 > +#define ZEND_FETCH_CLASS_NO_AUTOLOAD 0x80 > > /* variable parsing type (compile-time) */ > #define ZEND_PARSED_MEMBER (1<<0) > Index: Zend/zend_execute.c > =================================================================== > RCS file: /repository/ZendEngine2/zend_execute.c,v > retrieving revision 1.748 > diff -p -u -r1.748 zend_execute.c > --- Zend/zend_execute.c 10 Jul 2006 00:32:23 -0000 1.748 > +++ Zend/zend_execute.c 17 Jul 2006 19:47:30 -0000 > @@ -944,7 +944,7 @@ fetch_string_dim: > UChar *norm; > int norm_len; > > - if (!zend_normalize_identifier(&norm, > &norm_len, offset_key.u, offset_key_length, 0)) { > + if (zend_normalize_identifier(&norm, > &norm_len, offset_key.u, offset_key_length, 0) == FAILURE) { > zend_error(E_WARNING, "Could not > normalize identifier: %r", offset_key); > } else if (norm != offset_key.u) { > offset_key.u = norm; > Index: Zend/zend_execute.h > =================================================================== > RCS file: /repository/ZendEngine2/zend_execute.h,v > retrieving revision 1.100 > diff -p -u -r1.100 zend_execute.h > --- Zend/zend_execute.h 31 May 2006 13:02:15 -0000 1.100 > +++ Zend/zend_execute.h 17 Jul 2006 19:47:30 -0000 > @@ -71,7 +71,7 @@ static inline void safe_free_zval_ptr_re > } > ZEND_API int zend_lookup_class(char *name, int name_length, zend_class_entry ***ce TSRMLS_DC); > ZEND_API int zend_u_lookup_class(zend_uchar type, zstr name, int > name_length, zend_class_entry ***ce TSRMLS_DC); > -ZEND_API int zend_u_lookup_class_ex(zend_uchar type, zstr name, int > name_length, int use_autoload, zend_class_entry ***ce TSRMLS_DC); > +ZEND_API int zend_u_lookup_class_ex(zend_uchar type, zstr name, int > name_length, int use_autoload, int do_normalize, zend_class_entry ***ce TSRMLS_DC); > ZEND_API int zend_eval_string(char *str, zval *retval_ptr, char *string_name TSRMLS_DC); > ZEND_API int zend_eval_string_ex(char *str, zval *retval_ptr, char > *string_name, int handle_exceptions TSRMLS_DC); > ZEND_API int zend_u_eval_string(zend_uchar type, zstr str, zval > *retval_ptr, char *string_name TSRMLS_DC); > Index: Zend/zend_execute_API.c > =================================================================== > RCS file: /repository/ZendEngine2/zend_execute_API.c,v > retrieving revision 1.375 > diff -p -u -r1.375 zend_execute_API.c > --- Zend/zend_execute_API.c 12 Jul 2006 07:54:18 -0000 1.375 > +++ Zend/zend_execute_API.c 17 Jul 2006 19:47:30 -0000 > @@ -784,7 +784,7 @@ int zend_call_function(zend_fcall_info * > } > > if (Z_TYPE_P(fci->function_name) == IS_UNICODE) { > - if ((colon.u = > u_strstr(Z_USTRVAL_P(fci->function_name), (UChar*)":\0:\0")) != NULL) { > + if ((colon.u = > u_strstr(Z_USTRVAL_P(fci->function_name), u_doublecolon)) != NULL) { > fname_len = u_strlen(colon.u+2); > clen = Z_USTRLEN_P(fci->function_name) - fname_len - 2; > fname.u = colon.u + 2; > @@ -799,7 +799,7 @@ int zend_call_function(zend_fcall_info * > if (colon.v != NULL) { > zend_class_entry **pce, *ce_child = NULL; > > - lcname = > zend_u_str_case_fold(Z_TYPE_P(fci->function_name), > Z_UNIVAL_P(fci->function_name), clen, 0, &clen); > + lcname = > zend_u_str_case_fold(Z_TYPE_P(fci->function_name), > Z_UNIVAL_P(fci->function_name), clen, 1, &clen); > /* caution: lcname is not '\0' terminated */ > if (calling_scope && clen == sizeof("self") - 1 && > ZEND_U_EQUAL(Z_TYPE_P(fci->function_name), > lcname, clen, "self", sizeof("self")-1)) { > @@ -807,7 +807,7 @@ int zend_call_function(zend_fcall_info * > } else if (calling_scope && clen == sizeof("parent") - 1 && > ZEND_U_EQUAL(Z_TYPE_P(fci->function_name), > lcname, clen, "parent", sizeof("parent")-1)) { > ce_child = EG(active_op_array) && > EG(active_op_array)->scope ? EG(scope)->parent : NULL; > - } else if > (zend_u_lookup_class(Z_TYPE_P(fci->function_name), > Z_UNIVAL_P(fci->function_name), clen, &pce TSRMLS_CC) == SUCCESS) { > + } else if > (zend_u_lookup_class_ex(Z_TYPE_P(fci->function_name), lcname, clen, 1, 0, &pce TSRMLS_CC) == SUCCESS) { > ce_child = *pce; > } > efree(lcname.v); > @@ -1062,7 +1062,7 @@ int zend_call_function(zend_fcall_info * > } > > > -ZEND_API int zend_u_lookup_class_ex(zend_uchar type, zstr name, int > name_length, int use_autoload, zend_class_entry ***ce TSRMLS_DC) > +ZEND_API int zend_u_lookup_class_ex(zend_uchar type, zstr name, int > name_length, int use_autoload, int do_normalize, zend_class_entry ***ce TSRMLS_DC) > { > zval **args[1]; > zval autoload_function; > @@ -1080,10 +1080,17 @@ ZEND_API int zend_u_lookup_class_ex(zend > return FAILURE; > } > > - lc_name = zend_u_str_case_fold(type, name, name_length, 1, &lc_name_len); > + if (do_normalize) { > + lc_name = zend_u_str_case_fold(type, name, name_length, 1, &lc_name_len); > + } else { > + lc_name = name; > + lc_name_len = name_length; > + } > > if (zend_u_hash_find(EG(class_table), type, lc_name, > lc_name_len+1, (void **) ce) == SUCCESS) { > - efree(lc_name.v); > + if (do_normalize) { > + efree(lc_name.v); > + } > return SUCCESS; > } > > @@ -1091,7 +1098,9 @@ ZEND_API int zend_u_lookup_class_ex(zend > * (doesn't impact fuctionality of __autoload() > */ > if (!use_autoload || zend_is_compiling(TSRMLS_C)) { > - efree(lc_name.v); > + if (do_normalize) { > + efree(lc_name.v); > + } > return FAILURE; > } > > @@ -1101,7 +1110,9 @@ ZEND_API int zend_u_lookup_class_ex(zend > } > > if (zend_u_hash_add(EG(in_autoload), type, lc_name, > lc_name_len+1, (void**)&dummy, sizeof(char), NULL) == FAILURE) { > - efree(lc_name.v); > + if (do_normalize) { > + efree(lc_name.v); > + } > return FAILURE; > } > > @@ -1143,12 +1154,16 @@ ZEND_API int zend_u_lookup_class_ex(zend > > if (retval == FAILURE) { > EG(exception) = exception; > - efree(lc_name.v); > + if (do_normalize) { > + efree(lc_name.v); > + } > return FAILURE; > } > > if (EG(exception) && exception) { > - efree(lc_name.v); > + if (do_normalize) { > + efree(lc_name.v); > + } > zend_error(E_ERROR, "Function %s(%R) threw an exception > of type '%v'", ZEND_AUTOLOAD_FUNC_NAME, type, name, Z_OBJCE_P(EG(exception))->name); > return FAILURE; > } > @@ -1160,13 +1175,15 @@ ZEND_API int zend_u_lookup_class_ex(zend > } > > retval = zend_u_hash_find(EG(class_table), type, lc_name, lc_name_len + 1, (void **) ce); > - efree(lc_name.v); > + if (do_normalize) { > + efree(lc_name.v); > + } > return retval; > } > > ZEND_API int zend_u_lookup_class(zend_uchar type, zstr name, int > name_length, zend_class_entry ***ce TSRMLS_DC) > { > - return zend_u_lookup_class_ex(type, name, name_length, 1, ce TSRMLS_CC); > + return zend_u_lookup_class_ex(type, name, name_length, 1, 1, ce TSRMLS_CC); > } > > ZEND_API int zend_lookup_class(char *name, int name_length, zend_class_entry ***ce TSRMLS_DC) > @@ -1545,9 +1562,12 @@ void zend_unset_timeout(TSRMLS_D) > ZEND_API zend_class_entry *zend_u_fetch_class(zend_uchar type, zstr > class_name, uint class_name_len, int fetch_type TSRMLS_DC) > { > zend_class_entry **pce; > - int use_autoload = (fetch_type & ZEND_FETCH_CLASS_NO_AUTOLOAD) == 0; > + int use_autoload = (fetch_type & ZEND_FETCH_CLASS_NO_AUTOLOAD) ? 0 : 1; > + int do_normalize = (fetch_type & ZEND_FETCH_CLASS_NO_NORMALIZE) ? 0 : 1; > + zstr lcname = class_name; > + > + fetch_type = fetch_type & ~ZEND_FETCH_CLASS_FLAGS; > > - fetch_type = fetch_type & ~ZEND_FETCH_CLASS_NO_AUTOLOAD; > check_fetch_type: > switch (fetch_type) { > case ZEND_FETCH_CLASS_SELF: > @@ -1564,15 +1584,22 @@ check_fetch_type: > } > return EG(scope)->parent; > case ZEND_FETCH_CLASS_AUTO: { > - fetch_type = > zend_get_class_fetch_type(type, class_name, class_name_len); > + if (do_normalize) { > + lcname = > zend_u_str_case_fold(type, class_name, class_name_len, 1, &class_name_len); > + } > + fetch_type = > zend_get_class_fetch_type(type, lcname, class_name_len); > if (fetch_type!=ZEND_FETCH_CLASS_DEFAULT) { > + if (do_normalize) { > + efree(lcname.v); > + do_normalize = 0; /* > we've normalized it already, don't do it twice */ > + } > goto check_fetch_type; > } > } > break; > } > > - if (zend_u_lookup_class_ex(type, class_name, class_name_len, > use_autoload, &pce TSRMLS_CC)==FAILURE) { > + if (zend_u_lookup_class_ex(type, lcname, class_name_len, > use_autoload, do_normalize, &pce TSRMLS_CC)==FAILURE) { > if (use_autoload) { > if (fetch_type == ZEND_FETCH_CLASS_INTERFACE) { > zend_error(E_ERROR, "Interface '%R' not found", type, class_name); > @@ -1580,8 +1607,14 @@ check_fetch_type: > zend_error(E_ERROR, "Class '%R' not found", type, class_name); > } > } > + if (lcname.v != class_name.v) { > + efree(lcname.v); > + } > return NULL; > } else { > + if (lcname.v != class_name.v) { > + efree(lcname.v); > + } > return *pce; > } > } > Index: Zend/zend_language_scanner.l > =================================================================== > RCS file: /repository/ZendEngine2/zend_language_scanner.l,v > retrieving revision 1.155 > diff -p -u -r1.155 zend_language_scanner.l > --- Zend/zend_language_scanner.l 12 Jun 2006 17:06:39 -0000 1.155 > +++ Zend/zend_language_scanner.l 17 Jul 2006 19:47:30 -0000 > @@ -444,7 +444,7 @@ static inline int zend_check_and_normali > efree(Z_USTRVAL_P(zendlval)); > return 0; > } > - if (!zend_normalize_identifier(&norm, &norm_len, > Z_USTRVAL_P(zendlval), Z_USTRLEN_P(zendlval), 0)) { > + if (zend_normalize_identifier(&norm, &norm_len, > Z_USTRVAL_P(zendlval), Z_USTRLEN_P(zendlval), 0) == FAILURE) { > zend_error(E_COMPILE_WARNING, "Could not normalize > identifier: %r", Z_USTRVAL_P(zendlval)); > efree(Z_USTRVAL_P(zendlval)); > return 0; > Index: Zend/zend_operators.c > =================================================================== > RCS file: /repository/ZendEngine2/zend_operators.c,v > retrieving revision 1.248 > diff -p -u -r1.248 zend_operators.c > --- Zend/zend_operators.c 12 Jul 2006 19:34:55 -0000 1.248 > +++ Zend/zend_operators.c 17 Jul 2006 19:47:30 -0000 > @@ -2109,14 +2109,21 @@ ZEND_API zstr zend_u_str_case_fold(zend_ > zstr ret; > > if (type == IS_UNICODE) { > - int ret_len; > + int ret_len = length; > > if (normalize) { > - zend_normalize_identifier(&ret.u, &ret_len, source.u, length, 1); > + if (zend_normalize_identifier(&ret.u, &ret_len, source.u, length, 1) == FAILURE) { > + zend_error(E_NOTICE, "Could not normalize identifier"); > + ret.u = eustrndup(source.u, length); > + } > } else { > UErrorCode status = U_ZERO_ERROR; > > zend_case_fold_string(&ret.u, &ret_len, source.u, > length, U_FOLD_CASE_DEFAULT, &status); > + if (U_FAILURE(status)) { > + zend_error(E_NOTICE, "Could not case-fold string"); > + ret.u = eustrndup(source.u, length); > + } > } > > *new_len = ret_len; > Index: Zend/zend_unicode.c > =================================================================== > RCS file: /repository/ZendEngine2/zend_unicode.c,v > retrieving revision 1.25 > diff -p -u -r1.25 zend_unicode.c > --- Zend/zend_unicode.c 12 Jul 2006 17:35:06 -0000 1.25 > +++ Zend/zend_unicode.c 17 Jul 2006 19:47:30 -0000 > @@ -736,7 +736,7 @@ ZEND_API int zend_normalize_identifier(U > if (unorm_quickCheck(ident, ident_len, UNORM_NFKC, &status) != UNORM_YES) { > zend_normalize_string(&buffer, &buffer_len, ident, ident_len, &status); > if (U_FAILURE(status)) { > - return 0; > + return FAILURE; > } > ident = buffer; > ident_len = buffer_len; > @@ -748,7 +748,7 @@ ZEND_API int zend_normalize_identifier(U > efree(ident); > } > if (U_FAILURE(status)) { > - return 0; > + return FAILURE; > } > ident = buffer; > ident_len = buffer_len; > @@ -759,7 +759,7 @@ ZEND_API int zend_normalize_identifier(U > efree(ident); > } > if (U_FAILURE(status)) { > - return 0; > + return FAILURE; > } > ident = buffer; > ident_len = buffer_len; > @@ -768,7 +768,7 @@ ZEND_API int zend_normalize_identifier(U > > *dest = ident; > *dest_len = ident_len; > - return 1; > + return SUCCESS; > } > /* }}} */ > > Index: ext/reflection/php_reflection.c > =================================================================== > RCS file: /repository/php-src/ext/reflection/php_reflection.c,v > retrieving revision 1.243 > diff -p -u -r1.243 php_reflection.c > --- ext/reflection/php_reflection.c 13 Jul 2006 12:34:30 -0000 1.243 > +++ ext/reflection/php_reflection.c 17 Jul 2006 19:47:31 -0000 > @@ -1972,7 +1972,7 @@ ZEND_METHOD(reflection_parameter, getCla > GET_REFLECTION_OBJECT_PTR(param); > > if (param->arg_info->class_name.v) { > - if > (zend_u_lookup_class_ex(UG(unicode)?IS_UNICODE:IS_STRING, > param->arg_info->class_name, param->arg_info->class_name_len, 1, &pce TSRMLS_CC) == FAILURE) { > + if > (zend_u_lookup_class_ex(UG(unicode)?IS_UNICODE:IS_STRING, > param->arg_info->class_name, param->arg_info->class_name_len, 1, 1, &pce TSRMLS_CC) == FAILURE) { > zend_throw_exception_ex(reflection_exception_ptr, 0 TSRMLS_CC, > "Class %v does not exist", param->arg_info->class_name); > return;
Best regards, Marcus

Andi Gutmans

20 years ago
Looks OK to me. Can we call it ZEND_FETCH_CLASS_NO_LOWERCASE instead of ZEND_FETCH_CLASS_NO_NORMALIZE. No biggy but I always think of normalizing as in taking a Unicode string and normalizing it. Or is that what you're actually doing here? Asking because we agreed we won't normalize automagically in PHP. Andi

Marcus Börger

20 years ago
Hello Andi, well when i first came up with the patch there was only lowercasing involved. So you have to ask our unicode guru for the current state :-) best regards marcus Tuesday, July 18, 2006, 1:08:52 AM, you wrote:
> Looks OK to me. Can we call it ZEND_FETCH_CLASS_NO_LOWERCASE instead of > ZEND_FETCH_CLASS_NO_NORMALIZE. No biggy but I always think of normalizing as > in taking a Unicode string and normalizing it. Or is that what you're > actually doing here? Asking because we agreed we won't normalize > automagically in PHP.
> Andi
>> -----Original Message----- >> From: Marcus Boerger [mailto:helly@php.net] >> Sent: Monday, July 17, 2006 2:49 PM >> To: Andrei Zmievski >> Cc: Marcus Boerger; Andrei Zmievski; Zeev Suraski; Andi >> Gutmans; internals@lists.php.net >> Subject: Re: [PHP-DEV] Unicode and fetch class >> >> Hello Andrei, >> >> looks pretty fine to me, after your treatment, go ahead >> commit - Andi, Zeev? >> >> best regards >> marcus >> >> Monday, July 17, 2006, 9:50:44 PM, you wrote: >> >> > Index: Zend/zend_API.c >> > =================================================================== >> > RCS file: /repository/ZendEngine2/zend_API.c,v >> > retrieving revision 1.378 >> > diff -p -u -r1.378 zend_API.c >> > --- Zend/zend_API.c 13 Jul 2006 21:27:48 -0000 1.378 >> > +++ Zend/zend_API.c 17 Jul 2006 19:47:30 -0000 >> > @@ -2578,10 +2578,9 @@ ZEND_API int zend_disable_class(char *cl >> > static int zend_is_callable_check_func(int check_flags, zval >> > ***zobj_ptr_ptr, zend_class_entry *ce_org, zval *callable, >> > zend_class_entry **ce_ptr, zend_function **fptr_ptr TSRMLS_DC) { >> > int retval; >> > - zstr lcname, lmname, mname, colon; >> > + zstr lmname, mname, colon; >> > unsigned int clen, mlen; >> > zend_function *fptr; >> > - zend_class_entry **pce; >> > HashTable *ftable; >> > >> > *ce_ptr = NULL; >> > @@ -2601,18 +2600,7 @@ static int zend_is_callable_check_func(i >> > } >> > } >> > if (colon.v != NULL) { >> > - lcname = zend_u_str_case_fold(Z_TYPE_P(callable), >> > Z_UNIVAL_P(callable), clen, 0, &clen); >> > - /* caution: lcname is not '\0' terminated */ >> > - if (clen == sizeof("self") - 1 && >> > - ZEND_U_EQUAL(Z_TYPE_P(callable), >> lcname, clen, "self", sizeof("self")-1)) { >> > - *ce_ptr = EG(scope); >> > - } else if (clen == sizeof("parent") - 1 && >> > - ZEND_U_EQUAL(Z_TYPE_P(callable), >> lcname, clen, "parent", sizeof("parent")-1)) { >> > - *ce_ptr = EG(scope) ? >> EG(scope)->parent : NULL; >> > - } else if (zend_u_lookup_class(Z_TYPE_P(callable), >> > Z_UNIVAL_P(callable), clen, &pce TSRMLS_CC) == SUCCESS) { >> > - *ce_ptr = *pce; >> > - } >> > - efree(lcname.v); >> > + *ce_ptr = zend_u_fetch_class(Z_TYPE_P(callable), >> > Z_UNIVAL_P(callable), clen, ZEND_FETCH_CLASS_AUTO TSRMLS_CC); >> > if (!*ce_ptr) { >> > return 0; >> > } >> > @@ -2620,9 +2608,9 @@ static int zend_is_callable_check_func(i >> > if (ce_org && !instanceof_function(ce_org, >> *ce_ptr TSRMLS_CC)) { >> > return 0; >> > } >> > - lmname = >> zend_u_str_case_fold(Z_TYPE_P(callable), mname, mlen, 0, &mlen); >> > + lmname = zend_u_str_case_fold(Z_TYPE_P(callable), >> > + mname, mlen, 1, &mlen); >> > } else { >> > - lmname = zend_u_str_case_fold(Z_TYPE_P(callable), >> > Z_UNIVAL_P(callable), Z_UNILEN_P(callable), 0, &mlen); >> > + lmname = zend_u_str_case_fold(Z_TYPE_P(callable), >> > Z_UNIVAL_P(callable), Z_UNILEN_P(callable), 1, &mlen); >> > if (ce_org) { >> > ftable = &ce_org->function_table; >> > *ce_ptr = ce_org; >> > Index: Zend/zend_builtin_functions.c >> > =================================================================== >> > RCS file: /repository/ZendEngine2/zend_builtin_functions.c,v >> > retrieving revision 1.317 >> > diff -p -u -r1.317 zend_builtin_functions.c >> > --- Zend/zend_builtin_functions.c 9 Jul 2006 22:40:10 >> -0000 1.317 >> > +++ Zend/zend_builtin_functions.c 17 Jul 2006 19:47:30 -0000 >> > @@ -702,7 +702,7 @@ static void is_a_impl(INTERNAL_FUNCTION_ >> > >> > convert_to_text_ex(class_name); >> > >> > - if (zend_u_lookup_class_ex(Z_TYPE_PP(class_name), >> > Z_UNIVAL_PP(class_name), Z_UNILEN_PP(class_name), (instance_ce != >> > NULL), &ce TSRMLS_CC) == FAILURE) { >> > + if (zend_u_lookup_class_ex(Z_TYPE_PP(class_name), >> > Z_UNIVAL_PP(class_name), Z_UNILEN_PP(class_name), >> (instance_ce != NULL) ? >> > 1 : 0, 1, &ce TSRMLS_CC) == FAILURE) { >> > retval = 0; >> > } else { >> > if (only_subclass) { >> > Index: Zend/zend_compile.h >> > =================================================================== >> > RCS file: /repository/ZendEngine2/zend_compile.h,v >> > retrieving revision 1.343 >> > diff -p -u -r1.343 zend_compile.h >> > --- Zend/zend_compile.h 13 Jun 2006 12:56:20 -0000 1.343 >> > +++ Zend/zend_compile.h 17 Jul 2006 19:47:30 -0000 >> > @@ -615,7 +615,9 @@ int zendlex(znode *zendlval TSRMLS_DC); >> > #define ZEND_FETCH_CLASS_GLOBAL 4 >> > #define ZEND_FETCH_CLASS_AUTO 5 >> > #define ZEND_FETCH_CLASS_INTERFACE 6 >> > -#define ZEND_FETCH_CLASS_NO_AUTOLOAD 0x80 >> > +#define ZEND_FETCH_CLASS_FLAGS 0xF0 >> > +#define ZEND_FETCH_CLASS_NO_NORMALIZE 0x10 #define >> > +ZEND_FETCH_CLASS_NO_AUTOLOAD 0x80 >> > >> > /* variable parsing type (compile-time) */ >> > #define ZEND_PARSED_MEMBER (1<<0) >> > Index: Zend/zend_execute.c >> > =================================================================== >> > RCS file: /repository/ZendEngine2/zend_execute.c,v >> > retrieving revision 1.748 >> > diff -p -u -r1.748 zend_execute.c >> > --- Zend/zend_execute.c 10 Jul 2006 00:32:23 -0000 1.748 >> > +++ Zend/zend_execute.c 17 Jul 2006 19:47:30 -0000 >> > @@ -944,7 +944,7 @@ fetch_string_dim: >> > UChar *norm; >> > int norm_len; >> > >> > - if >> (!zend_normalize_identifier(&norm, >> > &norm_len, offset_key.u, offset_key_length, 0)) { >> > + if (zend_normalize_identifier(&norm, >> > &norm_len, offset_key.u, offset_key_length, 0) == FAILURE) { >> > >> zend_error(E_WARNING, "Could >> > not normalize identifier: %r", offset_key); >> > } else if (norm != offset_key.u) { >> > offset_key.u = norm; >> > Index: Zend/zend_execute.h >> > =================================================================== >> > RCS file: /repository/ZendEngine2/zend_execute.h,v >> > retrieving revision 1.100 >> > diff -p -u -r1.100 zend_execute.h >> > --- Zend/zend_execute.h 31 May 2006 13:02:15 -0000 1.100 >> > +++ Zend/zend_execute.h 17 Jul 2006 19:47:30 -0000 >> > @@ -71,7 +71,7 @@ static inline void safe_free_zval_ptr_re } >> > ZEND_API int zend_lookup_class(char *name, int name_length, >> > zend_class_entry ***ce TSRMLS_DC); ZEND_API int >> > zend_u_lookup_class(zend_uchar type, zstr name, int name_length, >> > zend_class_entry ***ce TSRMLS_DC); -ZEND_API int >> > zend_u_lookup_class_ex(zend_uchar type, zstr name, int name_length, >> > int use_autoload, zend_class_entry ***ce TSRMLS_DC); >> > +ZEND_API int zend_u_lookup_class_ex(zend_uchar type, zstr name, int >> > name_length, int use_autoload, int do_normalize, zend_class_entry >> > ***ce TSRMLS_DC); ZEND_API int zend_eval_string(char *str, zval >> > *retval_ptr, char *string_name TSRMLS_DC); ZEND_API int >> > zend_eval_string_ex(char *str, zval *retval_ptr, char *string_name, >> > int handle_exceptions TSRMLS_DC); ZEND_API int >> > zend_u_eval_string(zend_uchar type, zstr str, zval >> *retval_ptr, char >> > *string_name TSRMLS_DC); >> > Index: Zend/zend_execute_API.c >> > =================================================================== >> > RCS file: /repository/ZendEngine2/zend_execute_API.c,v >> > retrieving revision 1.375 >> > diff -p -u -r1.375 zend_execute_API.c >> > --- Zend/zend_execute_API.c 12 Jul 2006 07:54:18 -0000 >> 1.375 >> > +++ Zend/zend_execute_API.c 17 Jul 2006 19:47:30 -0000 >> > @@ -784,7 +784,7 @@ int zend_call_function(zend_fcall_info * >> > } >> > >> > if (Z_TYPE_P(fci->function_name) == IS_UNICODE) { >> > - if ((colon.u = >> > u_strstr(Z_USTRVAL_P(fci->function_name), >> (UChar*)":\0:\0")) != NULL) >> > { >> > + if ((colon.u = >> > u_strstr(Z_USTRVAL_P(fci->function_name), u_doublecolon)) != NULL) { >> > fname_len = u_strlen(colon.u+2); >> > clen = >> Z_USTRLEN_P(fci->function_name) - fname_len - 2; >> > fname.u = colon.u + 2; @@ -799,7 >> > +799,7 @@ int zend_call_function(zend_fcall_info * >> > if (colon.v != NULL) { >> > zend_class_entry **pce, *ce_child = NULL; >> > >> > - lcname = >> > zend_u_str_case_fold(Z_TYPE_P(fci->function_name), >> > Z_UNIVAL_P(fci->function_name), clen, 0, &clen); >> > + lcname = >> > zend_u_str_case_fold(Z_TYPE_P(fci->function_name), >> > Z_UNIVAL_P(fci->function_name), clen, 1, &clen); >> > /* caution: lcname is not '\0' terminated */ >> > if (calling_scope && clen == >> sizeof("self") - 1 && >> > >> ZEND_U_EQUAL(Z_TYPE_P(fci->function_name), >> > lcname, clen, "self", sizeof("self")-1)) { @@ -807,7 +807,7 @@ int >> > zend_call_function(zend_fcall_info * >> > } else if (calling_scope && clen == >> sizeof("parent") - 1 && >> > >> ZEND_U_EQUAL(Z_TYPE_P(fci->function_name), >> > lcname, clen, "parent", sizeof("parent")-1)) { >> > ce_child = EG(active_op_array) && >> > EG(active_op_array)->scope ? EG(scope)->parent : NULL; >> > - } else if >> > (zend_u_lookup_class(Z_TYPE_P(fci->function_name), >> > Z_UNIVAL_P(fci->function_name), clen, &pce TSRMLS_CC) == SUCCESS) { >> > + } else if >> > (zend_u_lookup_class_ex(Z_TYPE_P(fci->function_name), >> lcname, clen, 1, 0, &pce TSRMLS_CC) == SUCCESS) { >> > ce_child = *pce; >> > } >> > efree(lcname.v); @@ -1062,7 +1062,7 @@ int >> > zend_call_function(zend_fcall_info * } >> > >> > >> > -ZEND_API int zend_u_lookup_class_ex(zend_uchar type, zstr >> name, int >> > name_length, int use_autoload, zend_class_entry ***ce TSRMLS_DC) >> > +ZEND_API int zend_u_lookup_class_ex(zend_uchar type, zstr name, int >> > name_length, int use_autoload, int do_normalize, zend_class_entry >> > ***ce TSRMLS_DC) { >> > zval **args[1]; >> > zval autoload_function; >> > @@ -1080,10 +1080,17 @@ ZEND_API int zend_u_lookup_class_ex(zend >> > return FAILURE; >> > } >> > >> > - lc_name = zend_u_str_case_fold(type, name, >> name_length, 1, &lc_name_len); >> > + if (do_normalize) { >> > + lc_name = zend_u_str_case_fold(type, name, >> name_length, 1, &lc_name_len); >> > + } else { >> > + lc_name = name; >> > + lc_name_len = name_length; >> > + } >> > >> > if (zend_u_hash_find(EG(class_table), type, lc_name, >> > lc_name_len+1, (void **) ce) == SUCCESS) { >> > - efree(lc_name.v); >> > + if (do_normalize) { >> > + efree(lc_name.v); >> > + } >> > return SUCCESS; >> > } >> > >> > @@ -1091,7 +1098,9 @@ ZEND_API int zend_u_lookup_class_ex(zend >> > * (doesn't impact fuctionality of __autoload() >> > */ >> > if (!use_autoload || zend_is_compiling(TSRMLS_C)) { >> > - efree(lc_name.v); >> > + if (do_normalize) { >> > + efree(lc_name.v); >> > + } >> > return FAILURE; >> > } >> > >> > @@ -1101,7 +1110,9 @@ ZEND_API int zend_u_lookup_class_ex(zend >> > } >> > >> > if (zend_u_hash_add(EG(in_autoload), type, lc_name, >> > lc_name_len+1, (void**)&dummy, sizeof(char), NULL) == FAILURE) { >> > - efree(lc_name.v); >> > + if (do_normalize) { >> > + efree(lc_name.v); >> > + } >> > return FAILURE; >> > } >> > >> > @@ -1143,12 +1154,16 @@ ZEND_API int zend_u_lookup_class_ex(zend >> > >> > if (retval == FAILURE) { >> > EG(exception) = exception; >> > - efree(lc_name.v); >> > + if (do_normalize) { >> > + efree(lc_name.v); >> > + } >> > return FAILURE; >> > } >> > >> > if (EG(exception) && exception) { >> > - efree(lc_name.v); >> > + if (do_normalize) { >> > + efree(lc_name.v); >> > + } >> > zend_error(E_ERROR, "Function %s(%R) threw an >> > exception of type '%v'", ZEND_AUTOLOAD_FUNC_NAME, type, >> name, Z_OBJCE_P(EG(exception))->name); >> > return FAILURE; >> > } >> > @@ -1160,13 +1175,15 @@ ZEND_API int zend_u_lookup_class_ex(zend >> > } >> > >> > retval = zend_u_hash_find(EG(class_table), type, >> lc_name, lc_name_len + 1, (void **) ce); >> > - efree(lc_name.v); >> > + if (do_normalize) { >> > + efree(lc_name.v); >> > + } >> > return retval; >> > } >> > >> > ZEND_API int zend_u_lookup_class(zend_uchar type, zstr name, int >> > name_length, zend_class_entry ***ce TSRMLS_DC) { >> > - return zend_u_lookup_class_ex(type, name, >> name_length, 1, ce TSRMLS_CC); >> > + return zend_u_lookup_class_ex(type, name, >> name_length, 1, 1, >> > + ce TSRMLS_CC); >> > } >> > >> > ZEND_API int zend_lookup_class(char *name, int name_length, >> > zend_class_entry ***ce TSRMLS_DC) @@ -1545,9 +1562,12 @@ void >> > zend_unset_timeout(TSRMLS_D) ZEND_API zend_class_entry >> > *zend_u_fetch_class(zend_uchar type, zstr class_name, uint >> > class_name_len, int fetch_type TSRMLS_DC) { >> > zend_class_entry **pce; >> > - int use_autoload = (fetch_type & >> ZEND_FETCH_CLASS_NO_AUTOLOAD) == 0; >> > + int use_autoload = (fetch_type & >> ZEND_FETCH_CLASS_NO_AUTOLOAD) ? 0 : 1; >> > + int do_normalize = (fetch_type & >> ZEND_FETCH_CLASS_NO_NORMALIZE) ? 0 : 1; >> > + zstr lcname = class_name; >> > + >> > + fetch_type = fetch_type & ~ZEND_FETCH_CLASS_FLAGS; >> > >> > - fetch_type = fetch_type & ~ZEND_FETCH_CLASS_NO_AUTOLOAD; >> > check_fetch_type: >> > switch (fetch_type) { >> > case ZEND_FETCH_CLASS_SELF: >> > @@ -1564,15 +1584,22 @@ check_fetch_type: >> > } >> > return EG(scope)->parent; >> > case ZEND_FETCH_CLASS_AUTO: { >> > - fetch_type = >> > zend_get_class_fetch_type(type, class_name, class_name_len); >> > + if (do_normalize) { >> > + lcname = >> > zend_u_str_case_fold(type, class_name, class_name_len, 1, >> > &class_name_len); >> > + } >> > + fetch_type = >> > zend_get_class_fetch_type(type, lcname, class_name_len); >> > if >> > (fetch_type!=ZEND_FETCH_CLASS_DEFAULT) { >> > + if (do_normalize) { >> > + efree(lcname.v); >> > + do_normalize = 0; /* >> > we've normalized it already, don't do it twice */ >> > + } >> > goto check_fetch_type; >> > } >> > } >> > break; >> > } >> > >> > - if (zend_u_lookup_class_ex(type, class_name, class_name_len, >> > use_autoload, &pce TSRMLS_CC)==FAILURE) { >> > + if (zend_u_lookup_class_ex(type, lcname, class_name_len, >> > use_autoload, do_normalize, &pce TSRMLS_CC)==FAILURE) { >> > if (use_autoload) { >> > if (fetch_type == >> ZEND_FETCH_CLASS_INTERFACE) { >> > zend_error(E_ERROR, "Interface '%R' >> > not found", type, class_name); @@ -1580,8 +1607,14 @@ >> check_fetch_type: >> > zend_error(E_ERROR, "Class >> '%R' not found", type, class_name); >> > } >> > } >> > + if (lcname.v != class_name.v) { >> > + efree(lcname.v); >> > + } >> > return NULL; >> > } else { >> > + if (lcname.v != class_name.v) { >> > + efree(lcname.v); >> > + } >> > return *pce; >> > } >> > } >> > Index: Zend/zend_language_scanner.l >> > =================================================================== >> > RCS file: /repository/ZendEngine2/zend_language_scanner.l,v >> > retrieving revision 1.155 >> > diff -p -u -r1.155 zend_language_scanner.l >> > --- Zend/zend_language_scanner.l 12 Jun 2006 >> 17:06:39 -0000 1.155 >> > +++ Zend/zend_language_scanner.l 17 Jul 2006 19:47:30 -0000 >> > @@ -444,7 +444,7 @@ static inline int zend_check_and_normali >> > efree(Z_USTRVAL_P(zendlval)); >> > return 0; >> > } >> > - if (!zend_normalize_identifier(&norm, &norm_len, >> > Z_USTRVAL_P(zendlval), Z_USTRLEN_P(zendlval), 0)) { >> > + if (zend_normalize_identifier(&norm, &norm_len, >> > Z_USTRVAL_P(zendlval), Z_USTRLEN_P(zendlval), 0) == FAILURE) { >> > zend_error(E_COMPILE_WARNING, "Could not normalize >> > identifier: %r", Z_USTRVAL_P(zendlval)); >> > efree(Z_USTRVAL_P(zendlval)); >> > return 0; >> > Index: Zend/zend_operators.c >> > =================================================================== >> > RCS file: /repository/ZendEngine2/zend_operators.c,v >> > retrieving revision 1.248 >> > diff -p -u -r1.248 zend_operators.c >> > --- Zend/zend_operators.c 12 Jul 2006 19:34:55 -0000 >> 1.248 >> > +++ Zend/zend_operators.c 17 Jul 2006 19:47:30 -0000 >> > @@ -2109,14 +2109,21 @@ ZEND_API zstr zend_u_str_case_fold(zend_ >> > zstr ret; >> > >> > if (type == IS_UNICODE) { >> > - int ret_len; >> > + int ret_len = length; >> > >> > if (normalize) { >> > - zend_normalize_identifier(&ret.u, >> &ret_len, source.u, length, 1); >> > + if >> (zend_normalize_identifier(&ret.u, &ret_len, source.u, >> length, 1) == FAILURE) { >> > + zend_error(E_NOTICE, "Could >> not normalize identifier"); >> > + ret.u = eustrndup(source.u, length); >> > + } >> > } else { >> > UErrorCode status = U_ZERO_ERROR; >> > >> > zend_case_fold_string(&ret.u, &ret_len, >> > source.u, length, U_FOLD_CASE_DEFAULT, &status); >> > + if (U_FAILURE(status)) { >> > + zend_error(E_NOTICE, "Could >> not case-fold string"); >> > + ret.u = eustrndup(source.u, length); >> > + } >> > } >> > >> > *new_len = ret_len; >> > Index: Zend/zend_unicode.c >> > =================================================================== >> > RCS file: /repository/ZendEngine2/zend_unicode.c,v >> > retrieving revision 1.25 >> > diff -p -u -r1.25 zend_unicode.c >> > --- Zend/zend_unicode.c 12 Jul 2006 17:35:06 -0000 1.25 >> > +++ Zend/zend_unicode.c 17 Jul 2006 19:47:30 -0000 >> > @@ -736,7 +736,7 @@ ZEND_API int zend_normalize_identifier(U >> > if (unorm_quickCheck(ident, ident_len, UNORM_NFKC, >> &status) != UNORM_YES) { >> > zend_normalize_string(&buffer, &buffer_len, >> ident, ident_len, &status); >> > if (U_FAILURE(status)) { >> > - return 0; >> > + return FAILURE; >> > } >> > ident = buffer; >> > ident_len = buffer_len; @@ -748,7 +748,7 @@ >> ZEND_API >> > int zend_normalize_identifier(U >> > efree(ident); >> > } >> > if (U_FAILURE(status)) { >> > - return 0; >> > + return FAILURE; >> > } >> > ident = buffer; >> > ident_len = buffer_len; @@ -759,7 +759,7 @@ >> ZEND_API >> > int zend_normalize_identifier(U >> > efree(ident); >> > } >> > if (U_FAILURE(status)) { >> > - return 0; >> > + return FAILURE; >> > } >> > ident = buffer; >> > ident_len = buffer_len; @@ -768,7 +768,7 @@ >> > ZEND_API int zend_normalize_identifier(U >> > >> > *dest = ident; >> > *dest_len = ident_len; >> > - return 1; >> > + return SUCCESS; >> > } >> > /* }}} */ >> > >> > Index: ext/reflection/php_reflection.c >> > =================================================================== >> > RCS file: /repository/php-src/ext/reflection/php_reflection.c,v >> > retrieving revision 1.243 >> > diff -p -u -r1.243 php_reflection.c >> > --- ext/reflection/php_reflection.c 13 Jul 2006 >> 12:34:30 -0000 1.243 >> > +++ ext/reflection/php_reflection.c 17 Jul 2006 19:47:31 -0000 >> > @@ -1972,7 +1972,7 @@ ZEND_METHOD(reflection_parameter, getCla >> > GET_REFLECTION_OBJECT_PTR(param); >> > >> > if (param->arg_info->class_name.v) { >> > - if >> > (zend_u_lookup_class_ex(UG(unicode)?IS_UNICODE:IS_STRING, >> > param->arg_info->class_name, >> param->arg_info->class_name_len, 1, &pce >> > param->TSRMLS_CC) == FAILURE) { >> > + if >> > (zend_u_lookup_class_ex(UG(unicode)?IS_UNICODE:IS_STRING, >> > param->arg_info->class_name, param->arg_info->class_name_len, 1, 1, >> > param->&pce TSRMLS_CC) == FAILURE) { >> > >> zend_throw_exception_ex(reflection_exception_ptr, 0 TSRMLS_CC, >> > "Class %v does not exist", >> param->arg_info->class_name); >> > return;
Best regards, Marcus

Andrei Zmievski

20 years ago
We are actually normalizing, but only identifiers. -Andrei On Jul 17, 2006, at 4:08 PM, Andi Gutmans wrote:

Zeev Suraski

20 years ago
Aren't we both normalizing and lowercasing? Zeev At 02:45 18/07/2006, Andrei Zmievski wrote:

Andrei Zmievski

20 years ago
Yes, we are. But normalization (in our case) may involve case-folding (for class/function names), so it's really a single operation. If you are asking whether we should have 2 flags for it, I don't believe so. -Andrei On Jul 17, 2006, at 4:48 PM, Zeev Suraski wrote:

Andi Gutmans

20 years ago
Is there a good reason for normalizing? I thought we are fine with expecting developers to use standard form and not suffer performance penalties. I'm not a Unicode expert so my question might be somewhat ignorant..

Andrei Zmievski

20 years ago
We did talk about this before, on the php-unicode list if you remember. The UTR#31 has a section about normalization: http://www.unicode.org/reports/tr31/ I think we should see what kind of performance penalties identifier normalization brings with it, if any. Once we have the numbers, it's very easy to replace full normalization with simple case-folding. -Andrei On Jul 17, 2006, at 5:12 PM, Andi Gutmans wrote:

Andi Gutmans

20 years ago
OK sounds fine.