[RFC] Unicode Text Processing

Hey Derick, Hey all. On 15.12.22 16:34, Derick Rethans wrote:

Thanks for tackling this immense topic. I see a few challenges in the approach. My first question was: Why do we need a new implementation of the ICU library? Creating a userland implementation that wraps the currently existing mb-string and ICU functions into a class that allows better usability shouldn't add that much of a performance penalty. And including the mb-string and the intl extension by default wouldn't hurt. That way there would be no added maintenance burden on the core developers. In addition to that it looked to me that there are multiple things mixed up in this Text-class. If we want a Text-class to handle Unicode strings in a better way, why does the string itself need to be Locale-aware? The string itself is a collection of Unicode-Codepoints referencing Characters and Graphemes. Does the string itself need to be aware of a locale to aid in sorting? It needs to be aware of the internal normalization form for character-comparison for sure. But I would rather see a Normalizer handle normalization of the Text-content instead of the Text-class handling that itself. Similarily I'd see the Transliteration done by a separate class. Which then strongly looks similar to the Intl-extension. Which brings me back to the question: Do we really need a second Intl-extension in the core? I'm ambivalent about this. On the one hand it could make some things for sure easier. On the other hand it adds burden onto the core-developers that could be avoided by providing the intl (and mb-string) extension by default instead of having to add them separately. And then find a group if people willing to build a userland implementation. And yes, I know the intl-extension is everything but easy to use. Especially in the quirky edge-cases regarding Transliteration and Normalization. But the issue usually isn't using it but finding the appropriate documentation on the ICU page. Helping the ICU to improve on that documentation would also be a huge benefit. To all those trying to use the Intl-extension right now. But that's just my 0.02€ Cheers Andreas

On Thu, Dec 15, 2022 at 12:34 PM Derick Rethans <derick@php.net> wrote:

I appreciate the work behind this RFC. Although I can't comment much on the utility itself as I never really used `ext-intl` or much of UTF-8/16 stuff, I think PHP lacks A LOT of built-in classes to make matters simpler and welcoming. Sure, anybody in userland could write a class, but that leads to lots of implementations and an overwhelming amount of choices to be made. If this class can cover 80% of the use-case, folks can extend it to build their remaining 20% advanced use-case and PHP becomes easier. I really look forward to more basic utility-classes built-in and ready to go.

Hi On 12/15/22 16:34, Derick Rethans wrote:

Some first remarks: ---------- replaceText(): In the replaceText() section the description refers to a non-existent parameters '$maxReplacements' and '$collator'. Also the second paragraph in that section looks like a word is missing there. ---------- getPositionOfFirstOccurrence(): I agree this is too long. How about: - findOffset() - findOffsetLast() And for returnFromFirstOccurence(): - startingWith() - startingWithLast() ---------- firstToTitle(): I don't see how this differs from toTitle() and/or firstToUpper(). An example would likely be helpful. ---------- wordsToTitle(): Likewise. ---------- The return of many methods is not explicitly listed. I'd put it everywhere to make it 100% clear that all those functions return 'Text'. Best regards Tim Düsterhus

On 15.12.2022 at 16:34, Derick Rethans wrote:

| As the implementation requires ICU, this would also mean that PHP will | depend on the ICU library. Our current stance is that a minimal PHP should be buildable without requiring any "non-standard" libraries; this is the reason why we bundle PCRE. If we wanted to stick with that policy, we would need to bundle ICU, what might not be the best idea – it's generally not great to have bundled libraries which are still maintained outside of php-src, and especially for such huge libraries.

On 12/15/2022 7:34 AM, Derick Rethans wrote:

A few quick thoughts:

By this do you mean Normalization Form C (NFC)? "Unicode Canonical Form" isn't a phrase I'm familiar with. Assuming so, are modified texts (e.g. via join, replaceText, reverse) re-normalized? ---

This is also known as ZWNBSP (Zero Width No-Break Space). Will only a leading instance be stripped? If so, how can someone search for it (or a substring beginning with it) given that:

and all the search methods take `string|Text $search`. --- Why is this being introduced directly into PHP core rather than first an extension where it's easier to shake out the interface and behavior?

On Thu, Dec 15, 2022 at 9:34 AM Derick Rethans <derick@php.net> wrote:

Obviously, hurdle one is making the ICU library a requirement for building PHP. I'd almost make that it's own milestone in this project with the introduction of the Text class as a separate followon. A very casual (IANAL) read of the ICU license doesn't seem to make this a problem, so it may be more of a question of whether we put this on people wanting to build PHP. ICU is pretty widely available and used, so I also don't see this as a major stumbling block. Question 2 is that class. I know folks have been clammoring for a `String` class for some time and this actually fills that niche quite well. A part of me wonders if we can overload it a little to provide a psuedo locale of "binary" so that users can, optionally, treat it like a more generalized String class in specific cases, storing a normal `char*` zend_string under the hood in that case. Possibly as a specialzation tree. /* names as examples only */ interface Stringy { /* define all those APIs */ } class Text implements Stringy { /* ... */ } class BinaryString implements Stringy { /* ... */ } I think you'd get a lot more buy-in from the folks who worry that UTF16 is overhead they don't want, but who do like the idea of an OOPy string. It also provides a migration path to avoid having to rethink byte vs grapheme conversions up front, instead deferring that part of a migration till later. Overall, I'm more positive on this than negative, and I eagerly await the rest of this thread. -Sara

On 15/12/2022 15:34, Derick Rethans wrote:

As others have said already, thank you for taking a stab at this important topic. I agree that it would be a really useful feature for the language, but it's also a really difficult one to get right. Here are my initial thoughts... # Design Process Rather than designing the whole class "on paper", I think this really needs to be built as a prototype, where we can build up documentation and tests, plug variations into some real life scenarios, and have separate discussions about different details. If we limit ourselves initially to features already exposed by ext/intl (I think everything proposed so far is?), a prototype doesn't even need to be an extension, it can be in pure PHP. Then once the design is finalised, you have a ready-made polyfill for older PHP versions, and a set of tests for the native version :) We might also want to do some general investigation of what other languages and frameworks provide, and which decisions have proven good or bad in practice. # Lossy Transforms Automatic normalisation and stripping of BOMs seems useful, but it immediately rules out use of this class for anything where you want to get back what you put in. For instance, if an ORM used Text instances for strings in data models, it would generate extra Update queries on the database even when the string wasn't otherwise changed. I think it would be better to make this easy but explicit. # UTF-8 on the outside, UTF-16 on the inside I know this will be a very common combination, but it feels odd that an application which actually wanted to work with UTF-16 would need to perform round-trips through UTF-8 just to use this class. It should at least be possible to specify the encoding on input and output. Ruby takes an interesting approach where strings are tagged with their current binary encoding, and only converted to another form if actually required. If your input layer says "$name = new Text($_GET['name'], 'Windows-1252');" and your output layer says "echo $name->asBytes('Windows-1252');" the overhead of converting to UTF-16 can be skipped entirely, unless something in between says "$name = $name->wordsToUpper()". This also removes another source of lossy transformation, since some encoding conversions aren't perfectly reversible (e.g. the source encoding has more than one byte sequence mapped to the same Unicode code point). # Internationalisation Having locale and collation as state on the object, rather than parameters on relevant methods, feels like muddling responsibilities. It makes it hard to reason about what exactly some of the methods will do: Can I trust that this object will give me a sensible result from compareWith, or has it been assigned a collation somewhere else? What exactly will be the definition of "replace" or "contains" for this pair of objects? How users will work with these also needs careful thought - your first listed design goal is "keep it simple", but under locales and Internationalisation is the worrying sentence "This will require extensive documentation". This is one of those places where "doing it right" is really hard to combine with "making it easy", because language is inherently complex, but users will expect a simple answer to "how do I make it case-insensitive?" # Allowing other abstractions I 100% approve of your use of grapheme clusters, rather than code points, as the primary unit; so many implementations get that wrong. However, when interacting with other systems, reasoning about bytes (or sometimes even codepoints) is essential. One function that I would really like to see, for instance, is a grapheme-aware version of mb_strcut, to solve tasks like: "encode this abstract Unicode string as UTF-16BE, truncated to at most 200 bytes, without breaking apart any grapheme clusters". Thanks again for getting the ball rolling, and I look forward to helping iterate the design. Regards,

On 16/12/22 02:34, Derick Rethans wrote:

Using "collator" and "locale" interchangeably seems imprecise. If the input is an ICU locale string, then I think you should just call it locale. Then the user will be armed with the correct terminology when they go looking for more information in the ICU manual. In ICU, case conversion and BreakIterator need a locale, not a collator. I'm concerned about the time order of using grapheme offsets. For example, is subString() O(N) in $offset? If the idea is to be easy to use and performant, you don't want to have subtle algorithmic complexity traps. I'm probably not the target audience for this class, since I'm generally looking for maximum flexibility, not minimum complexity. As such, I'd like intl to have better documentation and more features. The RFC has a family of locale-aware case conversion functions which do not exist in intl. This was raised as an issue during the discussion on my ASCII case conversion RFC. It would be great if intl could get those functions too. I think you should consider making this Text class a part of the intl extension. You're adding a class which is similar to the classes in that extension. In terms of data, it's like IntlChar, except it's for strings not characters. Its constructor takes an ICU locale string, just like IntlBreakIterator or MessageFormatter. I can understand if you don't want to follow all the existing conventions of the intl extension. But if that is the rationale for the RFC, I'd like to see a discussion of the specific usability problems with the intl extension. -- Tim Starling

On 2022-12-15 8:34 AM, Derick Rethans wrote:

I love the idea, but we may want to think about using the terms `trimStart` and `trimEnd` instead of `trimLeft` and `trimRight`. Obviously, when the text in the parameter is in a right-to-left language (e.g., Hebrew, Arabic), `trimLeft` and `trimRight` are going to do the opposite of what one may expect from the names of the functions. I know that identifiers like `ltrim` and `rtrim` are longstanding conventions which everyone understands that they **really** mean trimming from the start and end respectively, but this may be an opportunity to amend the inaccuracy. If this ever gets expanded to add padding functions, those should probably be named `padStart` and `padEnd` as well.

On Thu, Dec 15, 2022, at 9:34 AM, Derick Rethans wrote:

Derick, thank you for tackling this. It's a decidedly not-simple problem space and I'm glad someone like you is looking into it. I'm overall in favor of the RFC, though I have some comments/pushback. First, here's my notes just as I'm reading through it: Re Text::create, which the RFC suggests can be aliased as a function for easier use: Are you sure? Symfony has stand-alone wrapper functions. You cannot alias a static method directly to a function. cf: https://3v4l.org/V4kP2 (It would be really nice if that worked! But it doesn't seem to.) Text::concat: What happens if different Text objects passed to that have different collations? Which gets used, the first, last, "best fit"...? Text->wrap: " If $cutLongWords is set, no Text element will be larger than $maxWidth." Can you include an example here? I have to mentally noodle through what this means, which means it could use an example. Text->getPositionOfFirstOccurrence() et al: These should not return false on not-found. That is an anti-pattern. That PHP's existing libraries do that is a bug, not a feature. It has caused no end of bugs. null is the correct thing to return here, especially with the new null-handling syntax we have now in PHP 8. Do not use "false" as a not-found return, ever. Another option to consider is if some of them should return an empty Text object. (That may not be the best answer, but it's one worth considering.) Also, all of those names are very long. :-( Text->returnFromFirstOccurence(): I much prefer startingWith(). It's half the length and just as if not more descriptive. It also implies, to me, that the $search will be included in the result, whereas startingAt(), for whatever reason, doesn't seem like it does. Text->contains(): The header is missing the defined return type. Comparing Text Objects: Oh, for being able to overload those operators. This would be a great use case for it. :-( The examples in case-conversion are hard to follow, because the font of code samples is not that different from normal text. Could you perhaps multi-line them, to make it clearer where the "in" text ends and the "out" text starts? How do toTitle() and wordsToUpper() differ? They sound like the same thing... (Please note the difference in the RFC.) Why two methods for length? And why confuse it with "character" when the text has been very consistent about using grapheme to this point. getCodePointCount(): I... don't understand how this is different from length, so I don't see why we'd use it. If it's kept, please include a better explanation of what it is or why I'd care. getWordCount(): The example uses getWordIterator as a property, when I think it's supposed to be a method. Also, it's not syntax highlighted. "The return of the iterators are effected by the text's locale." - affected, not effected. getCharacterIterator(): Again, dropping in the word character here. Calling it getGraphemeIterator() would be terrible, of course. :-) This feels like an older part of the text that wasn't updated when most of it started using grapheme. Perhaps skip the explanation here and move a central definition of "character" to the start of the RFC? (Which could be "means the same as grapheme in this case, NOT the same as byte.") getWordIterator(): It's not clear to me if this includes whitespace as its own Text objects. Would the string "Mr. Smith Goes to Washington" be a word iterator of ["Mr.", "Smith", "Goes", "to", "Washington"]? Or ["Mr", ".", "Smith", "Goes", "to", "Washington"]? Or ["Mr", ".", " ", "Smith", " ", "Goes", " ", "to", " ", "Washington"]? I'm not clear which is the intent here. (Feel free to steal this example for the RFC.) getLineIterator(): I do not understand this description at all. From the name, I'd expect it to break the string at newline characters. The description seems like it's something completely different I do not understand. getTitleIterator(): What's a title, in this context? Transliteration section: The formatting here seems wonky and confusing. Please clean up. ----- Second, there is a PHP-FIG Working Group on translation. It's mostly idle at the moment, as we're waiting on the MessageFormat working group at the W3C to stabilize their next version so we can just steal it. I don't know that there's any direct overlap between this RFC and that WG, but I'm mentioning it for transparency, and to encourage people to think about how they could both be developed to play nicely together, whatever that means. Third, is there some way to say "this string, but in some other collation?" It looks like the only way to do that is via Text::create($txt, 'new-collation') / new Text($txt, 'new-collation'). A ->withCollation('new-collation') method would be very helpful, especially as so many methods rely on the collation for things like case insensitivity. That way, we could do $txt->withCollation('case-insensitive-english')->split(',') (or similar). Fourth, that brings me to my biggest concern. "The format of this locale/collation name needs extensive documentation." - This line scares the ever-loving crap out of me. :-) We know from experience that complex formatting strings are trivially screwed up, mistyped, or otherwise gotten wrong. Especially when they're not self-evident. ("ks" means "case-insensitive"? I would never have guessed that in a million years.) The links provided to the Unicode sites don't really illuminate anything for me. This to me sounds like it cries out for either a builder object, an enumeration (or multiple), or some combination of those. It sounds like the TextCoallator class is maybe trying to be that, but it's still under-described, especially for anyone who hasn't already used intl. It also looks like it's just producing a string, rather than, what I'd consider probably more ergonomic, using the object directly like DateTimeZone. I'm not sure what the best answer here is since I don't know the problem space well enough, but I do know that "here's a string, GL" is insufficient. We should noodle on how to make that more ergonomic so that casual developers don't get it horribly subtly wrong. (Because you know they/we/I will on this topic.) That may also include extra utility methods like ->withCaseInsensitiveCollation() (which changes only the case sensitivity marker but leaves the rest of the collation alone), or something like that. Again, I'm not sure what the best design here is. Everything i just said also applies to the "$transliterationString", which is mentioned in passing but no description is provided. I have no clue what the syntax for that even is. And of course it would be off brand for me to not note that issues with a fixed set of methods on an object like this, rather than pipe-friendly functions that are innately more extensible. I know, I know, we don't have pipes yet, but I have to mention it anyway or people would be worried about me. :-) --Larry Garfield

2022年12月16日(金) 0:34 Derick Rethans <derick@php.net>:

Hi, Derick and Internals Is still available this topic? I have interesting this Text class. I'm glad to control based on grapheme cluster such as Swift's string type. I have some idea. 1. Move to Intl extension such as \Intl\Text * I think keep it simple for implementation. 2. Add Text type for grapheme_* function only such as string|Text. * It is some complexy for implementation but userland is simple 3. If UTF-8 validaion failed, throws an exception __toString method returns string type is seems good. Please consider this. Regards Yuya