Everything

php

strlen() under unicode.semantics

php.internals

Daniel Convissor

20 years ago

Hi:

Enjoyed Andrei's talk at the NYPHP Conference last week about unicode in
PHP 6.  He mentioned that when unicode.semantics is on, strlen() will
return the number of characters rather than the number of bytes, like
mb_string() does or strlen() if mbstring.func_overload is on.

The hitch here is there are situations where one needs to know how many
bytes are in a string.  Is there a function I've overlooked that does
this or will do this, please?

Thanks,

--Dan

--
T H E   A N A L Y S I S   A N D   S O L U T I O N S   C O M P A N Y
data intensive web and database programming
http://www.AnalysisAndSolutions.com/
4015 7th Ave #4, Brooklyn NY 11232  v: 718-854-0335 f: 718-854-0409

Sara Golemon

20 years ago

> Enjoyed Andrei's talk at the NYPHP Conference last week about unicode in
> PHP 6.  He mentioned that when unicode.semantics is on, strlen() will
> return the number of characters rather than the number of bytes, like
> mb_string() does or strlen() if mbstring.func_overload is on.
>
> The hitch here is there are situations where one needs to know how many
> bytes are in a string.  Is there a function I've overlooked that does
> this or will do this, please?
>

My first question is: Why do you need to know the number of bytes occupied
by a textual string?  Is it because you want to work with binary strings?
Because that's still very possible:

Even with unicode.semantics=on, the binary string type may be explicitly
used in a few ways:

$a = b"This string contains an 0xF0 byte: \xF0";
$alen = strlen($a);

This being the simplest, the lowercase b (or u) characters denote a string
as being a binary (or unicode) string explicitly.  Leaving these specifiers
off yield whatever type is appropriate to unicode.semantics.

In other cases, such as reading from a binary mode file:

$fp = fopen('foo.bin', 'rb');
$str = fread($fp, 100);

The string returned is always returned as a binary string regardless of
unicode semantics.  When reading a text-mode file conversely:

$fp = fopen('foo.txt', 'rt');
$str = fread($fp, 100);

The type of string returned will depend on the unicode.semantics switch (in
order to ensure maximum BC, since scripts designed for windows already use
text mode to handle linebreak transformation).

-Sara

Jared Williams

20 years ago

>
> > Enjoyed Andrei's talk at the NYPHP Conference last week
> about unicode in
> > PHP 6.  He mentioned that when unicode.semantics is on,
> strlen() will
> > return the number of characters rather than the number of
> bytes, like
> > mb_string() does or strlen() if mbstring.func_overload is on.
> >
> > The hitch here is there are situations where one needs to
> know how many
> > bytes are in a string.  Is there a function I've overlooked
> that does
> > this or will do this, please?
> >
> My first question is: Why do you need to know the number of
> bytes occupied
> by a textual string?  Is it because you want to work with
> binary strings?
> Because that's still very possible:
>
> Even with unicode.semantics=on, the binary string type may be
> explicitly
> used in a few ways:
>
> $a = b"This string contains an 0xF0 byte: \xF0";
> $alen = strlen($a);
>
> This being the simplest, the lowercase b (or u) characters
> denote a string
> as being a binary (or unicode) string explicitly.  Leaving
> these specifiers
> off yield whatever type is appropriate to unicode.semantics.
>
> In other cases, such as reading from a binary mode file:
>
> $fp = fopen('foo.bin', 'rb');
> $str = fread($fp, 100);

Hi,

What happens with

$fp = fopen('foo.bin', 'wb');
$written = fwrite($fp, $str);
if (strlen($str) != $written)
{
echo 'Not written', "\n";
}

Jared

Sara Golemon

20 years ago

> What happens with
>
> $fp = fopen('foo.bin', 'wb');
> $written = fwrite($fp, $str);
> if (strlen($str) != $written)
> {
>   echo 'Not written', "\n";
> }
>

Assuming $str is a binary string.  The above code works just fine.

If it's a unicode string:

Short version: Don't do that.

Writing a unicode string to a binary mode file yields the U16 character
stream that the string is internally made up of, and yes, in this case
strlen($str) will not equal $written, but it's important to note that doing
this *IS* wrong.  What you should be doing, if you want the U16 that the
string is encoded as, is to open the file for writing explicitly set the
encoding.  This can be done by either:

$fp = fopen('foo.txt', 'w');
stream_encoding($fp, 'utf16');

Or any of the context/filter based methods given at:
http://blog.libssh2.org/index.php?/archives/6-PHP6-Streams-Update.html
(This'll all make it into the manual eventually, just saving it for now)

If you're going to (for some innane reason) switch between encodings as you
write to a file (there's never a good reason for this), you can use the
filter approach and just swap unicode.to.* filters in and out as needed.

-Sara

Ron Korving

20 years ago

Still, it's gotta be useful to be know how many bytes it occupies. Perhaps
for Content-length headers or something. There are plenty of low level
concepts to think of where one might need this. And even if you can't think
of any reason now, you don't wanna get hit in the face by it and have to
implement such a function for PHP 6.0.1.

- Ron


""Sara Golemon"" <pollita@php.net> wrote in message
news:001d01c695a8$a9faad20$88051fac@OHRLVN4523SG...

> > What happens with
> >
> > $fp = fopen('foo.bin', 'wb');
> > $written = fwrite($fp, $str);
> > if (strlen($str) != $written)
> > {
> >   echo 'Not written', "\n";
> > }
> >
> Assuming $str is a binary string.  The above code works just fine.
>
> If it's a unicode string:
>
> Short version: Don't do that.
>
> Writing a unicode string to a binary mode file yields the U16 character
> stream that the string is internally made up of, and yes, in this case
> strlen($str) will not equal $written, but it's important to note that

doing

> this *IS* wrong.  What you should be doing, if you want the U16 that the
> string is encoded as, is to open the file for writing explicitly set the
> encoding.  This can be done by either:
>
> $fp = fopen('foo.txt', 'w');
> stream_encoding($fp, 'utf16');
>
> Or any of the context/filter based methods given at:
> http://blog.libssh2.org/index.php?/archives/6-PHP6-Streams-Update.html
> (This'll all make it into the manual eventually, just saving it for now)
>
> If you're going to (for some innane reason) switch between encodings as

you

Andrei Zmievski

20 years ago

It'll be there. strlen_bytes() perhaps?

-Andrei

On Jun 22, 2006, at 2:55 PM, Ron Korving wrote:

Andi Gutmans

20 years ago

Maybe sizeof() should not be an alias for strlen() when operating on
Unicode...?

Andi

Johannes Schlueter

20 years ago

Hi Andi,

sizeof() is an alias of count()  not strlen() ;-)

johannes

On Friday 23 June 2006 07:43, Andi Gutmans wrote:

Andi Gutmans

20 years ago

Oops, senile me :)

How about str_size()?

Andi

Andrei Zmievski

20 years ago

How about str_storage_size()? It is explicit enough that people will
be wary of using it.

-Andrei

On Jun 22, 2006, at 10:56 PM, Andi Gutmans wrote:

Sara Golemon

20 years ago

> Still, it's gotta be useful to be know how many bytes it occupies. Perhaps
> for Content-length headers or something. There are plenty of low level
> concepts to think of where one might need this. And even if you can't
> think
> of any reason now, you don't wanna get hit in the face by it and have to
> implement such a function for PHP 6.0.1.
>

For this type of usage, I'd think it'd be relevant to know how many bytes
the string will occupy in a given output encoding moreso that what it
happens to occupy in the underlying implementation.  In the example you
cited, string contents will more typically be sent as utf8 rather than the
utf16 of php's internal encoding.

$utf8str = unicode_encode($unistr, 'utf8');

header('Content-type: text/html; encoding="utf8"');
header('Content-length: ' . strlen($utf8str));
echo $utf8str;

I'm not saying it's impossible that a legitimate use will come up to know
the internal byte-usage of a unicode string, there's certainly no harm in
adding such a function (apart from the tired shot-foot argument).  I just
doubt you (or anyone) will come up such a case anytime soon.

-Sara

Andi Gutmans

20 years ago

I don't quite agree. I think there's a good chance people will want to save
Unicode strings in a binary format for performance reasons. Save it the way
it looks in memory, and put it back... Why convert to UTF-8 or any other
encoding if it's just about storage?

Andi

Andrei Zmievski

20 years ago

The only way they can get at the internal UTF-16 representation is
via unicode_encode($uni, 'UTF-16') which will return a binary UTF-16
string. In that case, strlen() will work just as well.

-Andrei


On Jun 22, 2006, at 11:30 PM, Andi Gutmans wrote:

Andi Gutmans

20 years ago

Hmm, I was thinking we might have some binary write function which would do
that automagically.  I think it'd be worth it.

Andrei Zmievski

20 years ago

Really? I think it's very rare that someone'd want to get at the
internals of a Unicode string.

-Andrei

On Jun 22, 2006, at 11:44 PM, Andi Gutmans wrote:

Sara Golemon

20 years ago

>> The only way they can get at the internal UTF-16
>> representation is via unicode_encode($uni, 'UTF-16') which
>> will return a binary UTF-16 string. In that case, strlen()
>> will work just as well.
>>
> Hmm, I was thinking we might have some binary write function which would
> do
> that automagically.  I think it'd be worth it.
>

Wasn't this the discussion we had regarding {} versus [] for string offset
access? (The former referring to characters, the later referring to U16
units.  I also recall we dropped that as the wtf potential outweighed the
potential performance gain.

-Sara

Ron Korving

20 years ago

Maybe it'd be useful if there was a function to "cast" a UTF string into a
binary string without changing anything on the inside. That way one could do
strlen(str_to_binary($string)). That would also be useful for binary storing
and reading (with binary_to_str).

Ron


"Andrei Zmievski" <andrei@gravitonic.com> schreef in bericht
news:A4FC2E25-5D10-4EFF-A4BE-3F0206354BAD@gravitonic.com...

Andrei Zmievski

20 years ago

There already is such a function:

strlen(unicode_encode($string, "UTF-16BE"));

I think wanting to have access to internal representation of Unicode
strings is an extremely rare operation in any case.

-Andrei

On Jun 23, 2006, at 12:16 AM, Ron Korving wrote: