Everything

php

Home » Project » Newsgroups » php.internals » Sanitize filters »

Sanitize filters

php.internals

Kamil Tekiela

3 years ago

Hi Internals,

For quite some time now, PHP's sanitize filters have "Rustled My Jimmies".
These filters bother me because I can't really justify their existence. I
can understand that a few of them are sensible and may come in handy, but I
would like to talk about some of these in particular.

In PHP 8.1, we have deprecated FILTER_SANITIZE_STRING which I deemed to be
a priority due to its confusing name and behaviour. The rest is slightly
less dangerous, but as was pointed out to me in a recent conversation with
a PHP developer, these filters are all very confusing.

I would like to have some opinions on the following filters. What do you
think we should do with them? Deprecate? Fix? Provide better documentation?

---

*FILTER_SANITIZE_ENCODED *- "URL-encode string, optionally strip or encode
special characters."
Now, what does that mean? PHP has two functions for URL encoding: urlencode
used for encoding query-string parts, and rawurlencode used for encoding
any other URL part (two different RFCs are followed by these functions).
Which of these RFCs is applied in this filter? Furthermore, the description
says that "special characters" can be stripped or encoded. Is one of these
actions the default and the other can be selected by a flag or are both
optional? What are these special characters? Are they special in the
context of URL? If so, why did we encode them first? If these are HTML
special characters (there's no single definition of special HTML chars),
then why does this filter encode them if the filter is for URL
sanitization? What does backtick have to do with any of this
(FILTER_FLAG_STRIP_BACKTICK)?

*FILTER_SANITIZE_ADD_SLASHES - "*Apply addslashes(). (Available as of PHP
7.3.0)"
This filter was added as a replacement for magic_quotes filter. According
to PHP documentation, addslashes is supposed to be used when injecting PHP
variables into eval'd string. Real-life showed that this function is used
in a lot of places that have nothing to do with PHP's eval. I am not sure
if the sanitize filter is misused in a similar fashion, but judging from
the fact that it was meant as a replacement for magic_quotes, my guess is
that it's very likely still abused.

*FILTER_SANITIZE_EMAIL *- "Remove all characters except letters, digits and
!#$%&'*+-=?^_`{|}~@.[]."
Which RFC does this adhere to? It strips slashes and quoted parts, doesn't
allow IPv6 addresses and doesn't accept RFC 6530 email addresses. This
filter is ok for simple usage, but it isn't true to any known specification
AFAIK.

*FILTER_SANITIZE_SPECIAL_CHARS *- "HTML-encode '"<>& and characters with
ASCII value less than 32, optionally strip or encode other special
characters."
What's the intended purpose of this filter? "Special characters" are still
not clearly defined, but at least it's more clear than
the FILTER_SANITIZE_ENCODED description. Same question about backticks
though: why? Why encode ASCII <32 chars?

*FILTER_SANITIZE_FULL_SPECIAL_CHARS *- "Equivalent to calling
htmlspecialchars() with ENT_QUOTES set. Encoding quotes can be disabled by
setting FILTER_FLAG_NO_ENCODE_QUOTES. Like htmlspecialchars(), this filter
is aware of the default_charset and if a sequence of bytes is detected that
makes up an invalid character in the current character set then the entire
string is rejected resulting in a 0-length string. When using this filter
as a default filter, see the warning below about setting the default flags
to 0."
Not to be mistaken with FILTER_SANITIZE_SPECIAL_CHARS. As long as it's not
used with filter_input(), it's the least problematic. We
have htmlspecialchars() though, so how useful is this filter?

*FILTER_UNSAFE_RAW *- What makes it unsafe? Why isn't this just
called FILTER_RAW_STRING? If the value being filtered is something other
than a string, what will this filter return? Integers, floats, booleans and
nulls are converted to a string, Arrays and objects make the filter fail.

---

Let's quickly mention the filter flags.

The FILTER_FLAG_STRIP_LOW flag will also remove tabs, carriage returns and
newlines as these are all less than 32 ASCII codes. When is this useful and
expected?

The FILTER_FLAG_ENCODE_LOW flag "encodes" ASCII <32 codes presumably into
HTML entities, although that's not specified anywhere in the PHP manual.
The word HTML does not appear on the
https://www.php.net/manual/en/filter.filters.flags.php page. What do these
characters look like when presented by HTML? When is it ever useful to use
this flag?

FILTER_FLAG_ENCODE_AMP & FILTER_FLAG_STRIP_BACKTICK - why is this even a
thing?

Due to flags, FILTER_VALIDATE_EMAIL will happily validate email addresses
that would be otherwise mangled by FILTER_SANITIZE_EMAIL.

These are just the things I found confusing and strange about the sanitize
filters. Let's try to put ourselves in the shoes of an average PHP
developer trying to comprehend these filters. It's quite easy to shoot
yourself in the foot if you try to use them. The PHP manual doesn't do a
good job of explaining them, but that's probably because they are not easy
to explain. I can't come up with good examples of when they should be used.

Regards,
Kamil

Lokrain

3 years ago

Hello Kamil,

I believe that PHP should not try to act as a “framework” that provides you
with ready solutions for such cases.

Being able to actually modify the default behaviour of some functions
through the ini .. is even scarier.

For 25 year writing in PHP I never relied on this “magic” for security:)

Regards,
Dimitar

On Sat, 1 Oct 2022 at 18:39, Kamil Tekiela <tekiela246@gmail.com> wrote:

Vasilii Shpilchin

3 years ago

All right if you are writing on PHP for 25 years, you noticed the PHP was
always about high-order web-focused functionality out-of-box. This is one
of basic benefits of PHP to other general-purpose languages where you can
write everything you want and you also have to write it since the language
itself is very basic. I'm for PHP to keep built-in solutions for most
common problems in the context of the web. Having passe ZCE exam and
writing just 15 years on php.

On Sun, Oct 2, 2022, 2:19 AM Lokrain <lokrain@gmail.com> wrote:

Lokrain

3 years ago

Hello Vasilii,

It’s okay to have different opinion I hope.

You are missing an important point here - beside my comments, the current
way this is developed brings confusion.

It would be great if you share your experience on this matter.

Regards,
Dimitar

On Sun, 2 Oct 2022 at 9:31, Vasilii Shpilchin <vasilii.b.shpilchin@gmail.com>
wrote:

Larry Garfield

3 years ago

On Sat, Oct 1, 2022, at 10:39 AM, Kamil Tekiela wrote:

> Hi Internals,
>
> For quite some time now, PHP's sanitize filters have "Rustled My Jimmies".
> These filters bother me because I can't really justify their existence. I
> can understand that a few of them are sensible and may come in handy, but I
> would like to talk about some of these in particular.
>
> In PHP 8.1, we have deprecated FILTER_SANITIZE_STRING which I deemed to be
> a priority due to its confusing name and behaviour. The rest is slightly
> less dangerous, but as was pointed out to me in a recent conversation with
> a PHP developer, these filters are all very confusing.
>
> I would like to have some opinions on the following filters. What do you
> think we should do with them? Deprecate? Fix? Provide better documentation?
>
> ---
>
> *FILTER_SANITIZE_ENCODED *- "URL-encode string, optionally strip or encode
> special characters."
> Now, what does that mean? PHP has two functions for URL encoding: urlencode
> used for encoding query-string parts, and rawurlencode used for encoding
> any other URL part (two different RFCs are followed by these functions).
> Which of these RFCs is applied in this filter? Furthermore, the description
> says that "special characters" can be stripped or encoded. Is one of these
> actions the default and the other can be selected by a flag or are both
> optional? What are these special characters? Are they special in the
> context of URL? If so, why did we encode them first? If these are HTML
> special characters (there's no single definition of special HTML chars),
> then why does this filter encode them if the filter is for URL
> sanitization? What does backtick have to do with any of this
> (FILTER_FLAG_STRIP_BACKTICK)?
>
> *FILTER_SANITIZE_ADD_SLASHES - "*Apply addslashes(). (Available as of PHP
> 7.3.0)"
> This filter was added as a replacement for magic_quotes filter. According
> to PHP documentation, addslashes is supposed to be used when injecting PHP
> variables into eval'd string. Real-life showed that this function is used
> in a lot of places that have nothing to do with PHP's eval. I am not sure
> if the sanitize filter is misused in a similar fashion, but judging from
> the fact that it was meant as a replacement for magic_quotes, my guess is
> that it's very likely still abused.
>
> *FILTER_SANITIZE_EMAIL *- "Remove all characters except letters, digits and
> !#$%&'*+-=?^_`{|}~@.[]."
> Which RFC does this adhere to? It strips slashes and quoted parts, doesn't
> allow IPv6 addresses and doesn't accept RFC 6530 email addresses. This
> filter is ok for simple usage, but it isn't true to any known specification
> AFAIK.
>
> *FILTER_SANITIZE_SPECIAL_CHARS *- "HTML-encode '"<>& and characters with
> ASCII value less than 32, optionally strip or encode other special
> characters."
> What's the intended purpose of this filter? "Special characters" are still
> not clearly defined, but at least it's more clear than
> the FILTER_SANITIZE_ENCODED description. Same question about backticks
> though: why? Why encode ASCII <32 chars?
>
> *FILTER_SANITIZE_FULL_SPECIAL_CHARS *- "Equivalent to calling
> htmlspecialchars() with ENT_QUOTES set. Encoding quotes can be disabled by
> setting FILTER_FLAG_NO_ENCODE_QUOTES. Like htmlspecialchars(), this filter
> is aware of the default_charset and if a sequence of bytes is detected that
> makes up an invalid character in the current character set then the entire
> string is rejected resulting in a 0-length string. When using this filter
> as a default filter, see the warning below about setting the default flags
> to 0."
> Not to be mistaken with FILTER_SANITIZE_SPECIAL_CHARS. As long as it's not
> used with filter_input(), it's the least problematic. We
> have htmlspecialchars() though, so how useful is this filter?
>
> *FILTER_UNSAFE_RAW *- What makes it unsafe? Why isn't this just
> called FILTER_RAW_STRING? If the value being filtered is something other
> than a string, what will this filter return? Integers, floats, booleans and
> nulls are converted to a string, Arrays and objects make the filter fail.
>
> ---
>
> Let's quickly mention the filter flags.
>
> The FILTER_FLAG_STRIP_LOW flag will also remove tabs, carriage returns and
> newlines as these are all less than 32 ASCII codes. When is this useful and
> expected?
>
> The FILTER_FLAG_ENCODE_LOW flag "encodes" ASCII <32 codes presumably into
> HTML entities, although that's not specified anywhere in the PHP manual.
> The word HTML does not appear on the
> https://www.php.net/manual/en/filter.filters.flags.php page. What do these
> characters look like when presented by HTML? When is it ever useful to use
> this flag?
>
> FILTER_FLAG_ENCODE_AMP & FILTER_FLAG_STRIP_BACKTICK - why is this even a
> thing?
>
> Due to flags, FILTER_VALIDATE_EMAIL will happily validate email addresses
> that would be otherwise mangled by FILTER_SANITIZE_EMAIL.
>
> These are just the things I found confusing and strange about the sanitize
> filters. Let's try to put ourselves in the shoes of an average PHP
> developer trying to comprehend these filters. It's quite easy to shoot
> yourself in the foot if you try to use them. The PHP manual doesn't do a
> good job of explaining them, but that's probably because they are not easy
> to explain. I can't come up with good examples of when they should be used.
>
> Regards,
> Kamil

The filter extension has always been a stillborn mess.  Its API is an absolute disaster and, as you note, its functionality is unclear at best, misleading at worst.  Frankly it's worse than SPL.

I'd be entirely on board with jettisoning the entire thing, but baring that, ripping out large swaths of it that are misleading suits me fine.

--Larry Garfield

Hans Henrik Bergan

3 years ago

FILTER_SANITIZE_EMAIL should burn. If you have a bad email address, i can't
imagine the correct solution is to remove characters until it becomes
valid, short of a trim()

On Sun, Oct 2, 2022, 17:10 Larry Garfield <larry@garfieldtech.com> wrote:

David Gebler

3 years ago

On Sun, Oct 2, 2022 at 4:10 PM Larry Garfield <larry@garfieldtech.com>
wrote:

> The filter extension has always been a stillborn mess.  Its API is an
> absolute disaster and, as you note, its functionality is unclear at best,
> misleading at worst.  Frankly it's worse than SPL.
>
> I'd be entirely on board with jettisoning the entire thing, but baring
> that, ripping out large swaths of it that are misleading suits me fine.
>
>

The whole thing is seriously grim. Looking at the documentation for
filter_var for example, look at what it says for the third parameter,
$options

>  Associative array of options or bitwise disjunction of flags. If filter

accepts options, flags can be provided in "flags" field of array. For the
"callback" filter, callable type should be passed.

At a glance, I think all the examples mentioned in this thread have better
existing alternatives already in core and could just be deprecated then
removed. But it's worth asking, is that what we're talking about here, or
is there a suggestion of replacing the filter API with a more modern,
object API?

Max Semenik

3 years ago

пн, 3 окт. 2022 г., 03:18 David Gebler <davidgebler@gmail.com>:

> At a glance, I think all the examples mentioned in this thread have better
> existing alternatives already in core and could just be deprecated then
> removed. But it's worth asking, is that what we're talking about here, or
> is there a suggestion of replacing the filter API with a more modern,
> object API?
>

Is there a compelling need to have this in the core, as opposed to Composer
packages? The ecosystem has changed a lot since the original function was
introduced.

Rowan Collins

3 years ago

On 3 October 2022 11:29:40 BST, Max Semenik <maxsem.wiki@gmail.com> wrote:

>пн, 3 окт. 2022 г., 03:18 David Gebler <davidgebler@gmail.com>:
>
>> At a glance, I think all the examples mentioned in this thread have better
>> existing alternatives already in core and could just be deprecated then
>> removed. But it's worth asking, is that what we're talking about here, or
>> is there a suggestion of replacing the filter API with a more modern,
>> object API?
>>
>
>Is there a compelling need to have this in the core, as opposed to Composer
>packages? The ecosystem has changed a lot since the original function was
>introduced.

Quite the opposite, in my opinion - there are compelling reasons *not* to have this in core.

It turns out that making a universal validation and sanitisation library is really hard, and breaking changes and diverging needs are pretty much guaranteed. That's pretty much the worst case for something distributed with the language, and exactly what Composer excels at.

The only thing that does belong in core are narrowly targeted low-level functions that someone might use to build such a library. Certainly not some huge OO monster reimplementing the whole of ext/filter and making a whole bunch of new mistakes.

Regards,

--
Rowan Tommins
[IMSoP]

juan carlos morales

3 years ago

My 2 cents on this.

We should keep what is web related IMO. It does not make any sense to
take things out, that later everyone will write by its own, or end up
using a 3rd party package.

PHP should have what is web related already to be use.

Another different thing is the naming, the implementation code, etc.

An RFC documenting each case would be very helpful, to centralize the
ideas on each case, instead of scrolling the mailing list.

Cheers.

David Gebler

3 years ago

On Mon, Oct 3, 2022 at 11:29 AM Max Semenik <maxsem.wiki@gmail.com> wrote:

>
> Is there a compelling need to have this in the core, as opposed to
> Composer packages? The ecosystem has changed a lot since the original
> function was introduced.
>

I don't know that there is, I suspect the answer is probably not and
sanitization and validation is probably better left to userland code. The
only argument I can offer as devil's advocate is that certain validations
or transformations will be faster in core than in library scripts. I would
wager the most common implementation of such userland libraries today are
heavily reliant on preg_* functions so having some fast, low level baseline
in core for common tasks in this category might still make sense.

While we're on the topic, can I bring up FILTER_SANITIZE_NUMBER_FLOAT? Why
is the default behaviour of FILTER_SANITIZE_NUMBER_FLOAT the same as
FILTER_SANITIZE_NUMBER_INT unless you add extra flags to permit fractions?
Why is the constant name FILTER_SANITIZE_NUMBER_FLOAT but its counterpart
for validation is FILTER_VALIDATE_FLOAT (no NUMBER_)? Why does validating a
float return a float but sanitizing a float return a string?

What about FILTER_VALIDATE_EMAIL which is notorious for being next to
useless?

Seems to me like there could at the very least be a plausible case for some
better to_float(), to_int(), is_valid_email() etc. type functions in core
to replace some of the filter API.

Vasilii Shpilchin

3 years ago

I believe we are still dIscussing about the sanitizing filters only. No
doubt the filter API in general should be kept in the core as it provides
functional access to input variables with the filter_input() function. The
filter_input() is the only alternative to accessing superglobal arrays
directly. I prefer to use them rather than userland helpers and facades
which may work differently to each other. If you wanted to get back to
superglobal arrays when coding without a framework in PHP 8.3 I won't
believe in that. The set of the sanitizing filters is not perfect, however;
some filters are great and userful:

FILTER_SANITIZE_EMAIL - helps to clean up typical mess caused by
copy-pasting an email.
FILTER_SANITIZE_URI - similar thing but to URIs.
FILTER_SANITIZE_NUMBER_FLOAT - nice since it provides a flag to control
scientific notation (did you know is_float("1e1") is false,
but is_float(1e1), however, you always get a string from input variables,
and there is no other way to handle this case without weird manipulations
on a string).

The purpose of some filters like FILTER_SANITIZE_STRING is difficult to
get, I agree, but the idea to solve common edge-cases with built-in
high-quality functionality is  great, PHP is a language for Web and should
consider web context.



On Mon, Oct 3, 2022 at 8:38 PM David Gebler <davidgebler@gmail.com> wrote:

Rowan Collins

3 years ago

On 04/10/2022 01:38, David Gebler wrote:

> What about FILTER_VALIDATE_EMAIL which is notorious for being next to
> useless?
> [...]
> Seems to me like there could at the very least be a plausible case for some
> better [...] is_valid_email() etc. type functions in core
> to replace some of the filter API.

The "notorious" thing I know is that validating e-mail addresses is next
to impossible because of multiple overlapping standards, and a huge
number of esoteric variations that might or might not actually be
deliverable in practice. If you think the implementation can be
improved, that doesn't need a new is_valid_email() function, just a
tested and documented patch to the existing one; if it can't be
improved, then any new function will be just as useless.

In practice, the most common typos don't result in invalid e-mail
addresses anyway, just incorrect ones - "gamil.com" instead of
"gmail.com", and so on. For those, you don't need to Validate or
Sanitize; you need to Escape and Verify: escape what you're given
(context-dependent, so necessarily part of an SMTP or API client
library), attempt to send an e-mail, and wait for the user to verify
they've received it.



On 04/10/2022 02:29, Vasilii Shpilchin wrote:

> filter_input() is the only alternative to accessing superglobal arrays
> directly.

[...]

> FILTER_SANITIZE_EMAIL - helps to clean up typical mess caused by
> copy-pasting an email.
> FILTER_SANITIZE_URI - similar thing but to URIs.
> FILTER_SANITIZE_NUMBER_FLOAT - nice since it provides a flag to control
> scientific notation

None of these sounds very useful to me, but I think that just confirms
the biggest problem with the extension: it's trying to be everything to
everyone, and ends up with a bewildering set of options as a result. I
don't think any rewrite or replacement can ever avoid that problem,
because it's inherent in the problem space.

I have a draft proposal I might share soon for some "strict cast"
functions, but even simple cases like "string to integer" could have a
dozen different implementations which would all be equally "valid"
according to some use case or opinion, so it's a bit of a quagmire.

Regards,

--
Rowan Tommins
[IMSoP]

David Gebler

3 years ago

On Tue, Oct 4, 2022 at 11:34 AM Rowan Tommins <rowan.collins@gmail.com>
wrote:

> The "notorious" thing I know is that validating e-mail addresses is next
> to impossible because of multiple overlapping standards, and a huge
> number of esoteric variations that might or might not actually be
> deliverable in practice. If you think the implementation can be
> improved, that doesn't need a new is_valid_email() function, just a
> tested and documented patch to the existing one; if it can't be
> improved, then any new function will be just as useless
>

There are multiple RFC standards for email address format but AFAIK PHP's
FILTER_SANITIZE_EMAIL doesn't conform to any of them.

The idea behind my suggestion for something like is_valid_email (whatever
it might be named) is as a step towards deprecating and removing the entire
existing filter API, which I think many of us agree is a mess. As you said
below "it's trying to be everything to everyone, and ends up with a
bewildering set of options" - a rewrite or replacement which also tries to
be everything to everyone won't solve that problem, but getting rid of it
entirely will.

That said, the nature of PHP as a web-first language means it's reasonable
to include some individual, smaller, better APIs for certain validations or
sanitizations on types of data which are very commonly encountered in HTTP
requests. Examples include strings we expect or want to be valid integers,
decimals, email addresses and URLs. I think these features should remain,
but I'd happily see them even as a set of new, individual core functions if
it meant binning off filter_var and filter_input in PHP 9.

Regardless, look - I don't want to derail here - if most people are happy
with just deprecating some of the crappier and more confusing sanitize
filters and leave it at that, I say great, go for it, it's still an
improvement. I'm just saying if someone's going to take the time to look at
that problem space, why not go more than half the distance and reconsider
the fundamental approach of something we all know is pretty sucky anyway?

Just food for thought.

Rowan Collins

3 years ago

On 05/10/2022 22:35, David Gebler wrote:

> There are multiple RFC standards for email address format but AFAIK
> PHP's FILTER_SANITIZE_EMAIL doesn't conform to any of them.

FILTER_SANITIZE_EMAIL is a very short list of characters which claims to
be based on RFC 822 section 6:
https://heap.space/xref/php-src/ext/filter/sanitizing_filters.c?r=4df3dd76#295

FILTER_VALIDATE_EMAIL doesn't say exactly which standard it's attempting
to adhere to; it's one of many long unreadable regexes I've seen online
claiming to cover all possible addresses. (Actually, there are now two
regexes there, because there's a different version to support
FILTER_FLAG_EMAIL_UNICODE).
https://heap.space/xref/php-src/ext/filter/logical_filters.c?r=d8fc05c0#651

> The idea behind my suggestion for something like is_valid_email
> (whatever it might be named) is as a step towards deprecating and
> removing the entire existing filter API, which I think many of us
> agree is a mess.

You described FILTER_VALIDATE_EMAIL as "notorious for being next to
useless"; that gives us two possibilities:

a) A new function will be just as useless, because it will be based on
the same implementation
b) There is a better implementation out there, which we should start
using in ext/filter right now

My gut feel is that (a) is true, and there is no point considering what
a new function would be called, because we don't know how to implement it.


Regards,

--
Rowan Tommins
[IMSoP]

Claude Pache

3 years ago

> Le 6 oct. 2022 à 10:19, Rowan Tommins <rowan.collins@gmail.com> a écrit :
>
> You described FILTER_VALIDATE_EMAIL as "notorious for being next to useless"; that gives us two possibilities:
>
> a) A new function will be just as useless, because it will be based on the same implementation
> b) There is a better implementation out there, which we should start using in ext/filter right now
>
> My gut feel is that (a) is true, and there is no point considering what a new function would be called, because we don't know how to implement it.

Hi,

While it may be difficult to validate an email according to some IETF’s RFC, the HTML standard has pragmatically adopted a pattern (used to validate `<input type=email>` fields) that is both readable and suitable for most practical purposes. See:

https://html.spec.whatwg.org/multipage/input.html#valid-e-mail-address <https://html.spec.whatwg.org/multipage/input.html#valid-e-mail-address>

—Claude

Thomas Hruska

3 years ago

On 10/6/2022 1:19 AM, Rowan Tommins wrote:

> On 05/10/2022 22:35, David Gebler wrote:
>> There are multiple RFC standards for email address format but AFAIK
>> PHP's FILTER_SANITIZE_EMAIL doesn't conform to any of them.
>
> FILTER_SANITIZE_EMAIL is a very short list of characters which claims to
> be based on RFC 822 section 6:
> https://heap.space/xref/php-src/ext/filter/sanitizing_filters.c?r=4df3dd76#295
>
> FILTER_VALIDATE_EMAIL doesn't say exactly which standard it's attempting
> to adhere to; it's one of many long unreadable regexes I've seen online
> claiming to cover all possible addresses. (Actually, there are now two
> regexes there, because there's a different version to support
> FILTER_FLAG_EMAIL_UNICODE).
> https://heap.space/xref/php-src/ext/filter/logical_filters.c?r=d8fc05c0#651
>
>> The idea behind my suggestion for something like is_valid_email
>> (whatever it might be named) is as a step towards deprecating and
>> removing the entire existing filter API, which I think many of us
>> agree is a mess.
>
> You described FILTER_VALIDATE_EMAIL as "notorious for being next to
> useless"; that gives us two possibilities:
>
> a) A new function will be just as useless, because it will be based on
> the same implementation

> b) There is a better implementation out there, which we should start
> using in ext/filter right now

For (b), well, there is always the option of handling email addresses
the way the IETF intended instead of using regexes.

For example, SMTP::MakeValidEmailAddress() from:

https://github.com/cubiclesoft/ultimate-email

Does three things quite differently from ext/filter:

1)  It uses a custom state engine to implement half of the relevant IETF
EBNF grammars and then cheats for the other half.  The very complex
specifications that the IETF (and W3C) produces should generally be
implemented as custom state engines (finite state machines or FSMs) in
software.  A custom state engine can correctly identify certain common
input errors and both transparently and correctly fix those errors in
very specific instances as it processes the input (e.g. gmail,com ->
gmail.com happens often).  State engines can also accurately and
correctly do things such as remove CFWS (comments and folding
whitespace) from email addresses, which are not necessary components of
an email address and CFWS causes all kinds of issues.  State engines,
when done right, can even outperform all other functional
implementations.  State engines can also read partial input and maintain
their internal state while using few resources to process very large
inputs (not particularly relevant in this case).  The current
regex-based approach in ext/filter is obviously causing some problems
that can probably be fixed by using a custom state engine.

Important caveat:  Custom state engines do run the risk of winding up in
an infinite loop when forgetting to properly transition between states
or forgetting to move pointers through the input, resulting in DoS
issues.  Been there, done that - they are both very easy things to do.

2)  It parses email addresses in reverse:  Domain part first, local part
second.  The EBNF grammars for the domain part are simpler and less
contentious than the grammars for the local part.  Also, IIRC, the
domain portion can't contain '@' while the local portion can - it's been
a while since I looked at the specs though.

3)  It considers sanitization and validation as being the same function.
There is no separate SMTP::IsValidEmailAddress() in the library
because there is no need for one.  If MakeValidEmailAddress() can't turn
an input into a valid email address string, it returns an error.  If the
returned email address is not the same as the one that was input, the
original address can be viewed as technically "invalid."  One shared
internal function for both FILTER_SANITIZE_EMAIL and
FILTER_VALIDATE_EMAIL would produce consistent output/results.


Other thoughts:  I'm aware that a regex is effectively defining a state
engine as a compact string.  However, as evidenced by the two Perl CPAN
regexes for email addresses currently in use, regexes are limited in
utility/function and are somewhat inflexible, get more difficult to read
and comprehend once they get longer than a few dozen bytes, and can't
readily correct errors or other problems in complex input strings.  The
~250 lines of userland code referenced above is also not perfect (e.g.
extracting characters using substr() is rather inefficient) but it works
well enough.  The userland code also performs a DNS MX record check by
default, but that is its own complex can of worms and was probably not
the best idea I've ever had.  However, the three main concepts are the
important takeaways here, not the referenced userland code.

> My gut feel is that (a) is true, and there is no point considering what
> a new function would be called, because we don't know how to implement it.

Perhaps the above will help to at least provide some new ideas to think
about/ponder.

--
Thomas Hruska
CubicleSoft President

CubicleSoft has over 80 original open source projects and counting.
Plus a couple of commercial/retail products.

What software are you looking to build?

Claude Pache

3 years ago

Hi,

> FILTER_SANITIZE_ENCODED
> FILTER_SANITIZE_SPECIAL_CHARS

See https://www.php.net/manual/en/function.filter-input.php <https://www.php.net/manual/en/function.filter-input.php> Example #1 for an example of use. Apparently, “escaping” is considered as part of  “sanitizing”?

If you want to educate your users, you can consider to deprecate them in favor of FILTER_DEFAULT followed by `urlencode()`, respectively `htmlspecialchars()`. Ditto for various other FILTER_SANITIZE_* filters.

> FILTER_UNSAFE_RAW

My wild guess is that “unsafe” means that “it is dangerous to use the result in random contexts (i.e., without properly escaping it, because we assume that you don’t even know what “escape” means). Use FILTER_SANITIZE_ENCODED, FILTER_SANITIZE_SPECIAL_CHARS and/or FILTER_SANITIZE_MAGIC_QUOTES if you want to be safe” (for some nonstandard definition of “safe”). Of course, it should be renamed, because “safety” may be achieved by alternative means.

—Claude

Yasuo Ohgaki

3 years ago

Kamil Tekiela <tekiela246@gmail.com>:

> These are just the things I found confusing and strange about the sanitize
> filters. Let's try to put ourselves in the shoes of an average PHP
> developer trying to comprehend these filters. It's quite easy to shoot
> yourself in the foot if you try to use them. The PHP manual doesn't do a
> good job of explaining them, but that's probably because they are not easy
> to explain. I can't come up with good examples of when they should be used.
>

I agree there are many confusing names/features/behaviors.
IMO, input validation and output sanitization should be 2 different
features.

https://wiki.sei.cmu.edu/confluence/display/seccode/Top+10+Secure+Coding+Practices

Input validation is the 1st secure coding principle for input data
handling. Output sanitization
is the 7th secure coding principle for output data handling. Filter module
is mixing these up.
(And input validation should not sanitize input, but validate. Otherwise,
the web app is not
OWASP TOP 10 compliant. i.e. OWASP TOP 10 A09:2021 requires to detect DAST
attacks)

I wrote the input validation part years ago, if anyone is interested.
https://github.com/yohgaki/validate-php (Obsolete  C module. Do not use)
https://github.com/yohgaki/validate-php-scr (PHP library)

--
Yasuo Ohgaki
yohgaki@ohgaki.net

Derick Rethans

3 years ago

Hi all,

On Sat, 1 Oct 2022, Kamil Tekiela wrote:

> For quite some time now, PHP's sanitize filters have "Rustled My
> Jimmies". These filters bother me because I can't really justify their
> existence. I can understand that a few of them are sensible and may
> come in handy, but I would like to talk about some of these in
> particular.

I want to provide some context to why we have ext/filter, and why the
filters that we currently have exist. At the time when we introduced
ext/filter (which I mostly wrote), we were beholden to the scourge of
"magic quotes".

In order for PHP to allow for a safer acceptance of input variables into
a script, we added the ext/filter API to do so. The filters and
sanitisers that we added were at that moment reasonable to add, and also
likely to be used. We did punt on a view, and I am sure we made some
'interesting' decisions.

For example the e-mail validator was not designed to allow for what the
full spec allowed, but instead what we thought would be in-put by
reasonable people.

The sanitising filters were added to get a rough, but reasonable filter
to make data safe for specific contexts.

Some of them were added so that people could easily upgrade, but for
example setting the default filter to "magic_quotes" (or "add_slashes").
They're probably less useful *now*, but that doesn't distract that they
might still be in use.

I do believe we need to be better in promoting ext/filter's *good use*,
of which there are plenty of cases. And evulating on how to *improve*
(and not *remove) filters and sanitisers would be useful too.

Do you have specific suggestions towards that?

cheers,
Derick