Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » A validator module for PHP7 »

A validator module for PHP7

php.internals

Yasuo Ohgaki

9 years ago
Hi all, I spent a little time for a new input validation module. It's not totally new module, but is based on Filter module's validation filter improvement RFC in many ways. [1] As all of us knew already, input validation is the most important practice in secure coding. [2][3] Yet, we don't provide usable feature out of box. Sadly, almost all apps do not have proper input validation at trust boundary. Unless we improve filter's validation, we need usable basic validator by default. IMO. Since I didn't get much feedbacks during the RFC discussion, I cannot tell what part is disliked. I guess too much features in filter is one reason. Another is messed up codes/features by providing both "filter" and "validation". Validator for PHP7 (validate module) gets rid of unneeded features. It only has features for basic PHP data type validations. Validation rule(spec) array is flexible enough. Almost any types of inputs could be handled by multiple and nested validation rules. Except some minor features like overflow checks, most planned features are implemented. https://github.com/yohgaki/validate-php Although the code is based on filter module's code, it's almost full rewrite except validation logic came from filter. Please consider this as under development module. Feedbacks are appreciated. Regards, [1] https://wiki.php.net/rfc/add_validate_functions_to_filter [2] https://www.securecoding.cert.org/confluence/display/seccode/Top+10+Secure+Coding+Practices [3] https://www.owasp.org/index.php/OWASP_Secure_Coding_Practices_-_Quick_Reference_Guide
-- Yasuo Ohgaki yohgaki@ohgaki.net

Rowan Collins

9 years ago
On 4 September 2017 07:33:41 BST, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote:
>Hi all, > >I spent a little time for a new input validation module. It's not >totally >new module, but is based on Filter module's validation filter >improvement >RFC in many ways. [1]
Hi Yasuo, Thanks for tackling this. I do think the current filter module is user unfriendly and it would be great to have something better. A couple of quick thoughts based on your README: - The use of nested arrays keeps things simple in one sense, but the deep nesting can get confusing. I wonder if a ValidationRule class would make the distinction between parameters and references to existing rules clearer. This would also give you a natural home for validating the validation rules themselves (probably by throwing an exception in the constructor). - A minor point, but most style guides would suggest function names should be verbs, not adjectives, so "validate()" rather than "valid()". - Is there a use case for valid_id() or is it a temporary debugging thing that won't be in the final version? In general, I like the idea, but would have to play around a bit to see if it felt easy to use in real world situations. Regards,
-- Rowan Collins [IMSoP]

Victor Bolshov

9 years ago
In most cases users would like more than just valid/invalid, i. e. which particular rule(s) failed and also human-readable error messages. As of simple validation that is almost always at hand, filter_* functions do a good job, although I agree that they could be better. I, for one, would like to see a full-featured validation as part of PHP. However, this RFC only looks like a slightly better version of filter_* functions, that is, the new module will provide almost the same functionality but with a different interface. I would vote against it. On Mon, Sep 4, 2017 at 2:54 PM Rowan Collins <rowan.collins@gmail.com> wrote:
> On 4 September 2017 07:33:41 BST, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote: > >Hi all, > > > >I spent a little time for a new input validation module. It's not > >totally > >new module, but is based on Filter module's validation filter > >improvement > >RFC in many ways. [1] > > Hi Yasuo, > > Thanks for tackling this. I do think the current filter module is user > unfriendly and it would be great to have something better. A couple of > quick thoughts based on your README: > > - The use of nested arrays keeps things simple in one sense, but the deep > nesting can get confusing. I wonder if a ValidationRule class would make > the distinction between parameters and references to existing rules > clearer. This would also give you a natural home for validating the > validation rules themselves (probably by throwing an exception in the > constructor). > > - A minor point, but most style guides would suggest function names should > be verbs, not adjectives, so "validate()" rather than "valid()". > > - Is there a use case for valid_id() or is it a temporary debugging thing > that won't be in the final version? > > In general, I like the idea, but would have to play around a bit to see if > it felt easy to use in real world situations. > > Regards, > > -- > Rowan Collins > [IMSoP] > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > > --
Best regards, Victor Bolshov

Marco Pivetta

9 years ago
On Mon, Sep 4, 2017 at 8:56 PM, Crocodile <crocodile2u@gmail.com> wrote:
> In most cases users would like more than just valid/invalid, i. e. which > particular rule(s) failed and also human-readable error messages. As of > simple validation that is almost always at hand, filter_* functions do a > good job, although I agree that they could be better. I, for one, would > like to see a full-featured validation as part of PHP. However, this RFC > only looks like a slightly better version of filter_* functions, that is, > the new module will provide almost the same functionality but with a > different interface. I would vote against it. >
And also, it would need to be better than all of these to be worth writing it in C: https://packagist.org/search/?q=validator Marco Pivetta http://twitter.com/Ocramius http://ocramius.github.com/

Paul M Jones

9 years ago
> On Sep 4, 2017, at 18:06, Marco Pivetta <ocramius@gmail.com> wrote: > > On Mon, Sep 4, 2017 at 8:56 PM, Crocodile <crocodile2u@gmail.com> wrote: > >> In most cases users would like more than just valid/invalid, i. e. which >> particular rule(s) failed and also human-readable error messages. As of >> simple validation that is almost always at hand, filter_* functions do a >> good job, although I agree that they could be better. I, for one, would >> like to see a full-featured validation as part of PHP. However, this RFC >> only looks like a slightly better version of filter_* functions, that is, >> the new module will provide almost the same functionality but with a >> different interface. I would vote against it. >> > > And also, it would need to be better than all of these to be worth writing > it in C: > > https://packagist.org/search/?q=validator
And these as well: https://packagist.org/search/?q=filter (Or at least have some level of parity.)
-- Paul M. Jones pmjones88@gmail.com http://paul-m-jones.com Modernizing Legacy Applications in PHP https://leanpub.com/mlaphp Solving the N+1 Problem in PHP https://leanpub.com/sn1php

Yasuo Ohgaki

9 years ago
Hi all, On Tue, Sep 5, 2017 at 12:19 PM, Paul Jones <pmjones88@gmail.com> wrote:
> > > On Sep 4, 2017, at 18:06, Marco Pivetta <ocramius@gmail.com> wrote: > > > > On Mon, Sep 4, 2017 at 8:56 PM, Crocodile <crocodile2u@gmail.com> wrote: > > > >> In most cases users would like more than just valid/invalid, i. e. which > >> particular rule(s) failed and also human-readable error messages. As of > >> simple validation that is almost always at hand, filter_* functions do a > >> good job, although I agree that they could be better. I, for one, would > >> like to see a full-featured validation as part of PHP. However, this RFC > >> only looks like a slightly better version of filter_* functions, that > is, > >> the new module will provide almost the same functionality but with a > >> different interface. I would vote against it. > >> > > > > And also, it would need to be better than all of these to be worth > writing > > it in C: > > > > https://packagist.org/search/?q=validator > > And these as well: https://packagist.org/search/?q=filter
I cannot guess people's thought. I appreciated feedback! Why do you think basic validation module should be better than full OO implementation(s)? Simple and basic type support NULL/BOOL/INT/FLOAT/STRING/ARRAY/OBJECT is enough for C written PHP module. IMHO, PHP modules are better of to provide basic features that may be extended by user scripts. I picked 1st one on the list as an example. This kind of rule construction is inefficient compare to simple array rules, so I doubt this is the way for basic validator module written by C. $validator = Validation::createValidator(); $violations = $validator->validate('Bernhard', array( new Length(array('min' => 10)), new NotBlank(), )); However, this particular validation class seems it could be good one that wraps validate module for nicer OO API and faster execution. Regards,
-- Yasuo Ohgaki yohgaki@ohgaki.net

Lester Caine

9 years ago
On 05/09/17 12:18, Yasuo Ohgaki wrote:
> I cannot guess people's thought. I appreciated feedback!
With a decent database layer a lot of the validation you are proposing is already covered but PDO does not help in this area. Adding another layer that does not integrate with a storage layer is just adding to the current mess ...
-- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

Rowan Collins

9 years ago
On 5 September 2017 12:36:42 BST, Lester Caine <lester@lsces.co.uk> wrote:
>On 05/09/17 12:18, Yasuo Ohgaki wrote: >> I cannot guess people's thought. I appreciated feedback! > >With a decent database layer a lot of the validation you are proposing >is already covered but PDO does not help in this area. Adding another >layer that does not integrate with a storage layer is just adding to >the >current mess ...
Validation should have nothing to do with the storage layer. Or at least, there should be a level of validation separate from the storage layer. Inputs may come from all sorts of sources: the HTTP request, an API, an import file, etc; and they may be going to all sorts of destinations: the HTTP response, a database, an API, an export file, an e-mail, etc. Regardless of where it came from, and where it's going to end up, the application knows what format that input data should be in to use as or populate appropriate domain models. For instance, "age_in_years is a non-negative integer" is an invariant fact about the domain being modelled, even if it's a value that goes nowhere near any form of database. I actually agree that this module doesn't need to replace the existing userland libraries, only to act as a useful base for them, as well as a useful fallback when writing "raw PHP". The key problem is balancing flexibility and usability such that people will reach for this tool rather than brewing their own. Regards,
-- Rowan Collins [IMSoP]

lists@rhsoft.net

9 years ago
Am 05.09.2017 um 13:36 schrieb Lester Caine:
> On 05/09/17 12:18, Yasuo Ohgaki wrote: >> I cannot guess people's thought. I appreciated feedback! > > With a decent database layer a lot of the validation you are proposing > is already covered but PDO does not help in this area. Adding another > layer that does not integrate with a storage layer is just adding to the > current mess ...
sorry, but you confuse "input validation" which this topic is about with something different - input validation and reject bad requests belongs some layers on top of any storage and should be done as soon as possible that should even happen long before you open a database connection at all because when you know the request is bad soon enough you won't talk to any database, filesystem or whatever storage layer at all the only question as applicaton developer is how you proceed in which cases * reject the whole request with a error-message * reset form-fields where you don't expect an array as input * reset from-fields with out-of-range input values here you go: https://en.wikipedia.org/wiki/Data_validation

Lester Caine

9 years ago
On 05/09/17 14:08, lists@rhsoft.net wrote:
> > > Am 05.09.2017 um 13:36 schrieb Lester Caine: >> On 05/09/17 12:18, Yasuo Ohgaki wrote: >>> I cannot guess people's thought. I appreciated feedback! >> >> With a decent database layer a lot of the validation you are proposing >> is already covered but PDO does not help in this area. Adding another >> layer that does not integrate with a storage layer is just adding to the >> current mess ... > > sorry, but you confuse "input validation" which this topic is about with > something different - input validation and reject bad requests belongs > some layers on top of any storage and should be done as soon as possible > > that should even happen long before you open a database connection at > all because when you know the request is bad soon enough you won't talk > to any database, filesystem or whatever storage layer at all > > the only question as applicaton developer is how you proceed in which cases > > * reject the whole request with a error-message > * reset form-fields where you don't expect an array as input > * reset from-fields with out-of-range input values > > here you go: > https://en.wikipedia.org/wiki/Data_validation
When the database layer provides a complete list of fields and validation rules as part of it's meta data, it is integral to any GOOD process. Copying all that data and manually creating filter rules is just unnecessary work. In addition much of the VALIDATION is best done at the browser end, and building that code is a lot easier when there is a standard validation base across all of the layers! Rejecting crap from hackers that have no format matching the fields on the browser page is something else and if the data set is corrupt then yes you can simply skip out before doing anything with it! But the problem these days is when hackers try injecting things like SQL into fields they think may be able to get through to the database. Provided that the validation layer can properly filter that injection requires knowledge that a string has reason to be rejected. Just as simply type casting a number to integer or float is only doing a small part of the job. Typing and validating a field by the metadata constraints has to be the right way forward?
-- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

lists@rhsoft.net

9 years ago
Am 05.09.2017 um 15:44 schrieb Lester Caine:
> On 05/09/17 14:08, lists@rhsoft.net wrote: >> the only question as applicaton developer is how you proceed in which cases >> >> * reject the whole request with a error-message >> * reset form-fields where you don't expect an array as input >> * reset from-fields with out-of-range input values >> >> here you go: >> https://en.wikipedia.org/wiki/Data_validation > > When the database layer provides a complete list of fields and > validation rules as part of it's meta data, it is integral to any GOOD > process
your first error is thinking every input is related to databases at all
> Copying all that data and manually creating filter rules is > just unnecessary work. In addition much of the VALIDATION is best done > at the browser end, and building that code is a lot easier when there is > a standard validation base across all of the layers!
NO VALIDATION is best done in the browser end because no attacker ever will execute your clientside validation code or operate a browser at all
> Rejecting crap from hackers that have no format matching the fields on > the browser page is something else and if the data set is corrupt then > yes you can simply skip out before doing anything with it!
and that's what the whole topic is about

Yasuo Ohgaki

9 years ago
Hi Lester, On Tue, Sep 5, 2017 at 8:36 PM, Lester Caine <lester@lsces.co.uk> wrote:
> On 05/09/17 12:18, Yasuo Ohgaki wrote: > > I cannot guess people's thought. I appreciated feedback! > > With a decent database layer a lot of the validation you are proposing > is already covered but PDO does not help in this area. Adding another > layer that does not integrate with a storage layer is just adding to the > current mess ... >
I'm fun of multiple tier and multiple layer of protections. For instance, Microsoft's SQL injection security page states as follows. - Never build Transact-SQL statements directly from user input; use stored procedures to validate user input. - Validate user input by testing type, length, format, and range. Use the Transact-SQL QUOTENAME() function to escape system names or the REPLACE() function to escape any character in a string. - Implement multiple layers of validation in each tier of your application. https://docs.microsoft.com/en-us/dotnet/framework/data/adonet/sql/writing-secure-dynamic-sql-in-sql-server This is what secure coding practice recommends, too. It may seem mess, but it's not. Outermost trust boundary that can be controlled is the most important trust boundary. For server side web app developers, outermost trust boundary is controller in MVC model. Input validations at model is a bit too late to mitigate risks involved with invalid(attacker) inputs. Both model and controller layer Input validations (as well as in the database, too) are good/important to have. There are one principle that developers are better to follow. https://en.wikipedia.org/wiki/Fail-fast If we follow this principle, validation at controller makes sense. Regards, P.S. For database administrators or web app developers who maintain application Models, outermost trust boundary is "database system" and "the Model layer" respectively. Outermost trust boundary is changed by what they can control. This kind of discussion could result in mess. I hope I explained well enough.
-- Yasuo Ohgaki yohgaki@ohgaki.net

Yasuo Ohgaki

8 years ago
On Mon, Sep 4, 2017 at 3:33 PM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote:
> Hi all, > > I spent a little time for a new input validation module. It's not totally > new module, but is based on Filter module's validation filter improvement > RFC in many ways. [1] > > As all of us knew already, input validation is the most important practice > in secure coding. [2][3] Yet, we don't provide usable feature out of box. > Sadly, almost all apps do not have proper input validation at trust > boundary. Unless we improve filter's validation, we need usable basic > validator by default. IMO. > > Since I didn't get much feedbacks during the RFC discussion, I cannot tell > what part is disliked. I guess too much features in filter is one reason. > Another is messed up codes/features by providing both "filter" and > "validation". > > Validator for PHP7 (validate module) gets rid of unneeded features. It > only has features for basic PHP data type validations. Validation > rule(spec) array is flexible enough. Almost any types of inputs could be > handled by multiple and nested validation rules. > > Except some minor features like overflow checks, most planned features are > implemented. > > https://github.com/yohgaki/validate-php > > Although the code is based on filter module's code, it's almost full > rewrite except validation logic came from filter. Please consider this as > under development module. > Feedbacks are appreciated. > > Regards, > > [1] https://wiki.php.net/rfc/add_validate_functions_to_filter > [2] https://www.securecoding.cert.org/confluence/display/ > seccode/Top+10+Secure+Coding+Practices > [3] https://www.owasp.org/index.php/OWASP_Secure_Coding_ > Practices_-_Quick_Reference_Guide > > -- > Yasuo Ohgaki > yohgaki@ohgaki.net >
I thought it would be nice to have PHP script version for Validate PHP. It a lot easier to modify API as needed. So I spend few hours last weekend. https://github.com/yohgaki/validate-php-scr Caution, I just wrote it and didn't debug it yet. However, it is good enough to play with, I suppose. API differs a little. This has more simplified parameter structure. Suggestions and comments are appreciated. Regards,
-- Yasuo Ohgaki yohgaki@ohgaki.net

Victor Bolshov

8 years ago
It's almost always the case that you need to provide a meaningful feedback about what exactly went wrong, rather then to just say "Failed!" While simplicity is nice and you cannot overrate value of validation, this whole thing is pretty much useless to me personally without this ability. Also, I don't think it's a good idea to mix validation of scalar values, arrays and even multiple arrays, in a single function. On Tue, Mar 27, 2018 at 11:43 AM Yasuo Ohgaki <yohgaki@ohgaki.net> wrote:
> On Mon, Sep 4, 2017 at 3:33 PM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote: > > > Hi all, > > > > I spent a little time for a new input validation module. It's not totally > > new module, but is based on Filter module's validation filter improvement > > RFC in many ways. [1] > > > > As all of us knew already, input validation is the most important > practice > > in secure coding. [2][3] Yet, we don't provide usable feature out of box. > > Sadly, almost all apps do not have proper input validation at trust > > boundary. Unless we improve filter's validation, we need usable basic > > validator by default. IMO. > > > > Since I didn't get much feedbacks during the RFC discussion, I cannot > tell > > what part is disliked. I guess too much features in filter is one reason. > > Another is messed up codes/features by providing both "filter" and > > "validation". > > > > Validator for PHP7 (validate module) gets rid of unneeded features. It > > only has features for basic PHP data type validations. Validation > > rule(spec) array is flexible enough. Almost any types of inputs could be > > handled by multiple and nested validation rules. > > > > Except some minor features like overflow checks, most planned features > are > > implemented. > > > > https://github.com/yohgaki/validate-php > > > > Although the code is based on filter module's code, it's almost full > > rewrite except validation logic came from filter. Please consider this as > > under development module. > > Feedbacks are appreciated. > > > > Regards, > > > > [1] https://wiki.php.net/rfc/add_validate_functions_to_filter > > [2] https://www.securecoding.cert.org/confluence/display/ > > seccode/Top+10+Secure+Coding+Practices > > [3] https://www.owasp.org/index.php/OWASP_Secure_Coding_ > > Practices_-_Quick_Reference_Guide > > > > -- > > Yasuo Ohgaki > > yohgaki@ohgaki.net > > > > > I thought it would be nice to have PHP script version for > Validate PHP. It a lot easier to modify API as needed. So > I spend few hours last weekend. > > https://github.com/yohgaki/validate-php-scr > > Caution, I just wrote it and didn't debug it yet. > However, it is good enough to play with, I suppose. > > API differs a little. This has more simplified parameter > structure. Suggestions and comments are appreciated. > > Regards, > > -- > Yasuo Ohgaki > yohgaki@ohgaki.net >
-- Best regards, Victor Bolshov

Yasuo Ohgaki

8 years ago
On Tue, Mar 27, 2018 at 8:03 PM, Crocodile <crocodile2u@gmail.com> wrote:
> It's almost always the case that you need to provide a meaningful feedback > about what exactly went wrong, rather then to just say "Failed!" While > simplicity is nice and you cannot overrate value of validation, this whole > thing is pretty much useless to me personally without this ability. Also, I > don't think it's a good idea to mix validation of scalar values, arrays and > even multiple arrays, in a single function. >
It seems you underrate validations. It's a fundamental requirement for programs to work correctly. Fundamental requirement: Programs can only work correctly with valid inputs. There are countless reason why one should validate ALL inputs other than this. Almost nobody does this now, even if it is "fundamental" requirement. How you would solve this?
-- Yasuo Ohgaki yohgaki@ohgaki.net

Yasuo Ohgaki

8 years ago
On Tue, Mar 27, 2018 at 8:03 PM, Crocodile <crocodile2u@gmail.com> wrote:
> It's almost always the case that you need to provide a meaningful feedback > about what exactly went wrong, rather then to just say "Failed!" While > simplicity is nice and you cannot overrate value of validation, this whole > thing is pretty much useless to me personally without this ability. Also, I > don't think it's a good idea to mix validation of scalar values, arrays and > even multiple arrays, in a single function. >
There are 3 types of validations in general. There are only 3 types of inputs. Explaining them need lots of word as well as for other fundamentals/principles related to this. https://blog.ohgaki.net/computer-programming-fundamentals-and-basics-input-validation You are referring to "Business Logic Validation" which needs feedback to users in case they send incorrect values. "App Level Input Validation" does not require feedback other than "You sent really bad request. This has been reported to admins." Therefore, App Level Input Validation can validate ALL inputs at once and there is no good reason not to do this. Btw, you can use the validator for a single value validation as well. It also able to disable Exception so that one can use it for "Business Logic Validations" as well. https://github.com/yohgaki/validate-php-scr/tree/master/src/examples Regards,
-- Yasuo Ohgaki yohgaki@ohgaki.net