The case for transpiled generics

php.internals

Brent

5 years ago
Hello internals Today I'd like to hear your thoughts on what might be a controversial topic, though I think it's worth having this discussion. I want to make the case for adding generic syntax, without actually enforing any additional type checks at runtime. Please hear me out. We've been discussing generics for years now [1][2], all without any result. Nikita's latest attempt [3] stalled because, from what I gathered and amongst other things, doing generic type checks at runtime has a significant impact on performance. On the other hand, static analysers have been making their rise for a few years now. Granted: not the whole community might like this kind of type strictness, and PHP doesn't force them to; but still projects like PhpStorm acknowledge their significance — they will add built-in support for both psalm and PHPStan later this year [4]. Rasmus Lerdorf also showed interest in the idea of improving PHP's static analysis capabilities two years ago [5]. That all to say that there's a significant part of the PHP community who's interested in embracing the benefits of static analysis. If we look outside of our PHP bubble, we can see the same thing happening in JavaScript: the core benefit that TypeScript adds is its robust static analysis. Sure those developers need an extra compilation step to transpile their code to plain old JavaScript, but it seems that they are… fine with that? I'd like to discuss a similar idea for PHP. If runtime generics aren't possible because of performance issues, why not explore the other option: adding generic syntax that is ignored by the interpreter, but can be used by static analysis tools — third party of built-into PHP, that's another discussion. I realise this thought goes against the "PHP mindset" we've been programming with for more than 20 years, but we I think we shouldn't ignore what's happening in the PHP- and wider progamming community: static analysis is relevant, whether you want to use it or not, and a stricter type system is prefered by many. Now I know there are alternatives we can use today. Static analysers already support generics, using doc blocks. I'm not trying to argue that it's impossible to achieve the same results with the toolset we have, but rather that there's room for improvement from the developer's point of view. History has shown that such convenience additions to PHP have been a difficult pill to swallow for some, but on the other hand those kind of changes _have_ been happening more and more often anyway: property promotion, short closures, named arguments, attributes, yes even types themselves: you can write the same working PHP program without any of those features, and yet they have been proven so useful and wanted over the last years. As a sidenote: the idea of transpiling is already present in PHP. Looking at constructor property promotion: a purely syntactical feature, which is transformed to simpler PHP code at runtime. Nikita called this principle "desugaring" in the constructor property promotion RFC [6]. So here's my case for transpiled generics summarized: - There's no significant runtime performance impact - The PHP community is already embracing static analysis - Transpiling has been proved to be a viable workflow, thanks to TypeScript - As with all things-PHP: it's opt-in. You don't have to use the syntax if you don't want to and you won't experience any downsides So with all that being said, I'm looking forward to hearing your thoughts. Kind regards Brent [1] https://wiki.php.net/rfc/generics <https://wiki.php.net/rfc/generics> [2] https://wiki.php.net/rfc/generic-arrays <https://wiki.php.net/rfc/generic-arrays> [3] https://github.com/PHPGenerics/php-generics-rfc/issues/45 <https://github.com/PHPGenerics/php-generics-rfc/issues/45> [4] https://blog.jetbrains.com/phpstorm/2020/07/phpstan-and-psalm-support-coming-to-phpstorm/ <https://blog.jetbrains.com/phpstorm/2020/07/phpstan-and-psalm-support-coming-to-phpstorm/> [5] https://externals.io/message/101477#101592 <https://externals.io/message/101477#101592> [6] https://wiki.php.net/rfc/constructor_promotion#desugaring <https://wiki.php.net/rfc/constructor_promotion#desugaring>

Girgias

5 years ago
On Thu, 17 Sep 2020 at 13:33, Brent Roose <brendt@stitcher.io> wrote:
> Hello internals > > Today I'd like to hear your thoughts on what might be a controversial > topic, though I think it's worth having this discussion. I want to make the > case for adding generic syntax, without actually enforing any additional > type checks at runtime. Please hear me out. > > We've been discussing generics for years now [1][2], all without any > result. Nikita's latest attempt [3] stalled because, from what I gathered > and amongst other things, doing generic type checks at runtime has a > significant impact on performance. > > On the other hand, static analysers have been making their rise for a few > years now. Granted: not the whole community might like this kind of type > strictness, and PHP doesn't force them to; but still projects like PhpStorm > acknowledge their significance — they will add built-in support for both > psalm and PHPStan later this year [4]. Rasmus Lerdorf also showed interest > in the idea of improving PHP's static analysis capabilities two years ago > [5]. > > That all to say that there's a significant part of the PHP community who's > interested in embracing the benefits of static analysis. > > If we look outside of our PHP bubble, we can see the same thing happening > in JavaScript: the core benefit that TypeScript adds is its robust static > analysis. Sure those developers need an extra compilation step to transpile > their code to plain old JavaScript, but it seems that they are… fine with > that? > > I'd like to discuss a similar idea for PHP. If runtime generics aren't > possible because of performance issues, why not explore the other option: > adding generic syntax that is ignored by the interpreter, but can be used > by static analysis tools — third party of built-into PHP, that's another > discussion. I realise this thought goes against the "PHP mindset" we've > been programming with for more than 20 years, but we I think we shouldn't > ignore what's happening in the PHP- and wider progamming community: static > analysis is relevant, whether you want to use it or not, and a stricter > type system is prefered by many. > > Now I know there are alternatives we can use today. Static analysers > already support generics, using doc blocks. I'm not trying to argue that > it's impossible to achieve the same results with the toolset we have, but > rather that there's room for improvement from the developer's point of > view. History has shown that such convenience additions to PHP have been a > difficult pill to swallow for some, but on the other hand those kind of > changes _have_ been happening more and more often anyway: property > promotion, short closures, named arguments, attributes, yes even types > themselves: you can write the same working PHP program without any of those > features, and yet they have been proven so useful and wanted over the last > years. > > As a sidenote: the idea of transpiling is already present in PHP. Looking > at constructor property promotion: a purely syntactical feature, which is > transformed to simpler PHP code at runtime. Nikita called this principle > "desugaring" in the constructor property promotion RFC [6]. > > So here's my case for transpiled generics summarized: > > - There's no significant runtime performance impact > - The PHP community is already embracing static analysis > - Transpiling has been proved to be a viable workflow, thanks to TypeScript > - As with all things-PHP: it's opt-in. You don't have to use the syntax if > you don't want to and you won't experience any downsides > > So with all that being said, I'm looking forward to hearing your thoughts. > > Kind regards > Brent > > [1] https://wiki.php.net/rfc/generics <https://wiki.php.net/rfc/generics> > [2] https://wiki.php.net/rfc/generic-arrays < > https://wiki.php.net/rfc/generic-arrays> > [3] https://github.com/PHPGenerics/php-generics-rfc/issues/45 < > https://github.com/PHPGenerics/php-generics-rfc/issues/45> > [4] > https://blog.jetbrains.com/phpstorm/2020/07/phpstan-and-psalm-support-coming-to-phpstorm/ > < > https://blog.jetbrains.com/phpstorm/2020/07/phpstan-and-psalm-support-coming-to-phpstorm/> > > [5] https://externals.io/message/101477#101592 < > https://externals.io/message/101477#101592> > [6] https://wiki.php.net/rfc/constructor_promotion#desugaring < > https://wiki.php.net/rfc/constructor_promotion#desugaring>
Hello Brent, I'm going to make the argument I've already done on Reddit once [1], IMHO TypeScript is just a nicer pipeline for a preprocessor and a static analyser and not a language per say. Let me explain with a simpler example than generics, adding immutable objects to PHP without native language support using a preprocessor and Psalm as the back-end for static-analysis. It would be rather trivial to do a search-and-replace of the word immutable and replace it with /** @psalm-immutable */ at the (currently non existent) preprocessor level then run Psalm and have it shout at you if "you are doing it wrong"TM. Now bundle the two tools into a nice CLI command run TypePHP root_of_project and voilà support for immutable by just writing immutable class Foo {} And I'm fairly certain that one can achieve support for generic templates in the same way, probably harder than adding support for immutable objects. It is also possible to have various back-ends for the static analysers which would change the doc-annotation to the one understood by the analyser asked. Most importantly this can be done purely in PHP and distributed as a composer package. Therefore needing no support from the language itself, thus this can even work in PHP 7. As such I see no benefit in supporting generics at the language level if they don't have any runtime checks. Best regards George P. Banyard [1] https://www.reddit.com/r/PHP/comments/i9h1v8/considering_php/g1h9umy?context=3

Brent

5 years ago
Hi George Thanks for the reply!
> On 17 Sep 2020, at 14:34, G. P. B. <george.banyard@gmail.com> wrote: > > Hello Brent, > > I'm going to make the argument I've already done on Reddit once [1], IMHO > TypeScript is just a nicer pipeline for a preprocessor and a static > analyser and not a language per say. > > Let me explain with a simpler example than generics, adding immutable > objects to PHP without native language support using a preprocessor and > Psalm as the back-end for static-analysis. It would be rather trivial to do > a search-and-replace of the word immutable and replace it with /** > @psalm-immutable */ at the (currently non existent) preprocessor level then > run Psalm and have it shout at you if "you are doing it wrong"TM. > Now bundle the two tools into a nice CLI command run TypePHP root_of_project > and voilà support for immutable by just writing > > immutable class Foo {} > > And I'm fairly certain that one can achieve support for generic templates > in the same way, probably harder than adding support for immutable objects. > It is also possible to have various back-ends for the static analysers > which would change the doc-annotation to the one understood by the analyser > asked. > > Most importantly this can be done purely in PHP and distributed as a > composer package. > Therefore needing no support from the language itself, thus this can even > work in PHP 7.
Preprocessing PHP has been done [1], and the reality is that it didn't work. Not because it's technically impossible, but because there's no tooling around it: no IDE integration and no static analysis. The pre project actually tried developing a plugin for PhpStorm, which failed due to plugin API limitations. Tooling and IDEs tend to follow the language spec, because there's no way around it. That power lies within the PHP project, and no one else in the community has the same amount of influence.
> > As such I see no benefit in supporting generics at the language level if > they don't have any runtime checks.
Like I said in my mail: it's not about being able to achieve a specific kind of functionality — static analysers already support generics — it's about giving developers an easy, integrated way of writing them, with as less overhead as possible. I think most new features added to PHP these days are a testament to that mindset — I've already mentioned them in my previous post.
> > Best regards > > George P. Banyard > > [1] > https://www.reddit.com/r/PHP/comments/i9h1v8/considering_php/g1h9umy?context=3
King regards Brent [1] https://preprocess.io/#/

Matthew Brown

5 years ago
Quick thing before I get into my own reaction: Transpiling is normally thought of as the process of converting one language into another. Tools like Babel transpile TypeScript to JavaScript. What's being proposed here (AFAICT) is type erasure – the generic type information would be erased during the conversion to opcodes. Python (the language) has type erasure for all its types (including its generic types), so no popular Python interpreters check that a function call's argument types match up with the function signature. PHP currently does not erase any types so the opcodes generated by PHP's interpreter include type checks for all the arguments passed to a typed function signature. Hack follows PHP's model, but erases generic types by default (though has more recently introduced the concept of reified generics) in much the same way you're proposing.

Larry Garfield

5 years ago
On Thu, Sep 17, 2020, at 10:21 AM, Matthew Brown wrote:
> Quick thing before I get into my own reaction: > > Transpiling is normally thought of as the process of converting one > language into another. Tools like Babel transpile TypeScript to JavaScript. > > What's being proposed here (AFAICT) is type erasure – the generic type > information would be erased during the conversion to opcodes. > > Python (the language) has type erasure for all its types (including its > generic types), so no popular Python interpreters check that a function > call's argument types match up with the function signature. > > PHP currently does not erase any types so the opcodes generated by PHP's > interpreter include type checks for all the arguments passed to a typed > function signature. > > Hack follows PHP's model, but erases generic types by default (though has > more recently introduced the concept of reified generics) in much the same > way you're proposing.
I concur; this is type elidiing, not transpiling. Transpiling implies a user-triggered pre-run step. I know Sara has mused about it before but it's not happened yet. Basically what is being proposed (for those who need a concrete example) is you'd write this: class Collection<T> { public function add(T $item) { ... } } $c = new Collection<Product>(). And linters would recognize that, and it would parse, but at runtime it would compile to: class Collection { public function add(mixed $item) { ... } } $c = new Collection(). And that's the opcodes that would be saved. (I'm hand-waving a lot here.) I would be on board with this, although I would ask if it's possible to provide some compile time checks; I'm thinking LSP validation, which is still relevant for inheritance with generics. Beyond that, leave the runtime to not enforce it. (I have no idea how viable that is, but Sara seemed to think it would work so it probably would.) Another possibility could be to desugar that original to: class Collection { public $__T; public function add(mixed $item) { if (!$item instanceof $this->__T) { throw new TypeError(); } } } $c = new Collection(). $c->__T == 'Product'; Which is essentially what you can do in userspace today, but automated. I don't know how viable that is, but it's another thing to consider. --Larry Garfield

Brent

5 years ago
Hi Matthew and Larry
> On 17 Sep 2020, at 18:21, Larry Garfield <larry@garfieldtech.com> wrote: > > On Thu, Sep 17, 2020, at 10:21 AM, Matthew Brown wrote: >> Quick thing before I get into my own reaction: >> >> Transpiling is normally thought of as the process of converting one >> language into another. Tools like Babel transpile TypeScript to JavaScript.
I looked up the definition [1] before writing my mail and it said
> A […] transpiler is a type of translator that takes the source code of a program written in a programming language as its input and produces an equivalent source code in the same or a different programming language
But I don't want to get stuck on phrasing, if elidiing is the right term as Larry suggests, let's go with it!
>> >> What's being proposed here (AFAICT) is type erasure – the generic type >> information would be erased during the conversion to opcodes. >> >> Python (the language) has type erasure for all its types (including its >> generic types), so no popular Python interpreters check that a function >> call's argument types match up with the function signature.
I didn't know about Python's approach before, and I think it's validating the idea, it's at least something worth looking into.
>> >> PHP currently does not erase any types so the opcodes generated by PHP's >> interpreter include type checks for all the arguments passed to a typed >> function signature. >> >> Hack follows PHP's model, but erases generic types by default (though has >> more recently introduced the concept of reified generics) in much the same >> way you're proposing. > > I concur; this is type elidiing, not transpiling. Transpiling implies a user-triggered pre-run step. I know Sara has mused about it before but it's not happened yet. > > Basically what is being proposed (for those who need a concrete example) is you'd write this: > > class Collection<T> { > > public function add(T $item) { ... } > > } > $c = new Collection<Product>(). > > And linters would recognize that, and it would parse, but at runtime it would compile to: > > class Collection { > > public function add(mixed $item) { ... } > > } > $c = new Collection(). > > > And that's the opcodes that would be saved. (I'm hand-waving a lot here.) > > I would be on board with this, although I would ask if it's possible to provide some compile time checks; I'm thinking LSP validation, which is still relevant for inheritance with generics. Beyond that, leave the runtime to not enforce it. (I have no idea how viable that is, but Sara seemed to think it would work so it probably
I agree that's an approach worth looking into. I didn't want to discuss specifics yet, and first get a feel of what people think of the general idea, but it's good that you're mentioning this, and I'm glad others are open to the idea.
> > Another possibility could be to desugar that original to: > > class Collection { > public $__T; > > public function add(mixed $item) { > if (!$item instanceof $this->__T) { > throw new TypeError(); > } > } > } > $c = new Collection(). > $c->__T == 'Product'; > > Which is essentially what you can do in userspace today, but automated. I don't know how viable that is, but it's another thing to consider. > > --Larry Garfield > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: https://www.php.net/unsub.php >
Kind regards Brent [1] https://en.wikipedia.org/wiki/Source-to-source_compiler

Matthew Brown

5 years ago
> On Sep 17, 2020, at 1:28 PM, Brent Roose <brendt@stitcher.io> wrote: > > But I don't want to get stuck on phrasing, if elidiing is the right term as Larry suggests, let's go with it!
No, the correct term is “type erasure”: https://en.m.wikipedia.org/wiki/Type_erasure Its opposite is called reification: https://en.m.wikipedia.org/wiki/Reification_(computer_science)

Unnamed Person

5 years ago
On Thu, Sep 17, 2020 at 1:00 PM Matthew Brown <matthewmatthew@gmail.com> wrote:
> > On Sep 17, 2020, at 1:28 PM, Brent Roose <brendt@stitcher.io> wrote: > > But I don't want to get stuck on phrasing, if elidiing is the right term as Larry suggests, let's go with it! > > No, the correct term is “type erasure”: > https://en.m.wikipedia.org/wiki/Type_erasure
"type erasure" with respect to generics means that they are checked at compile-time and erased at run-time. For instance, Java does generics through type erasure, and you had better believe those get checked at compile-time! This isn't what is being proposed, at least not how I understood it. Really these are just "hints", but historically in PHP the term "type hints" meant actually type checks. In summary: - Checked at run-time (and probably also compile-time when possible): reified generics. - Checked only at compile-time and thrown away at run-time: erased generics. - Not checked at all: ____?. Maybe we can call them "ignored generics"?

Matthew Brown

5 years ago
> On Sep 17, 2020, at 3:25 PM, Levi Morrison <levi.morrison@datadoghq.com> wrote: > >  >> >> On Thu, Sep 17, 2020 at 1:00 PM Matthew Brown <matthewmatthew@gmail.com> wrote: >>>> On Sep 17, 2020, at 1:28 PM, Brent Roose <brendt@stitcher.io> wrote: >>> But I don't want to get stuck on phrasing, if elidiing is the right term as Larry suggests, let's go with it! >> >> No, the correct term is “type erasure”: >> https://en.m.wikipedia.org/wiki/Type_erasure > > "type erasure" with respect to generics means that they are checked at > compile-time and erased at run-time. For instance, Java does generics > through type erasure, and you had better believe those get checked at > compile-time! This isn't what is being proposed, at least not how I > understood it. > > Really these are just "hints", but historically in PHP the term "type > hints" meant actually type checks. > > In summary: > - Checked at run-time (and probably also compile-time when possible): > reified generics. > - Checked only at compile-time and thrown away at run-time: erased generics. > - Not checked at all: ____?. Maybe we can call them "ignored generics"?
Seems like Python calls this same exact behaviour type erasure: https://www.python.org/dev/peps/pep-0484/#instantiating-generic-classes-and-type-erasure The term term type erasure is also used in documentation for an unofficial Python type checker: https://mypy.readthedocs.io/en/stable/generics.html Looks like Hack also uses the term in this context: https://docs.hhvm.com/hack/generics/type-erasure (Hack’s compiler doesn’t perform any generic type-checking)

Rowan Collins

5 years ago
On Thu, 17 Sep 2020 at 20:25, Levi Morrison via internals < internals@lists.php.net> wrote:
> > In summary: > - Checked at run-time (and probably also compile-time when possible): > reified generics. > - Checked only at compile-time and thrown away at run-time: erased > generics. > - Not checked at all: ____?. Maybe we can call them "ignored generics"? >
I think this is making the common assumption that "official" static analysis and compilation are necessarily the same process, but the compiler itself only needs to "understand" enough to choose a target representation; running other analysis at the same time is just a choice for programmer convenience. My understanding is that in reality there are three stages: 1) Static analysis: checks may be made whether erasing or reifying, because erasure hasn't happened yet. 2) Compilation: erased generics are converted to non-generic types; reified generics are compiled to a representation available at run-time 3) Run-time: reified generics have run-time behaviour; erased generics aren't ignored, they simply don't exist because the compiler already erased them In PHP, we currently have a compile step, run on-demand, but do not have an official static analysis step. I have said before that in order to make the type system richer without sacrificing run-time performance, we need to consider having such an official tool. That could be run automatically alongside the existing on-demand compilation; it could be entirely manual as a separate tool; or it could be combined with a new, manually executed compiler or transpiler. I believe there is precedent for all of these approaches. Regards,
-- Rowan Tommins [IMSoP]

Ilija Tovilo

5 years ago
Hi Brent
> Today I'd like to hear your thoughts on what might be a controversial topic, though I think it's worth having this discussion. I want to make the case for adding generic syntax, without actually enforing any additional type checks at runtime. Please hear me out.
I think having the option to omit runtime type checks could be valuable even for non-generic code. If your own internal classes all use static analysis the chance of passing a wrong type is pretty low. This would probably have to be on a per class or even per function basis as you can't reason about other people's code. I do think we need some form of generic runtime check. If we don't, how generics work is completely open to interpretation. Static analysis tools copy the behavior of PHP. If PHP has no behavior in this regard tools will deviate and generics won't have any real meaning at all. There are many nuances to generics (ask Matthew who wrote Psalm), we shouldn't underestimate that. With a hybrid approach we could also do full type checking in testing environments and disable it in more performance critical situations with more confidence. Ilija

Brent

5 years ago
Hi Ilija
> On 17 Sep 2020, at 19:01, Ilija Tovilo <tovilo.ilija@gmail.com> wrote: > > Hi Brent > >> Today I'd like to hear your thoughts on what might be a controversial topic, though I think it's worth having this discussion. I want to make the case for adding generic syntax, without actually enforing any additional type checks at runtime. Please hear me out. > > I think having the option to omit runtime type checks could be > valuable even for non-generic code. If your own internal classes all > use static analysis the chance of passing a wrong type is pretty low. > This would probably have to be on a per class or even per function > basis as you can't reason about other people's code.
I agree, if there's enough interest in this, we might need to consider the scope of one or several RFCs.
> > I do think we need some form of generic runtime check. If we don't, > how generics work is completely open to interpretation. Static > analysis tools copy the behavior of PHP. If PHP has no behavior in > this regard tools will deviate and generics won't have any real > meaning at all. There are many nuances to generics (ask Matthew who > wrote Psalm), we shouldn't underestimate that. With a hybrid approach > we could also do full type checking in testing environments and > disable it in more performance critical situations with more > confidence.
Larry also shared this concern, and I agree that it's important to at least have an official spec from the PHP project. Whether part of it should be enforced at runtime is up for debate, but it's definitely important.
> > Ilija > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: https://www.php.net/unsub.php >
Kind regards Brent

Matthew Brown

5 years ago
I think the addition of runtime-erased generics would be a good thing for the language. I don’t think PHP should perform any sort of checking of generic parameters at compile time. No other equivalent interpreter performs those compile-time checks, and it’s also *very* complex — since adding support for generics in Psalm I’ve spent the ensuing two years fixing many bugs and filling various holes that have cropped up. Hack, introduced in 2014 and written by bunch of people far smarter than me, had at least one hole in its type-checker’s generics handling that was only patched last year. PHP could develop a separate official type-checker to be used alongside its interpreter, similar to Hack’s (hh_check) or TypeScript’s (typescript-checker), but that’s an entirely separate conversation. ——— If this proposal happens, the documentation should make it *super* clear that all generic types are erased at runtime. Erased generics would not support code like this: ``` function filter<T of object>(array<int, object> $arr) : array<int, T> { $new = []; foreach ($arr as $a) { if ($a instanceof T) { $new[] = $a; } } } ``` Instead PHP would need to support a type classname<T> (or similar) to allow passing generic params explicitly. ``` function filter<T of object>(array<int, object> $arr, classname<T> $t_class) : array<int, T> { $new = []; foreach ($arr as $a) { if ($a instanceof $t_class) { $new[] = $a; } } } ``` The implementation would also need a way of denoting covariant (pretty common) and contravariant (very rare) generic parameters, like Hack does.

Brent

5 years ago
Hi all To make myself clear, since I realise I used a confusing term in the subject of my mail. What I'm proposing is runtime-erased generics, the way Matthew describes them: PHP completely ignores them during runtime, and wouldn't do anything with them during compile time. It's up to third party tooling to do type checks. And I just want to point out to people who haven't used any static analysers in PHP before: they are very_ insightful in knowning what your code does, without running it. I don't think there's reason to fear to not checking these types at runtime, since static analysis truly takes care of them. Thanks Matthew for pitching in, I believe the opinion of the ones building and maintaining the current static analysis toolset in PHP is essential in this discussion. Kind regards Brent