Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » Official Preprocessor »

Official Preprocessor

php.internals

Илья Сомов

3 years ago
Hi Internals The main gist: -------------- I suggest to introduce an official type-checker / static analyser / preprocessor / transpiler for PHP, somewhat similar to TypeScript in the JavaScript world, which will allow to introduce the features otherwise impossible for the PHP world, at the cost of including an additional transpilation step. The reasoning: -------------- You all know there are a lot of problems with the language, especially large gaps with the current type system, which have large community demand to overcome, but still cannot be solved due to the paradigm chosen by PHP. And the paradigm is to be statically typed (to some extent), but dynamically checked language, which limits the number of available options due to performance reasons. The introduction of static types have gradually eliminated the need of using external means of communicating the types (phpdoc @param and @return annotations), replacing them with native language constructs, making the language more expressive and enterprise-ready. However, there are a lot of things still missing, like types of arrays, including lists, hash-maps and structure-like arrays, and of course generic classes and functions. These are still covered nowadays with external means like phpdoc annotations and static analysers / type checkers to validate them. This leaves PHP in the odd mixture of native types and externally validated annotations via comments. Paradigm of a dynamically checked language also leaves out the problem of type-checking before running the code. Even though the types are defined statically, you still have to physically run the code, with all possible execution paths, to catch a type error. To avoid that you still would have to use an external typechecker / static analyser, even if PHP could cover 100% of type demand natively. Python solves this problem nicely I think. It is dynamically typed, but it allows a special syntax for static types and also has a separate static type checker. I think this is really a good approach and it would be great if PHP followed this path for the beginning, but we have what we have already. There were attempts to create preprocessors for PHP (like PHP++ IIRC), which allowed features like generics, but they did not gain enough popularity, weren't baked by large enough companies, didn't make their way to major IDEs support. I believe the only solution is to make an official one. Therefore, I suggest the following thing: 1. Introduce a new #declare directive which will mark the file as requiring pre-processing. 2. Let PHP introduce special features that only work in such "pre-processing required" files, such as typed lists and generics. 3. Implement a static type-checker in PHP that will verify those typed lists and generics are indeed used in a valid way. 4. Make the PHP preprocessor output the "normal" PHP file somewhere, which has list types and generics erased. I think something like that was suggested earlier many times. As an alternative to a new #declare feature + validation + transpilation in "plain old PHP", the Python way of just ignoring statically checked stuff could be applied. This approach though looks odd to me in PHP world, because having some stuff checked in runtime, and some only statically may confuse developers a lot. Basically, this will split PHP in two languages, one being superset of other, but checked statically. I realize this might be much of a paradigm shift, it may require lots of work if accepted (I can only dream of it though), but still I want to share my thoughts with you and propose something for consideration. Regards, Illia / someniatko

Илья Сомов

3 years ago
I'd like to also emphasize that the concept of compilation and static type-checking / analysis is not foreign to the PHP community. Major frameworks like Symfony already have a concept of compiling e.g. a DI container, and static analyzers like Psalm, PHPStan and others are actively used.

Deleu

3 years ago
On Thu, Feb 2, 2023, 11:22 AM someniatko <someniatko@gmail.com> wrote:
> I'd like to also emphasize that the concept of compilation and static > type-checking / analysis is not foreign to the PHP community. Major > frameworks like Symfony already have a concept of compiling e.g. a DI > container, and static analyzers like Psalm, PHPStan and others are > actively used. > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: https://www.php.net/unsub.php
I think the idea overall is pretty solid and could get a few community supporters. But the devil is in the details. 1. Making it official is a trade-off. It's the highest chance of widespread adoption and the heaviest burden on PHP Internals. 2. Who's going to fund it, develop it, maintain it? It's a significant project and a heavy burden. 3. What's the MVP? How do we measure usefulness, adoption and maintenance complexity? 4. Is it an aborttable idea? If things go south and can't be supported anymore, how bad for PHP adopters would it be? 5. How would voting on the spec be like? Complete years-work-hours spec or small incremental functionalities? 6. Is versioning tightly coupled with PHP version or does it have its own version support system targeting multiple php versions at the same time? Overall, I think PHP has been constantly losing popularity in favor of Typescript and a transpiler could create an exciting solution for some of PHP's problems, but it does not seem like a pet project.

Илья Сомов

3 years ago
Hi, These all are valid concerns! I cannot speak about management and cost issues though, as I am not an active PHP dev team member.
> 3. What's the MVP? How do we measure usefulness, adoption and maintenance complexity?
As a shot in the dark, the first step could be implementation of typed lists in my opinion. Do not implement full-blown static analyser nor the type checker for each possible case, just only check the usage of typed lists within the pre-processed files. Assume that whatever comes from the non-preprocessed ones is correct. Make only simple naive assumptions for all internal array_ functions as well as basic array operations. If there is some runtime magic going on with constructing the list, just fallback to infering the type as a simple array. For such cases, allow a special attribute which will tell the type checker this variable / property etc will indeed contain the typed list, like a @var annotation for IDEs / static analysers nowadays. The preprocessor / type checker **should not aim to 100% guarantee the correct types will be in the runtime**: TypeScript does this neither. Rowan has mentioned Dart language in a parallel thread, which makes most of the checks statically, but for some cases the compiler still adds runtime checks. Rowan warned that PHP might have a lot of such cases possible though. But I still think this also might be a good idea for MVP.
> 4. Is it an aborttable idea? If things go south and can't be supported anymore, how bad for PHP adopters would it be?
I don't think any of PHP new features that go live are abortable. This affects not only my suggestion, but also everything what has been done for PHP in past. Once a feature is in the language and released to the public, you cannot revert it. There must be a long multi-year process preceded with a deprecation at least. A concept of experimental features was suggested earlier, but this is completely different topic, so I believe we should not go into that rabbit hole here. Overall, I consider this question orthogonal (or irrelevant if you prefer) to the original topic, let's take as an axiom that there is no way back.
> 5. How would voting on the spec be like? Complete years-work-hours spec or small incremental functionalities?
Small incremental functionalities. Gradually more stuff could be proven statically, more complex cases covered, and while they are not, an attribute / static assertion could be used. Maybe some of such assertions could be allowed to be preserved in runtime, at the cost of some performance penalty, IDK. But I believe we should leave such choice to developers, whether they want the checks in runtime as well.
> 6. Is versioning tightly coupled with PHP version or does it have its own version support system targeting multiple php versions at the same time?
I strongly believe it must go hand in hand with the PHP version, and not as a separate tool, because I see it as a feature of a PHP language itself.
> Overall, I think PHP has been constantly losing popularity in favor of Typescript and a transpiler could create an exciting solution for some of PHP's problems, but it does not seem like a pet project.
Sure, it should be considered seriously and requires some upfront planning. Though, well, PHP itself started as a pet project :D

Rowan Collins

3 years ago
On 02/02/2023 14:19, someniatko wrote:
> I suggest to introduce an official type-checker / static analyser / > preprocessor / transpiler for PHP, somewhat similar to TypeScript in > the JavaScript world, which will allow to introduce the features > otherwise impossible for the PHP world, at the cost of including an > additional transpilation step.
This is something I have thought about a lot over the years, and in some ways, it is very appealing; but it does have some problems. One of the biggest is the concept of "type erasure": in a pre-processed or transpiled language, features like Reflection are crippled by not knowing the full information present in the source code. More importantly, dynamic language features - of which PHP has many - also have no access to the type information. That means that a List<Dog> can sometimes end up with a Cat in it at run-time, because a dynamic call doesn't know that's a problem. I think it's interesting to look around at different approaches to this elsewhere: You mention Python, which I think has probably the *worst* approach: it has official syntax which appears to convey type information, but doesn't actually provide any guarantees if you forget to run an external tool. Outside of that tool, the type information is just another type of comment, just less clearly so than the annotations understood by tools like PhpStan and Psalm in today's PHP. Another approach is TypeScript, which is essentially a complete language in its own right, with JavaScript as a compilation target. As I understand it, you can call JS code from inside a TS project (and vice versa, once the TS is transpiled), but that won't always be seamless, so the two languages will tend to end up with separate ecosystems of libraries and tooling. Implementing a new language also takes time, and ultimately money - in this case, Microsoft's money. A similar project, closer to PHP's heart, is Hack; the main difference being that rather than transpiling to PHP, it initially aimed to compile both languages to a common bytecode, to be executed in a shared VM. However, it turns out that as well as *adding* features on top of PHP, it was necessary for Hack to *remove* features of PHP that weren't compatible with a static typing system, and eventually the interoperability was dropped altogether. It also has money behind it, in this case from Facebook. Dart, which was written completely from scratch (the money this time is Google's), has a type system checked *mostly* at compile-time, but with certain run-time assertions inserted where the compiler can't prove soundness. The approach seems appealing, but PHP would probably end up with a lot of such cases - all the features that the Hack team decided to remove - leading to poor performance and confusing developer experience. I don't have a conclusion to this e-mail, I just thought I'd throw out these thoughts. Regards,
-- Rowan Tommins [IMSoP]

Olle Härstedt

3 years ago
2023-02-02 15:19 GMT+01:00, someniatko <someniatko@gmail.com>:
> Hi Internals > > The main gist: > -------------- > > I suggest to introduce an official type-checker / static analyser / > preprocessor / transpiler for PHP, somewhat similar to TypeScript in > the JavaScript world, which will allow to introduce the features > otherwise impossible for the PHP world, at the cost of including an > additional transpilation step. > > The reasoning: > -------------- > > You all know there are a lot of problems with the language, especially > large gaps with the current type system, which have large community > demand to overcome, but still cannot be solved due to the paradigm > chosen by PHP. And the paradigm is to be statically typed (to some > extent), but dynamically checked language, which limits the number of > available options due to performance reasons. > > The introduction of static types have gradually eliminated the need of > using external means of communicating the types (phpdoc @param and > @return annotations), replacing them with native language constructs, > making the language more expressive and enterprise-ready. However, > there are a lot of things still missing, like types of arrays, > including lists, hash-maps and structure-like arrays, and of course > generic classes and functions. These are still covered nowadays with > external means like phpdoc annotations and static analysers / type > checkers to validate them. This leaves PHP in the odd mixture of > native types and externally validated annotations via comments. > > Paradigm of a dynamically checked language also leaves out the problem > of type-checking before running the code. Even though the types are > defined statically, you still have to physically run the code, with > all possible execution paths, to catch a type error. To avoid that you > still would have to use an external typechecker / static analyser, > even if PHP could cover 100% of type demand natively. > > Python solves this problem nicely I think. It is dynamically typed, > but it allows a special syntax for static types and also has a > separate static type checker. I think this is really a good approach > and it would be great if PHP followed this path for the beginning, but > we have what we have already. > > There were attempts to create preprocessors for PHP (like PHP++ IIRC), > which allowed features like generics, but they did not gain enough > popularity, weren't baked by large enough companies, didn't make their > way to major IDEs support. I believe the only solution is to make an > official one. > > Therefore, I suggest the following thing: > 1. Introduce a new #declare directive which will mark the file as > requiring pre-processing. > 2. Let PHP introduce special features that only work in such > "pre-processing required" files, such as typed lists and generics. > 3. Implement a static type-checker in PHP that will verify those typed > lists and generics are indeed used in a valid way.
No need to waste resources implementing a new one when we already have two very competent already: Psalm and Phpstan? Or why is those not enough for your use-case? "The greatest enemy to a perfect solution is a good enough solution already in place." ;) Olle

Flávio Heleno

3 years ago
On Fri, Feb 3, 2023 at 6:19 AM Olle Härstedt <olleharstedt@gmail.com> wrote:
> 2023-02-02 15:19 GMT+01:00, someniatko <someniatko@gmail.com>: > > Hi Internals > > > > The main gist: > > -------------- > > > > I suggest to introduce an official type-checker / static analyser / > > preprocessor / transpiler for PHP, somewhat similar to TypeScript in > > the JavaScript world, which will allow to introduce the features > > otherwise impossible for the PHP world, at the cost of including an > > additional transpilation step. > > > > The reasoning: > > -------------- > > > > You all know there are a lot of problems with the language, especially > > large gaps with the current type system, which have large community > > demand to overcome, but still cannot be solved due to the paradigm > > chosen by PHP. And the paradigm is to be statically typed (to some > > extent), but dynamically checked language, which limits the number of > > available options due to performance reasons. > > > > The introduction of static types have gradually eliminated the need of > > using external means of communicating the types (phpdoc @param and > > @return annotations), replacing them with native language constructs, > > making the language more expressive and enterprise-ready. However, > > there are a lot of things still missing, like types of arrays, > > including lists, hash-maps and structure-like arrays, and of course > > generic classes and functions. These are still covered nowadays with > > external means like phpdoc annotations and static analysers / type > > checkers to validate them. This leaves PHP in the odd mixture of > > native types and externally validated annotations via comments. > > > > Paradigm of a dynamically checked language also leaves out the problem > > of type-checking before running the code. Even though the types are > > defined statically, you still have to physically run the code, with > > all possible execution paths, to catch a type error. To avoid that you > > still would have to use an external typechecker / static analyser, > > even if PHP could cover 100% of type demand natively. > > > > Python solves this problem nicely I think. It is dynamically typed, > > but it allows a special syntax for static types and also has a > > separate static type checker. I think this is really a good approach > > and it would be great if PHP followed this path for the beginning, but > > we have what we have already. > > > > There were attempts to create preprocessors for PHP (like PHP++ IIRC), > > which allowed features like generics, but they did not gain enough > > popularity, weren't baked by large enough companies, didn't make their > > way to major IDEs support. I believe the only solution is to make an > > official one. > > > > Therefore, I suggest the following thing: > > 1. Introduce a new #declare directive which will mark the file as > > requiring pre-processing. > > 2. Let PHP introduce special features that only work in such > > "pre-processing required" files, such as typed lists and generics. > > 3. Implement a static type-checker in PHP that will verify those typed > > lists and generics are indeed used in a valid way. > > No need to waste resources implementing a new one when we already have > two very competent already: Psalm and Phpstan? Or why is those not > enough for your use-case? > > "The greatest enemy to a perfect solution is a good enough solution > already in place." ;) > > Olle > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: https://www.php.net/unsub.php > >
Why not follow the steps of other projects that have been created independently from the core and, at some point, gained enough support/traction that were incorporated into it?

Olle Härstedt

3 years ago
2023-02-03 17:38 GMT+01:00, Flávio Heleno <flaviohbatista@gmail.com>:
> On Fri, Feb 3, 2023 at 6:19 AM Olle Härstedt <olleharstedt@gmail.com> > wrote: > >> 2023-02-02 15:19 GMT+01:00, someniatko <someniatko@gmail.com>: >> > Hi Internals >> > >> > The main gist: >> > -------------- >> > >> > I suggest to introduce an official type-checker / static analyser / >> > preprocessor / transpiler for PHP, somewhat similar to TypeScript in >> > the JavaScript world, which will allow to introduce the features >> > otherwise impossible for the PHP world, at the cost of including an >> > additional transpilation step. >> > >> > The reasoning: >> > -------------- >> > >> > You all know there are a lot of problems with the language, especially >> > large gaps with the current type system, which have large community >> > demand to overcome, but still cannot be solved due to the paradigm >> > chosen by PHP. And the paradigm is to be statically typed (to some >> > extent), but dynamically checked language, which limits the number of >> > available options due to performance reasons. >> > >> > The introduction of static types have gradually eliminated the need of >> > using external means of communicating the types (phpdoc @param and >> > @return annotations), replacing them with native language constructs, >> > making the language more expressive and enterprise-ready. However, >> > there are a lot of things still missing, like types of arrays, >> > including lists, hash-maps and structure-like arrays, and of course >> > generic classes and functions. These are still covered nowadays with >> > external means like phpdoc annotations and static analysers / type >> > checkers to validate them. This leaves PHP in the odd mixture of >> > native types and externally validated annotations via comments. >> > >> > Paradigm of a dynamically checked language also leaves out the problem >> > of type-checking before running the code. Even though the types are >> > defined statically, you still have to physically run the code, with >> > all possible execution paths, to catch a type error. To avoid that you >> > still would have to use an external typechecker / static analyser, >> > even if PHP could cover 100% of type demand natively. >> > >> > Python solves this problem nicely I think. It is dynamically typed, >> > but it allows a special syntax for static types and also has a >> > separate static type checker. I think this is really a good approach >> > and it would be great if PHP followed this path for the beginning, but >> > we have what we have already. >> > >> > There were attempts to create preprocessors for PHP (like PHP++ IIRC), >> > which allowed features like generics, but they did not gain enough >> > popularity, weren't baked by large enough companies, didn't make their >> > way to major IDEs support. I believe the only solution is to make an >> > official one. >> > >> > Therefore, I suggest the following thing: >> > 1. Introduce a new #declare directive which will mark the file as >> > requiring pre-processing. >> > 2. Let PHP introduce special features that only work in such >> > "pre-processing required" files, such as typed lists and generics. >> > 3. Implement a static type-checker in PHP that will verify those typed >> > lists and generics are indeed used in a valid way. >> >> No need to waste resources implementing a new one when we already have >> two very competent already: Psalm and Phpstan? Or why is those not >> enough for your use-case? >> >> "The greatest enemy to a perfect solution is a good enough solution >> already in place." ;) >> >> Olle >> >> -- >> PHP Internals - PHP Runtime Development Mailing List >> To unsubscribe, visit: https://www.php.net/unsub.php >> >> > Why not follow the steps of other projects that have been created > independently from the core and, > at some point, gained enough support/traction that were incorporated into > it?
You'd have to rewrite Psalm or Phpstan in C if you want it to be part of php-src. Or write a completely new project. Olle

Rowan Collins

3 years ago
On 3 February 2023 18:30:00 GMT, "Olle Härstedt" <olleharstedt@gmail.com> wrote:
>You'd have to rewrite Psalm or Phpstan in C if you want it to be part >of php-src. Or write a completely new project.
I don't think that's necessarily true. There are already tools like run-tests written in PHP and shipped with php-src, PECL is written in PHP, and there have even been suggestions that rewriting some included functions in PHP would be a good long-term goal. The far larger concerns are governance, resources, and release cycles. Regards,
-- Rowan Tommins [IMSoP]

Thomas Krüger

3 years ago
Hi, why not using the existing Typescript programming language and transpile it to PHP? :) There are community examples for doing that: "ts2php" https://www.npmjs.com/package/ts2php Maybe creating an own PHP transpiler for Typescript written in pure PHP (community friendly) would be an alternative of creating a complete new high ievel language developed in C-programming langage. Best regards, Thomas Am 03.02.2023 20:17 schrieb Rowan Tommins:

Rowan Collins

3 years ago
On 3 February 2023 22:48:14 GMT, "Thomas Krüger" <thomas.pakkanen@posteo.de> wrote:
>Hi, > >why not using the existing Typescript programming language and transpile it to PHP? :) > >There are community examples for doing that: >"ts2php" >https://www.npmjs.com/package/ts2php
I can think of a few reasons. Firstly, TypeScript is a language designed with a very specific purpose: to build a statically typed language on top of the existing functionality of JavaScript. Although some of the decisions that follow from that would also aid compatibility with PHP, some would lead to serious mismatches - for example, JavaScript has an entirely different object model, with prototypes rather than classes [1], and methods more closely resembling closures bound to the object. Secondly, designing the language is only half the battle, probably less. It requires expertise to do well, but the bulk of the time is spent *implementing* that design. It can be deceptively easy to implement a proof of concept that can run some carefully chosen sample code, but actually making something production ready is a much bigger effort. It doesn't surprise me at all that the project you linked had some initial beta releases, and then went quiet for several years. Thirdly, if this approach made sense, TypeScript wouldn't need to exist - Microsoft could have written tooling to translate C# to JavaScript instead. The reason languages like TypeScript are created is specifically to bridge existing code and existing knowledge from an existing language. That might mean running in the same run-time as the older language, or just being similar enough to allow a supervised one-off conversion. For PHP, that's the thinking behind Hack (from HHVM), Zephir (from the Phalcon framework), and more recently Ara (https://ara-lang.io/). To succeed, a new language needs a rare combination of committed resources (which generally means commercial backing to pay multiple full time developers), plus a critical mass of users. Zephir lacked both; Hack had backing inside Facebook, but didn't gain traction outside it; I wish Saif luck with Ara, but I'm not holding my breath. [1] Yes, there's now a "class" keyword, but it's just syntactic sugar; the fundamental inheritance model still consists of prototype chains.
-- Rowan Tommins [IMSoP]