Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » generating code from AST »

generating code from AST

php.internals

Adam Baratz

9 years ago
I'm exploring how to automate some basic kinds of refactor operations. One approach I'm considering: - Generate an AST - Rearrange it as needed - Turn it back into userland code Is this something anyone's explored? Thanks, Adam

Sara Golemon

9 years ago
On Mon, Mar 6, 2017 at 3:37 PM, Adam Baratz <adambaratz@php.net> wrote:
> I'm exploring how to automate some basic kinds of refactor operations. One > approach I'm considering: > > - Generate an AST > - Rearrange it as needed > - Turn it back into userland code > > Is this something anyone's explored? >
The engine's internals actually have an ast export method which does this alread (it's used by the assert message generator). astkit makes use of this in an extension to do much of what you describe: https://github.com/sgolemon/astkit -Sara

Sara Golemon

9 years ago
On Mon, Mar 6, 2017 at 3:40 PM, Sara Golemon <pollita@php.net> wrote:
> On Mon, Mar 6, 2017 at 3:37 PM, Adam Baratz <adambaratz@php.net> wrote: >> I'm exploring how to automate some basic kinds of refactor operations. One >> approach I'm considering: >> >> - Generate an AST >> - Rearrange it as needed >> - Turn it back into userland code >> >> Is this something anyone's explored? >> > The engine's internals actually have an ast export method which does > this alread (it's used by the assert message generator). astkit > makes use of this in an extension to do much of what you describe: > https://github.com/sgolemon/astkit >
Hrmmm, I seem to have not updated the README.md, but it is in there: https://github.com/sgolemon/astkit/blob/master/astkit-node.c#L204-L214 -Sara

Sebastian Bergmann

9 years ago
Am 06.03.2017 um 22:40 schrieb Sara Golemon:
> https://github.com/sgolemon/astkit
There's also https://github.com/nikic/php-ast. But wouldn't it be great if PHP 7.2 shipped with a built-in, enabled-by-default extension for AST operations?

David Walker

9 years ago
On Mon, Mar 6, 2017 at 22:55 Sebastian Bergmann <sebastian@php.net> wrote:
> Am 06.03.2017 um 22:40 schrieb Sara Golemon: > > https://github.com/sgolemon/astkit > > There's also https://github.com/nikic/php-ast. > > But wouldn't it be great if PHP 7.2 shipped with a built-in, > enabled-by-default extension for AST operations?
I'd be happy if this were to go forward. When/if there are changes to AST, having a tool like these in core would require them to be updated at the same time. Would this concept also want to extend to a VLD-esque extension as well for the same reasons? I'd be receptive for abilities to expose the core internal ast/ops to userland more easily as a core extension However I'm sure there are valid reasons why VLD (or similar) have not been included; and would wonder if those would also end up applying to the AST
-- Dave

Sebastian Bergmann

9 years ago
Am 07.03.2017 um 07:01 schrieb David Walker:
> When/if there are changes to AST, having a tool like these in core would require them to be updated at the same time.
Exactly.
> Would this concept also want to extend to a VLD-esque extension as well for > the same reasons? I'd be receptive for abilities to expose the core > internal ast/ops to userland more easily as a core extension
Having a bytecode disassembler in core would also be nice, yes.

Derick Rethans

9 years ago
On Tue, 7 Mar 2017, Sebastian Bergmann wrote:
> Am 07.03.2017 um 07:01 schrieb David Walker: > > > When/if there are changes to AST, having a tool like these in core > > would require them to be updated at the same time. > > Exactly. > > > Would this concept also want to extend to a VLD-esque extension as well for > > the same reasons? I'd be receptive for abilities to expose the core > > internal ast/ops to userland more easily as a core extension > > Having a bytecode disassembler in core would also be nice, yes.
Because installing an extension is too hard? It sounds like reinventing the wheel. cheers, Derick

Rowan Collins

9 years ago
On 07/03/2017 10:33, Derick Rethans wrote:
>> Having a bytecode disassembler in core would also be nice, yes. > Because installing an extension is too hard? It sounds like reinventing > the wheel.
Not necessarily reinventing it, just pre-attaching it to the carriage ;) By which I mean, there's no reason "bundling a VLD-esque extension" couldn't mean "bundling VLD". But as Dave said:
> However I'm sure there are valid reasons why VLD (or similar) have not been > included; and would wonder if those would also end up applying to the AST
Basically, we're back to the age-old discussion of exactly what are the pros and cons of "bundling" an extension, given the way PHP is packaged by vendors in practice. Regards,
-- Rowan Collins [IMSoP]

Sebastian Bergmann

9 years ago
Am 07.03.2017 um 11:33 schrieb Derick Rethans:
> Because installing an extension is too hard?
No. To ensure that userland functionality that is based on compiler internals (token stream, abstract syntax tree, bytecode) does not fall out of sync with the compiler. We already have an extension that is built-in and enabled-by-default for the token stream. Why not have one for the abstract syntax tree and bytecode representations?

Nikita

9 years ago
> On 8 Mar 2017, at 08:49, Sebastian Bergmann <sebastian@php.net> wrote: > > Am 07.03.2017 um 11:33 schrieb Derick Rethans: >> Because installing an extension is too hard? > > No. To ensure that userland functionality that is based on compiler > internals (token stream, abstract syntax tree, bytecode) does not fall out > of sync with the compiler. > > We already have an extension that is built-in and enabled-by-default for > the token stream. Why not have one for the abstract syntax tree and > bytecode representations?
To be honest, that’s exactly what we might want to avoid. Last time someone wanted to do actual improvements to our lexing they were stopped because changing tokens would mean breaking code that relies on `token_get_all()` which is a core extension user land function. It might be though that it’s only the problem with lexing and it should not be a problem for AST because it is, ahem, abstract.