Mailing List Archive: Modified parser does not seem to work. What am I doing wrong?

Modified parser does not seem to work. What am I doing wrong?

Oct 16, 2020, 2:00 PM

Post #1 of 4 (316 views)

Hello,

I am trying to implement PEP-637, and I started modifying the parser
and the grammar, but I don't know what I am missing.

The PR is here

https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-implementation-attempt-1?expand=1

It includes other stuff but the core is that I extended the Subscript
in the asdl to accept the keyword args

| Subscript(expr value, expr slice, keyword* keywords, expr_context ctx)

which seems to work:

>>> ast.parse('a[3]').body[0].value.keywords
[]

I also added a few "productions" (I believe they are called like this,
my compiler theory is very approximated), one of which is now

primary[expr_ty]:
....
| a=primary '[' b=slices c=[',' k=kwargs {k}]']' {
_Py_Subscript(a, b, c, Load, EXTRA) }

I also tried with additional formats like

| a=primary '[' b=slices c=kwargs ']' { _Py_Subscript(a, b, c,
Load, EXTRA) }

or even just

| a=primary '[' c=kwargs ']' { _Py_Subscript(a, NULL, c, Load, EXTRA) }

but I always get a SyntaxError:

>>> ast.parse('a[k=3]').body[0].value.keywords
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/sbo/Work/Projects/stefanoborini/cpython/Lib/ast.py",
line 50, in parse
return compile(source, filename, mode, flags,
File "<unknown>", line 1
a[k=3]
^
SyntaxError: invalid syntax

Note that I always recreated ast parser and pegen.c with 'make
regen-all' and recompiled with make.
I tried to enable debug and print the pegen.c debug, but it's a bit
heavy and I could not immediately spot the issue. I suspect I am
missing something somewhere.

Thanks

--
Kind regards,

Stefano Borini
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RZYDHYZPTPGLBX4VCR6HTYGJ3GL2CWIX/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: Modified parser does not seem to work. What am I doing wrong? [ In reply to ]

pablogsal at gmail

Oct 16, 2020, 3:43 PM

Post #2 of 4 (314 views)

Permalink

Hi Stefano,

One of the problems you have is that the rule for slices has a negative
lookahead for the comma:

slices[expr_ty]:
| a=slice !',' { a }

IIRC the reason that is there is to allow "x[3,]" to be parsed.

Also, to allow "a[k=3]" you need to to create a rule that allows skipping
the "slices" part of the subscript and allow the 2nd argument of
_Py_Subscript to be NULL. For example:

| a=t_primary '[' k=kwargs ']' !t_lookahead { _Py_Subscript(a, NULL, k,
Store, EXTRA) }

Regards,
Pablo

On Fri, 16 Oct 2020 at 22:07, Stefano Borini <stefano.borini@gmail.com>
wrote:

> Hello,
>
> I am trying to implement PEP-637, and I started modifying the parser
> and the grammar, but I don't know what I am missing.
>
> The PR is here
>
>
> https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-implementation-attempt-1?expand=1
>
> It includes other stuff but the core is that I extended the Subscript
> in the asdl to accept the keyword args
>
> | Subscript(expr value, expr slice, keyword* keywords, expr_context ctx)
>
> which seems to work:
>
> >>> ast.parse('a[3]').body[0].value.keywords
> []
>
> I also added a few "productions" (I believe they are called like this,
> my compiler theory is very approximated), one of which is now
>
> primary[expr_ty]:
> ....
> | a=primary '[' b=slices c=[',' k=kwargs {k}]']' {
> _Py_Subscript(a, b, c, Load, EXTRA) }
>
> I also tried with additional formats like
>
> | a=primary '[' b=slices c=kwargs ']' { _Py_Subscript(a, b, c,
> Load, EXTRA) }
>
> or even just
>
> | a=primary '[' c=kwargs ']' { _Py_Subscript(a, NULL, c, Load, EXTRA) }
>
> but I always get a SyntaxError:
>
> >>> ast.parse('a[k=3]').body[0].value.keywords
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/Users/sbo/Work/Projects/stefanoborini/cpython/Lib/ast.py",
> line 50, in parse
> return compile(source, filename, mode, flags,
> File "<unknown>", line 1
> a[k=3]
> ^
> SyntaxError: invalid syntax
>
> Note that I always recreated ast parser and pegen.c with 'make
> regen-all' and recompiled with make.
> I tried to enable debug and print the pegen.c debug, but it's a bit
> heavy and I could not immediately spot the issue. I suspect I am
> missing something somewhere.
>
> Thanks
>
>
> --
> Kind regards,
>
> Stefano Borini
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RZYDHYZPTPGLBX4VCR6HTYGJ3GL2CWIX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

Re: Modified parser does not seem to work. What am I doing wrong? [ In reply to ]

thomas at python

Oct 16, 2020, 4:11 PM

Post #3 of 4 (314 views)

Permalink

On Fri, 16 Oct 2020, 23:04 Stefano Borini, <stefano.borini@gmail.com> wrote:

> Hello,
>
> I am trying to implement PEP-637, and I started modifying the parser
> and the grammar, but I don't know what I am missing.
>
> The PR is here
>
>
> https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-implementation-attempt-1?expand=1
>
> It includes other stuff but the core is that I extended the Subscript
> in the asdl to accept the keyword args
>
> | Subscript(expr value, expr slice, keyword* keywords, expr_context ctx)
>
> which seems to work:
>
> >>> ast.parse('a[3]').body[0].value.keywords
> []
>
> I also added a few "productions" (I believe they are called like this,
> my compiler theory is very approximated), one of which is now
>
> primary[expr_ty]:
> ....
> | a=primary '[' b=slices c=[',' k=kwargs {k}]']' {
> _Py_Subscript(a, b, c, Load, EXTRA) }
>
> I also tried with additional formats like
>
> | a=primary '[' b=slices c=kwargs ']' { _Py_Subscript(a, b, c,
> Load, EXTRA) }
>
> or even just
>
> | a=primary '[' c=kwargs ']' { _Py_Subscript(a, NULL, c, Load, EXTRA) }
>
> but I always get a SyntaxError:
>
> >>> ast.parse('a[k=3]').body[0].value.keywords
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/Users/sbo/Work/Projects/stefanoborini/cpython/Lib/ast.py",
> line 50, in parse
> return compile(source, filename, mode, flags,
> File "<unknown>", line 1
> a[k=3]
> ^
> SyntaxError: invalid syntax
>
> Note that I always recreated ast parser and pegen.c with 'make
> regen-all' and recompiled with make.
> I tried to enable debug and print the pegen.c debug, but it's a bit
> heavy and I could not immediately spot the issue. I suspect I am
> missing something somewhere.
>

Have you manually checked that the generated source in pegen.c contains
your changes? You should be able to see by just looking for calls to
_Py_Subscript, or other code snippets from the modified grammar. Or, you
can introduce an error in the grammar's code snippets and check that you
get the expected error. FWIW, I found I had to explicitly regenerate
pegen.c with 'make regen-pegen'; 'make regen-all' wasn't enough.

> Thanks
>
>
> --
> Kind regards,
>
> Stefano Borini
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RZYDHYZPTPGLBX4VCR6HTYGJ3GL2CWIX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

Re: Modified parser does not seem to work. What am I doing wrong? [ In reply to ]

stefano.borini at gmail

Oct 17, 2020, 10:51 AM

Post #4 of 4 (314 views)

Permalink

That's what I suspected looking at the parser debug in fact. Good to
see that I was on the right track. Thanks! I'll play with it a bit
more tonight.

On Fri, 16 Oct 2020 at 23:43, Pablo Galindo Salgado <pablogsal@gmail.com> wrote:
>
> Hi Stefano,
>
> One of the problems you have is that the rule for slices has a negative lookahead for the comma:
>
> slices[expr_ty]:
> | a=slice !',' { a }
>
> IIRC the reason that is there is to allow "x[3,]" to be parsed.
>
> Also, to allow "a[k=3]" you need to to create a rule that allows skipping the "slices" part of the subscript and allow the 2nd argument of _Py_Subscript to be NULL. For example:
>
> | a=t_primary '[' k=kwargs ']' !t_lookahead { _Py_Subscript(a, NULL, k, Store, EXTRA) }
>
> Regards,
> Pablo
>
> On Fri, 16 Oct 2020 at 22:07, Stefano Borini <stefano.borini@gmail.com> wrote:
>>
>> Hello,
>>
>> I am trying to implement PEP-637, and I started modifying the parser
>> and the grammar, but I don't know what I am missing.
>>
>> The PR is here
>>
>> https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-implementation-attempt-1?expand=1
>>
>> It includes other stuff but the core is that I extended the Subscript
>> in the asdl to accept the keyword args
>>
>> | Subscript(expr value, expr slice, keyword* keywords, expr_context ctx)
>>
>> which seems to work:
>>
>> >>> ast.parse('a[3]').body[0].value.keywords
>> []
>>
>> I also added a few "productions" (I believe they are called like this,
>> my compiler theory is very approximated), one of which is now
>>
>> primary[expr_ty]:
>> ....
>> | a=primary '[' b=slices c=[',' k=kwargs {k}]']' {
>> _Py_Subscript(a, b, c, Load, EXTRA) }
>>
>> I also tried with additional formats like
>>
>> | a=primary '[' b=slices c=kwargs ']' { _Py_Subscript(a, b, c,
>> Load, EXTRA) }
>>
>> or even just
>>
>> | a=primary '[' c=kwargs ']' { _Py_Subscript(a, NULL, c, Load, EXTRA) }
>>
>> but I always get a SyntaxError:
>>
>> >>> ast.parse('a[k=3]').body[0].value.keywords
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File "/Users/sbo/Work/Projects/stefanoborini/cpython/Lib/ast.py",
>> line 50, in parse
>> return compile(source, filename, mode, flags,
>> File "<unknown>", line 1
>> a[k=3]
>> ^
>> SyntaxError: invalid syntax
>>
>> Note that I always recreated ast parser and pegen.c with 'make
>> regen-all' and recompiled with make.
>> I tried to enable debug and print the pegen.c debug, but it's a bit
>> heavy and I could not immediately spot the issue. I suspect I am
>> missing something somewhere.
>>
>> Thanks
>>
>>
>> --
>> Kind regards,
>>
>> Stefano Borini
>> _______________________________________________
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-leave@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RZYDHYZPTPGLBX4VCR6HTYGJ3GL2CWIX/
>> Code of Conduct: http://python.org/psf/codeofconduct/

--
Kind regards,

Stefano Borini
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/266N3PGL4IXGVZB4QNQGR54DP5PHNABP/
Code of Conduct: http://python.org/psf/codeofconduct/