Hi,
I have started a project to move the parsing off-strings to the parser and
the grammar. Appart
from some maintenance improvements (we can drop a considerable amount of
hand-written code),
there are some interesting things we **could** (emphasis on could) get out
of this and I wanted
to discuss what people think about them.
* The parser will likely have "\n" characters and backslashes in f-strings
expressions, which currently is impossible:
>>> f"blah blah {'\n'} blah"
File "<stdin>", line 1
f"blah blah {'\n'} blah"
^
SyntaxError: f-string expression part cannot include a backslash
* The parser will allow nesting quote characters. This means that we
**could** allow reusing the same quote type in nested expressions
like this:
f"some text { my_dict["string1"] } more text"
* The parser will naturally allow more control over error messages and AST
positions.
* The **grammar** of the f-string will be fully specified without
ambiguity. Currently, the "grammar" that we have in the docs
(
https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals)
is not really formal grammar because
not only is mixing lexing details with grammar details (the definition of "
literal_char") but also is not compatible with the current python
lexing schema (for instance, it recognizes "{{" as its own token, which the
language doesn't allow because something like "{{a:b}:c}"
is tokenized as "{", "{", "a" ... not as "{{", "a". Adding a formal grammar
could help syntax highlighters, IDEs, parsers and other tools
to make sure they properly recognize everything that there is.
There may be some other advantages that we have not explored still.
The work is at a point where the main idea works (all the grammar is
already there and working), but we need to make sure that all existing
errors and specifics are properly ported to the new code, which is a
considerable amount of work still so I wanted to make sure we are on the
same page before we decide to invest more time on this (Batuhan is helping
me with this and Lyssandros will likely join us). We are doing
this work in this branch:
https://github.com/we-like-parsers/cpython/blob/fstring-grammar
Tell me what you think.
P.S. If you are interested to help with this project, please reach out to
me. If we decide to go ahead we can use your help! :)
Regards from cloudy London,
Pablo Galindo Salgado
I have started a project to move the parsing off-strings to the parser and
the grammar. Appart
from some maintenance improvements (we can drop a considerable amount of
hand-written code),
there are some interesting things we **could** (emphasis on could) get out
of this and I wanted
to discuss what people think about them.
* The parser will likely have "\n" characters and backslashes in f-strings
expressions, which currently is impossible:
>>> f"blah blah {'\n'} blah"
File "<stdin>", line 1
f"blah blah {'\n'} blah"
^
SyntaxError: f-string expression part cannot include a backslash
* The parser will allow nesting quote characters. This means that we
**could** allow reusing the same quote type in nested expressions
like this:
f"some text { my_dict["string1"] } more text"
* The parser will naturally allow more control over error messages and AST
positions.
* The **grammar** of the f-string will be fully specified without
ambiguity. Currently, the "grammar" that we have in the docs
(
https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals)
is not really formal grammar because
not only is mixing lexing details with grammar details (the definition of "
literal_char") but also is not compatible with the current python
lexing schema (for instance, it recognizes "{{" as its own token, which the
language doesn't allow because something like "{{a:b}:c}"
is tokenized as "{", "{", "a" ... not as "{{", "a". Adding a formal grammar
could help syntax highlighters, IDEs, parsers and other tools
to make sure they properly recognize everything that there is.
There may be some other advantages that we have not explored still.
The work is at a point where the main idea works (all the grammar is
already there and working), but we need to make sure that all existing
errors and specifics are properly ported to the new code, which is a
considerable amount of work still so I wanted to make sure we are on the
same page before we decide to invest more time on this (Batuhan is helping
me with this and Lyssandros will likely join us). We are doing
this work in this branch:
https://github.com/we-like-parsers/cpython/blob/fstring-grammar
Tell me what you think.
P.S. If you are interested to help with this project, please reach out to
me. If we decide to go ahead we can use your help! :)
Regards from cloudy London,
Pablo Galindo Salgado