Mailing List Archive

We had a similar thing at Dropbox, where `# coding: pyxl` would enable a
preprocessor that allowed HTML embedded in the Python code. It translated
this to function calls and string literals.

There were however several drawbacks:

- Installing the codec is a bit tricky, and if you don't have it the code
can't run
- Other tooling won't know about the new syntax (someone went through
heroic efforts to write a PyCharm extension for pyxl but it got stale very
quickly)
- The codec slows down the import process
- Keeping line numbers accurate in the codec is painful
- Occasionally, when debugging mysteries, you end up needing to see the
source code after preprocessing, which requires yet another custom tool

In the end the project was withdrawn and we switched to more mainstream
templating solutions.

I suspect that many of the issues with pyxl would also plague using this
approach as a way to customize typing syntax, and I don't think it would be
an improvement over the status quo.




On Mon, Apr 19, 2021 at 11:26 AM Luciano Ramalho <luciano@ramalho.org>
wrote:

> How about leveraging the `# coding=<encoding name>` hook that exists
> since 2001 to enable the alternative syntax some are advocating for
> type hints?
>
> PEP 263—Defining Python Source Code Encodings
> https://www.python.org/dev/peps/pep-0263/
>
> I've seen experiments in the wild using that to support syntax
> extensions to Python. Just for fun, years ago I wrote a POC for
> Sucuri—a Python dialect with the keywords in Portuguese. The
> pre-processor simply replaced the Portuguese keywords with the English
> ones.
>
> Cheers,
>
> Luciano
>
>
> --
> Luciano Ramalho
> | Author of Fluent Python (O'Reilly, 2015)
> | http://shop.oreilly.com/product/0636920032519.do
> | Technical Principal at ThoughtWorks
> | Twitter: @ramalhoorg
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/XRLAOOIJZRPDT2AW6LW4UVBEJD5NKLV5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On 4/19/2021 12:44 PM, Guido van Rossum wrote:
> We had a similar thing at Dropbox, where `# coding: pyxl` would enable
> a preprocessor that allowed HTML embedded in the Python code. It
> translated this to function calls and string literals.
>
> There were however several drawbacks:
>
> - Installing the codec is a bit tricky, and if you don't have it the
> code can't run
> - Other tooling won't know about the new syntax (someone went through
> heroic efforts to write a PyCharm extension for pyxl but it got stale
> very quickly)
> - The codec slows down the import process
> - Keeping line numbers accurate in the codec is painful
> - Occasionally, when debugging mysteries, you end up needing to see
> the source code after preprocessing, which requires yet another custom
> tool
>
> In the end the project was withdrawn and we switched to more
> mainstream templating solutions.
>
> I suspect that many of the issues with pyxl would also plague using
> this approach as a way to customize typing syntax, and I don't think
> it would be an improvement over the status quo.


How does this "similar thing" compare to the recently announced imphook
module?


_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QYSPZ2NNJQNEKXD2F72AFQLCX7XGO4LD/
Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Apr 19, 2021 at 8:16 PM Glenn Linderman <v+python@g.nevcal.com>
wrote:

> On 4/19/2021 12:44 PM, Guido van Rossum wrote:
> > We had a similar thing at Dropbox, where `# coding: pyxl` would enable
> > a preprocessor that allowed HTML embedded in the Python code. It
> > translated this to function calls and string literals.
> >
> > There were however several drawbacks:
> >
> > - Installing the codec is a bit tricky, and if you don't have it the
> > code can't run
> > - Other tooling won't know about the new syntax (someone went through
> > heroic efforts to write a PyCharm extension for pyxl but it got stale
> > very quickly)
> > - The codec slows down the import process
> > - Keeping line numbers accurate in the codec is painful
> > - Occasionally, when debugging mysteries, you end up needing to see
> > the source code after preprocessing, which requires yet another custom
> > tool
> >
> > In the end the project was withdrawn and we switched to more
> > mainstream templating solutions.
> >
> > I suspect that many of the issues with pyxl would also plague using
> > this approach as a way to customize typing syntax, and I don't think
> > it would be an improvement over the status quo.
>
>
> How does this "similar thing" compare to the recently announced imphook
> module?
>

For one, pyxl is a better name. :-)

Seriously, as long as the purpose is to allow using a different grammar
(for part of the file) that is then transpiled to proper Python, all but
the first issue apply exactly the same.

Getting the hook to run (my first bullet) actually sounds *more*
complicated, because you have to include code in your `__main__` module
that installs the hook(s).

But I have zero experience with imphook, I just skimmed its PyPI home page,
which has an example that calls
```
imphook.add_import_hook(hook, (".conf",))
```

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

I proposed PEP 511 "API for code transformers" for Python 3.6 (in
2016) and it was rejected:
https://www.python.org/dev/peps/pep-0511/#rejection-notice

"""
This PEP was rejected by its author.

This PEP was seen as blessing new Python-like programming languages
which are close but incompatible with the regular Python language. It
was decided to not promote syntaxes incompatible with Python.

This PEP was also seen as a nice tool to experiment new Python
features, but it is already possible to experiment them without the
PEP, only with importlib hooks. If a feature becomes useful, it should
be directly part of Python, instead of depending on an third party
Python module.

Finally, this PEP was driven was the FAT Python optimization project
which was abandoned in 2016, since it was not possible to show any
significant speedup, but also because of the lack of time to implement
the most advanced and complex optimizations.
"""

IMO the most important part of the PEP 511 was to change the pyc
filename depending on the code transformer used to modify the code.

Victor

On Mon, Apr 19, 2021 at 8:21 PM Luciano Ramalho <luciano@ramalho.org> wrote:
>
> How about leveraging the `# coding=<encoding name>` hook that exists
> since 2001 to enable the alternative syntax some are advocating for
> type hints?
>
> PEP 263—Defining Python Source Code Encodings
> https://www.python.org/dev/peps/pep-0263/
>
> I've seen experiments in the wild using that to support syntax
> extensions to Python. Just for fun, years ago I wrote a POC for
> Sucuri—a Python dialect with the keywords in Portuguese. The
> pre-processor simply replaced the Portuguese keywords with the English
> ones.
>
> Cheers,
>
> Luciano
>
>
> --
> Luciano Ramalho
> | Author of Fluent Python (O'Reilly, 2015)
> | http://shop.oreilly.com/product/0636920032519.do
> | Technical Principal at ThoughtWorks
> | Twitter: @ramalhoorg
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XRLAOOIJZRPDT2AW6LW4UVBEJD5NKLV5/
> Code of Conduct: http://python.org/psf/codeofconduct/



--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7HTSTMQK53APVR2OLUTDSYBZLTN2OQSM/
Code of Conduct: http://python.org/psf/codeofconduct/

Hello,

On Mon, 19 Apr 2021 20:23:08 -0700
Guido van Rossum <guido@python.org> wrote:

[]

> > How does this "similar thing" compare to the recently announced
> > imphook module?
> >
>
> For one, pyxl is a better name. :-)

I would humbly disagree ;-).

> Seriously, as long as the purpose is to allow using a different
> grammar (for part of the file) that is then transpiled to proper
> Python, all but the first issue apply exactly the same.

That's one of the biggest differences - imphook
(https://pypi.org/project/imphook/) written with the idea that macro
capabilities have always been first-class citizen of the Python
ecosystem (well, maybe 25 years of its 30 years).

But it's also written with the full understanding and acknowledgment
that there's a disconnect between advanced Python users, who have been
using Python macro capabilities for decades, and mainstream community.
For the latter, the use of these facilities was always complicated and
esoteric.

And given the attitude of some core CPython developers (examples:
self-policing like in PEP511 or telling pyxl anecdotes), there's
suspicion that the state of complication and confusion is held on the
purpose (of course, with the good intentions of saving users from even
more confusion).

> Getting the hook to run (my first bullet) actually sounds *more*
> complicated, because you have to include code in your `__main__`
> module that installs the hook(s).

Right, importing a module is a complicated task for Python users, or
should be presented as such.

Well, then there's also a runner which allows to not patch your code to
use macro facilities:

python3 -m imphook -i mod_conf example_conf.py

But that's command line, so of course it's still complicated. Well,
maybe not that much, and people who were able to deal with "python2",
"python3", "python3.5", "python3.6", "py", etc. can deal with that too.

> But I have zero experience with imphook, I just skimmed its PyPI home
> page, which has an example that calls
> ```
> imphook.add_import_hook(hook, (".conf",))
> ```

That's the code which needs to be run by "macro" implementations, not
by users of a particular "macro".


--
Best regards,
Paul mailto:pmiscml@gmail.com
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VECHGF5WEZWKRNDBJRIDZYJTHD22LRME/
Code of Conduct: http://python.org/psf/codeofconduct/

Hello,

On Tue, 20 Apr 2021 11:08:56 +0200
Victor Stinner <vstinner@python.org> wrote:

> I proposed PEP 511 "API for code transformers" for Python 3.6 (in
> 2016) and it was rejected:
> https://www.python.org/dev/peps/pep-0511/#rejection-notice

Well, it wasn't rejected, it was self-rejected on the thought-crime
grounds.

It's however a big philosophical question what's worse for Python -
adding questionable re: implementation aspects features (PEP649), NIH
way to handle exceptions (PEP654) (a couple of random examples,
really), - or embrace macro-level extensibility, which is usually a
part of any advanced programming language out there. (And which is
de-facto available for Python either).

[]

> IMO the most important part of the PEP 511 was to change the pyc
> filename depending on the code transformer used to modify the code.

Right, for "production" macro usage, the question of tracking macro
dependencies for precompiled bytecode (and invalidating it if
dependencies changed) is important. PEP 638
(https://www.python.org/dev/peps/pep-0638/) also touches on that, but I
also don't see it to be well thought out/articulated. Instead of adhoc
version/flavor identifiers (which in large-scale will likely lead to
version hell as we know), it literally should be that a .pyc file
can depend on multiple (not one, like now) .py files (or maybe
module names).

[]


--
Best regards,
Paul mailto:pmiscml@gmail.com
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XQ45QDSBEQP45TQKISJI4YFOFQVEQSLO/
Code of Conduct: http://python.org/psf/codeofconduct/