Mailing List Archive

Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object
Here's one alternate idea for how to implement the "forward class" syntax.

The entire point of the "forward class" statement is that it creates
the real actual class object.  But what if it wasn't actually the
"real" class object?  What if it was only a proxy for the real object?

In this scenario, the syntax of "forward object" remains the same.
You define the class's bases and metaclass.  But all "forward class"
does is create a simple, lightweight class proxy object.  This object
has a few built-in dunder values, __name__ etc.  It also allows you
to set attributes, so let's assume (for now) it calls
metaclass.__prepare__ and uses the returned "dict-like object" as
the class proxy object __dict__.

"continue class" internally performs all the rest of the
class-creation machinery.  (Everything except __prepare__, as we
already called that.)  The first step is metaclass.__new__, which
returns the real class object.  "continue class" takes that
object and calls a method on the class proxy object that says
"here's your real class object".  From that moment on, the proxy
becomes a pass-through for the "real" class object, and nobody
ever sees a reference to the "real" class object ever again.
Every interaction with the class proxy object is passed through
to the underlying class object.  __getattribute__ calls on the
proxy look up the attribute in the underlying class object.  If
the object returned is a bound method object, it rebinds that
callable with the class proxy instead, so that the "self" passed
in to methods is the proxy object.  Both base_cls.__init_subclass__
and cls.__init__ see the proxy object during class creation.  As far
as Python user code is concerned, the class proxy *is* the class,
in every way, important or not.

The upside: this moves all class object creation code into "continue
class" call.  We don't have to replace __new__ with two new calls.

The downside: a dinky overhead to every interaction with a "forward
class" class object and with instances of a "forward class" class
object.


A huge concern: how does this interact with metaclasses implemented
in C?  If you make a method call on a proxy class object, and that
calls a C function from the metaclass, we'd presumably have to pass
in the "real class object", not the proxy class object.  Which means
references to the real class object could leak out somewhere, and
now we have a real-class-object vs proxy-class-object identity crisis.
Is this a real concern?


A possible concern: what if metaclass.__new__ keeps a reference to
the object it created?  Now we have two objects with an identity
crisis.  I don't know if people ever do that.  Fingers crossed that
they don't.  Or maybe we add a new dunder method:

    @special_cased_staticmethod
    metaclass.__bind_proxy__(metaclass, proxy, cls)

This tells the metaclass "bind cls to this proxy object", so
metaclasses that care can update their database or whatever.
The default implementation uses the appropriate mechanism,
whatever it is.

One additional probably-bad idea: in the case where it's just a
normal "class" statement, and we're not binding it to a proxy,
should we call this?

    metaclass.__bind_proxy__(metaclass, None, cls)

The idea there being "if you register the class objects you create,
do the registration in __bind_proxy__, it's always called, and you'll
always know the canonical object in there".  I'm guessing probably not,
in which case we tell metaclasses that track the class objects we
create "go ahead and track the object you return from __new__, but
be prepared to update your tracking info in case we call __bind_proxy__
on you".


A small but awfully complicated wrinkle here: what do we do if the
metaclass implements __del__?  Obviously, we have to call __del__
with the "real" class object, so it can be destroyed properly.
But __del__ might resurrect that object, which means someone took a
reference to it.



One final note.  Given that, in this scenario, all real class creation
happens in "continue class", we could move the bases and metaclass
declaration down to the "continue class" statement.  The resulting
syntax would look like:

    forward class X

    ...

    continue class X(base1, base2, metaclass=AmazingMeta, rocket="booster")

Is that better? worse? doesn't matter?  I don't have an intuition about
it right now--I can see advantages to both sides, and no obvious
deciding factor.  Certainly this syntax prevents us from calling
__prepare__ so early, so we'd have to use a real dict in the "forward
class" proxy object until we reached continue, then copy the values from
that dict into the "dict-like object", etc.

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OJRA7F7EMRJCIXQPRRKZZ7YMFD2ZKQV2/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object [ In reply to ]
TL;DR
(literally, I will go back and read it now, but after reading the first
paragraphs:
_a proxy_ object yes, then dividing class creation in 2 blocks would
not break things)

/me goes back to text.


On Fri, Apr 22, 2022 at 10:20 PM Larry Hastings <larry@hastings.org> wrote:

>
> Here's one alternate idea for how to implement the "forward class" syntax.
>
> The entire point of the "forward class" statement is that it creates
> the real actual class object. But what if it wasn't actually the
> "real" class object? What if it was only a proxy for the real object?
>
> In this scenario, the syntax of "forward object" remains the same.
> You define the class's bases and metaclass. But all "forward class"
> does is create a simple, lightweight class proxy object. This object
> has a few built-in dunder values, __name__ etc. It also allows you
> to set attributes, so let's assume (for now) it calls
> metaclass.__prepare__ and uses the returned "dict-like object" as
> the class proxy object __dict__.
>
> "continue class" internally performs all the rest of the
> class-creation machinery. (Everything except __prepare__, as we
> already called that.) The first step is metaclass.__new__, which
> returns the real class object. "continue class" takes that
> object and calls a method on the class proxy object that says
> "here's your real class object". From that moment on, the proxy
> becomes a pass-through for the "real" class object, and nobody
> ever sees a reference to the "real" class object ever again.
> Every interaction with the class proxy object is passed through
> to the underlying class object. __getattribute__ calls on the
> proxy look up the attribute in the underlying class object. If
> the object returned is a bound method object, it rebinds that
> callable with the class proxy instead, so that the "self" passed
> in to methods is the proxy object. Both base_cls.__init_subclass__
> and cls.__init__ see the proxy object during class creation. As far
> as Python user code is concerned, the class proxy *is* the class,
> in every way, important or not.
>
> The upside: this moves all class object creation code into "continue
> class" call. We don't have to replace __new__ with two new calls.
>
> The downside: a dinky overhead to every interaction with a "forward
> class" class object and with instances of a "forward class" class
> object.
>
>
> A huge concern: how does this interact with metaclasses implemented
> in C? If you make a method call on a proxy class object, and that
> calls a C function from the metaclass, we'd presumably have to pass
> in the "real class object", not the proxy class object. Which means
> references to the real class object could leak out somewhere, and
> now we have a real-class-object vs proxy-class-object identity crisis.
> Is this a real concern?
>
>
> A possible concern: what if metaclass.__new__ keeps a reference to
> the object it created? Now we have two objects with an identity
> crisis. I don't know if people ever do that. Fingers crossed that
> they don't. Or maybe we add a new dunder method:
>
> @special_cased_staticmethod
> metaclass.__bind_proxy__(metaclass, proxy, cls)
>
> This tells the metaclass "bind cls to this proxy object", so
> metaclasses that care can update their database or whatever.
> The default implementation uses the appropriate mechanism,
> whatever it is.
>
> One additional probably-bad idea: in the case where it's just a
> normal "class" statement, and we're not binding it to a proxy,
> should we call this?
>
> metaclass.__bind_proxy__(metaclass, None, cls)
>
> The idea there being "if you register the class objects you create,
> do the registration in __bind_proxy__, it's always called, and you'll
> always know the canonical object in there". I'm guessing probably not,
> in which case we tell metaclasses that track the class objects we
> create "go ahead and track the object you return from __new__, but
> be prepared to update your tracking info in case we call __bind_proxy__
> on you".
>
>
> A small but awfully complicated wrinkle here: what do we do if the
> metaclass implements __del__? Obviously, we have to call __del__
> with the "real" class object, so it can be destroyed properly.
> But __del__ might resurrect that object, which means someone took a
> reference to it.
>
>
>
> One final note. Given that, in this scenario, all real class creation
> happens in "continue class", we could move the bases and metaclass
> declaration down to the "continue class" statement. The resulting
> syntax would look like:
>
> forward class X
>
> ...
>
> continue class X(base1, base2, metaclass=AmazingMeta,
> rocket="booster")
>
> Is that better? worse? doesn't matter? I don't have an intuition about
> it right now--I can see advantages to both sides, and no obvious
> deciding factor. Certainly this syntax prevents us from calling
> __prepare__ so early, so we'd have to use a real dict in the "forward
> class" proxy object until we reached continue, then copy the values from
> that dict into the "dict-like object", etc.
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OJRA7F7EMRJCIXQPRRKZZ7YMFD2ZKQV2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
Re: Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object [ In reply to ]
So -

good idea on creating the proxy.

But would you really __need__ this badly that the
proxy object would "become" the new class object,
preserving its "id"?

Just name re-binding
(and the static type checkers _knowing_ the name
will be re-assign to the actuall class object later) seems
to be pretty straightforward, and would break too little.

If one assigns the proxy to a different name in the meantime,
and keep a reference to it: "consenting adults" apply: one just
got a somewhat useless stub object to play along with.

Ok, it would be more of a "stub" object than a "proxy" object,


So, if the "proxy" is really needed - there is one type
of existing proxy to classes in Python that is really transparent,
and that is not easily creatable by Python code -
those are the instances of "super()" - the will
answer proper even to isinstance and issubclass calls,
as it goes beyond what is possible by customizing
__getattribute__. So, if a proxy is really needed
instead of a simple name rebind, probably the existing
code for "super' can be reused for them.

As for: "do people keep references for the classes
inside "__new__"? - Yes, all the time.
The "__bind_proxy__" method you describe would
again break compatibility (although in a much
more reasonable way than the "__new__" split.)
But then, the re-binding, if any, could take place
inside "type.__new__", before it returns
the newly created "cls" to the call to it made
inside the custom metaclass.__new__ .
Some code might break when getting back a proxied
class at this point, but, up to that point, a lot of code
could also break with this kind of proxies - "super()" instances work well
but there are, of course, lots of corner cases.
All in all, I think this is overkill for the problem at hand.

As I wrote before: I stand with what Paul Moore wrote:
in an ideal universe annotations for type checking should
be optional. In the real world, there is a lot of pressure,
surging from everywhere, for strict type-checking in
all types of projects, open source or not, and I find
this a very sad state of things. Breaking the language
compatibility and features because it is needed
for "optional" type checking is sad-squared.


On Fri, Apr 22, 2022 at 10:20 PM Larry Hastings <larry@hastings.org> wrote:

>
> Here's one alternate idea for how to implement the "forward class" syntax.
>
> The entire point of the "forward class" statement is that it creates
> the real actual class object. But what if it wasn't actually the
> "real" class object? What if it was only a proxy for the real object?
>
> In this scenario, the syntax of "forward object" remains the same.
> You define the class's bases and metaclass. But all "forward class"
> does is create a simple, lightweight class proxy object. This object
> has a few built-in dunder values, __name__ etc. It also allows you
> to set attributes, so let's assume (for now) it calls
> metaclass.__prepare__ and uses the returned "dict-like object" as
> the class proxy object __dict__.
>
> "continue class" internally performs all the rest of the
> class-creation machinery. (Everything except __prepare__, as we
> already called that.) The first step is metaclass.__new__, which
> returns the real class object. "continue class" takes that
> object and calls a method on the class proxy object that says
> "here's your real class object". From that moment on, the proxy
> becomes a pass-through for the "real" class object, and nobody
> ever sees a reference to the "real" class object ever again.
> Every interaction with the class proxy object is passed through
> to the underlying class object. __getattribute__ calls on the
> proxy look up the attribute in the underlying class object. If
> the object returned is a bound method object, it rebinds that
> callable with the class proxy instead, so that the "self" passed
> in to methods is the proxy object. Both base_cls.__init_subclass__
> and cls.__init__ see the proxy object during class creation. As far
> as Python user code is concerned, the class proxy *is* the class,
> in every way, important or not.
>
> The upside: this moves all class object creation code into "continue
> class" call. We don't have to replace __new__ with two new calls.
>
> The downside: a dinky overhead to every interaction with a "forward
> class" class object and with instances of a "forward class" class
> object.
>
>
> A huge concern: how does this interact with metaclasses implemented
> in C? If you make a method call on a proxy class object, and that
> calls a C function from the metaclass, we'd presumably have to pass
> in the "real class object", not the proxy class object. Which means
> references to the real class object could leak out somewhere, and
> now we have a real-class-object vs proxy-class-object identity crisis.
> Is this a real concern?
>
>
> A possible concern: what if metaclass.__new__ keeps a reference to
> the object it created? Now we have two objects with an identity
> crisis. I don't know if people ever do that. Fingers crossed that
> they don't. Or maybe we add a new dunder method:
>
> @special_cased_staticmethod
> metaclass.__bind_proxy__(metaclass, proxy, cls)
>
> This tells the metaclass "bind cls to this proxy object", so
> metaclasses that care can update their database or whatever.
> The default implementation uses the appropriate mechanism,
> whatever it is.
>
> One additional probably-bad idea: in the case where it's just a
> normal "class" statement, and we're not binding it to a proxy,
> should we call this?
>
> metaclass.__bind_proxy__(metaclass, None, cls)
>
> The idea there being "if you register the class objects you create,
> do the registration in __bind_proxy__, it's always called, and you'll
> always know the canonical object in there". I'm guessing probably not,
> in which case we tell metaclasses that track the class objects we
> create "go ahead and track the object you return from __new__, but
> be prepared to update your tracking info in case we call __bind_proxy__
> on you".
>
>
> A small but awfully complicated wrinkle here: what do we do if the
> metaclass implements __del__? Obviously, we have to call __del__
> with the "real" class object, so it can be destroyed properly.
> But __del__ might resurrect that object, which means someone took a
> reference to it.
>
>
>
> One final note. Given that, in this scenario, all real class creation
> happens in "continue class", we could move the bases and metaclass
> declaration down to the "continue class" statement. The resulting
> syntax would look like:
>
> forward class X
>
> ...
>
> continue class X(base1, base2, metaclass=AmazingMeta,
> rocket="booster")
>
> Is that better? worse? doesn't matter? I don't have an intuition about
> it right now--I can see advantages to both sides, and no obvious
> deciding factor. Certainly this syntax prevents us from calling
> __prepare__ so early, so we'd have to use a real dict in the "forward
> class" proxy object until we reached continue, then copy the values from
> that dict into the "dict-like object", etc.
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OJRA7F7EMRJCIXQPRRKZZ7YMFD2ZKQV2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
Re: Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object [ In reply to ]
On 23. 04. 22 3:15, Larry Hastings wrote:
>
> Here's one alternate idea for how to implement the "forward class" syntax.
>
> The entire point of the "forward class" statement is that it creates
> the real actual class object.  But what if it wasn't actually the
> "real" class object?  What if it was only a proxy for the real object?
>
> In this scenario, the syntax of "forward object" remains the same.
> You define the class's bases and metaclass.  But all "forward class"
> does is create a simple, lightweight class proxy object.  This object
> has a few built-in dunder values, __name__ etc.  It also allows you
> to set attributes, so let's assume (for now) it calls
> metaclass.__prepare__ and uses the returned "dict-like object" as
> the class proxy object __dict__.
>
> "continue class" internally performs all the rest of the
> class-creation machinery.  (Everything except __prepare__, as we
> already called that.)  The first step is metaclass.__new__, which
> returns the real class object.  "continue class" takes that
> object and calls a method on the class proxy object that says
> "here's your real class object".  From that moment on, the proxy
> becomes a pass-through for the "real" class object, and nobody
> ever sees a reference to the "real" class object ever again.
> Every interaction with the class proxy object is passed through
> to the underlying class object.  __getattribute__ calls on the
> proxy look up the attribute in the underlying class object.  If
> the object returned is a bound method object, it rebinds that
> callable with the class proxy instead, so that the "self" passed
> in to methods is the proxy object.  Both base_cls.__init_subclass__
> and cls.__init__ see the proxy object during class creation.  As far
> as Python user code is concerned, the class proxy *is* the class,
> in every way, important or not.

Sadly, I think that if you try to implement this you'll discover a
fractal of little issues, each requiring another hack to solve.
(But it seems the “main” forward/continue class proposal suffered a
similar fate, hasn't it?)

I can see lots of possible issues in interaction with C code (which
Python user code would call).

What would the __class__ attribute hold?
What would C's ob_type be?
What would be in the MROs?
How would Exception subclasses work? (taking `except` as an example of
something that uses the real inheritance chain rather than
__isinstance__/__getattribute__ magic)



> The upside: this moves all class object creation code into "continue
> class" call.  We don't have to replace __new__ with two new calls.
>
> The downside: a dinky overhead to every interaction with a "forward
> class" class object and with instances of a "forward class" class
> object.
>
>
> A huge concern: how does this interact with metaclasses implemented
> in C?  If you make a method call on a proxy class object, and that
> calls a C function from the metaclass, we'd presumably have to pass
> in the "real class object", not the proxy class object.  Which means
> references to the real class object could leak out somewhere, and
> now we have a real-class-object vs proxy-class-object identity crisis.
> Is this a real concern?
>
>
> A possible concern: what if metaclass.__new__ keeps a reference to
> the object it created?  Now we have two objects with an identity
> crisis.  I don't know if people ever do that.  Fingers crossed that
> they don't.  Or maybe we add a new dunder method:
>
>     @special_cased_staticmethod
>     metaclass.__bind_proxy__(metaclass, proxy, cls)
>
> This tells the metaclass "bind cls to this proxy object", so
> metaclasses that care can update their database or whatever.
> The default implementation uses the appropriate mechanism,
> whatever it is.
>
> One additional probably-bad idea: in the case where it's just a
> normal "class" statement, and we're not binding it to a proxy,
> should we call this?
>
>     metaclass.__bind_proxy__(metaclass, None, cls)
>
> The idea there being "if you register the class objects you create,
> do the registration in __bind_proxy__, it's always called, and you'll
> always know the canonical object in there".  I'm guessing probably not,
> in which case we tell metaclasses that track the class objects we
> create "go ahead and track the object you return from __new__, but
> be prepared to update your tracking info in case we call __bind_proxy__
> on you".
>
>
> A small but awfully complicated wrinkle here: what do we do if the
> metaclass implements __del__?  Obviously, we have to call __del__
> with the "real" class object, so it can be destroyed properly.
> But __del__ might resurrect that object, which means someone took a
> reference to it.
>
>
>
> One final note.  Given that, in this scenario, all real class creation
> happens in "continue class", we could move the bases and metaclass
> declaration down to the "continue class" statement.  The resulting
> syntax would look like:
>
>     forward class X
>
>     ...
>
>     continue class X(base1, base2, metaclass=AmazingMeta,
> rocket="booster")
>
> Is that better? worse? doesn't matter?  I don't have an intuition about
> it right now--I can see advantages to both sides, and no obvious
> deciding factor.  Certainly this syntax prevents us from calling
> __prepare__ so early, so we'd have to use a real dict in the "forward
> class" proxy object until we reached continue, then copy the values from
> that dict into the "dict-like object", etc.
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OJRA7F7EMRJCIXQPRRKZZ7YMFD2ZKQV2/
>
> Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AHEY5L46ZCWLJOYFQ2Y2JRER25ICDNVV/
Code of Conduct: http://python.org/psf/codeofconduct/