Mailing List Archive

Actually, there are a few steps that `type.__new__` perform that are not
customizable in metaclasses.

I had sometimes thought about mailing this here, or Python ideas, but could
not
come up with a "real world" use case where the customization of those
would be meaningful.

Let'me see if I recall all cases - two of them are the calls to
`__init_subclass__` and the descriptors `__set_name__` as you put it,
I think there is a third behavior that can't be separated from
`type.__new__` - but
I can't remember it now


Anyway, the "thing to do" that always occurred to me about it is to add
"soft" method slots
to `type` itself - so that `type.__new__` would call those on the
corresponding initialization phases.

Since these are to be run only when classes are created, their impact
should be negligible.

In other words, have `type` implement methods like `__run_init_subclass__`,
`__run_descriptor_setname__`,
(and one for the other task I can't remember now). So, all metaclass code
written up to today remains valid,
and these behaviors become properly customizable.

Adding keyword parameters to `type.__new__`, IMHO, besides a little bit
fishy as we are talking of
arguments to change the behavior of the method, would themselves compete
and have to be
filtered out, or otherwise special-cased in the `__init_subclass__` method
itself.
I mean - let's suppose we add `__suppress_init_subclass__` as an named
parameter to
`type.__new__` - what would happen with this argument in
`__init_subclass__` ? Would it show
up in the kwargs? Otherwise it would be the _only_ kwarg popped out and
not passed
to __init_subclass__, being an inconvenient exception.

Having an overridable, separate, method in type to run __init_subclass__
and __set_name__
bypass these downsides.

In time, Happy holidays everyone!

js
-><-

On Fri, 25 Dec 2020 at 00:38, Ethan Furman <ethan@stoneleaf.us> wrote:

> PEP 487 introduced __init_subclass__ and __set_name__, and both of those
> were wins for the common cases of metaclass usage.
>
> Unfortunately, the implementation of PEP 487 with regards to
> __init_subclass__ has made the writing of correct
> metaclasses significantly harder, if not impossible.
>
> The cause is that when a metaclass calls type.__new__ to actually create
> the class, type.__new__ calls the
> __init_subclass__ methods of the new class' parents, passing it the newly
> created, but incomplete, class. In code:
>
> ```
> class Meta(type):
> #
> def __new__(mcls, name, bases, namespace, **kwds):
> # create new class, which will call __init_subclass__ and
> __set_name__
> new_class = type.__new__(mcls, name, bases, namespace, **kwds)
> # finish setting up class
> new_class.some_attr = 9
> ```
>
> As you can deduce, when the parent __init_subclass__ is called with the
> new class, `some_attr` has not been added yet --
> the new class is incomplete.
>
> For Enum, this means that __init_subclass__ doesn't have access to the new
> Enum's members (they haven't beet added yet).
>
> For ABC, this means that __init_subclass__ doesn't have access to
> __abstract_methods__ (it hasn't been created yet).
>
> Because Enum is pure Python code I was able to work around it:
> - remove new __init_subclass__ (if it exists)
> - insert dummy class with a no-op __init_subclass__
> - call type.__new__
> - save any actual __init_subclass__
> - add back any new __init_subclass__
> - rewrite the new class' __bases__, removing the no-op dummy class
> - finish creating the class
> - call the parent __init_subclass__ with the now complete Enum class
>
> I have not been able to work around the problem for ABC.
>
> Two possible solutions I can think of:
>
> - pass a keyword argument to type.__new__ that suppresses the call to
> __init_subclass__; and
> - provide a way to invoke new class' parent's __init_subclass__ before
> returning it
>
> or
>
> - instead of type.__new__ doing that work, have type.__init__ do it.
>
> Thoughts?
>
> --
> ~Ethan~
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZMRRNSFSLJZDGGZ66CFCYQBINU62CDNX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

For the record - the 3rd process that is currently un-customizable when
creating a class, i.e. things that happen in an opaque way inside
`type.__new__`,
is the ABC class machinery. I could not recall it when
writing the previous e-mail.

Still - I think this might be very little disruptive, and yet provide
metaclasses back with full customization power, including
being able to address the problem brought up by Etham.


js
-><-

On Fri, 25 Dec 2020 at 01:40, Joao S. O. Bueno <jsbueno@python.org.br>
wrote:

> Actually, there are a few steps that `type.__new__` perform that are not
> customizable in metaclasses.
>
> I had sometimes thought about mailing this here, or Python ideas, but
> could not
> come up with a "real world" use case where the customization of those
> would be meaningful.
>
> Let'me see if I recall all cases - two of them are the calls to
> `__init_subclass__` and the descriptors `__set_name__` as you put it,
> I think there is a third behavior that can't be separated from
> `type.__new__` - but
> I can't remember it now
>
>
> Anyway, the "thing to do" that always occurred to me about it is to add
> "soft" method slots
> to `type` itself - so that `type.__new__` would call those on the
> corresponding initialization phases.
>
> Since these are to be run only when classes are created, their impact
> should be negligible.
>
> In other words, have `type` implement methods like
> `__run_init_subclass__`, `__run_descriptor_setname__`,
> (and one for the other task I can't remember now). So, all metaclass code
> written up to today remains valid,
> and these behaviors become properly customizable.
>
> Adding keyword parameters to `type.__new__`, IMHO, besides a little bit
> fishy as we are talking of
> arguments to change the behavior of the method, would themselves compete
> and have to be
> filtered out, or otherwise special-cased in the `__init_subclass__` method
> itself.
> I mean - let's suppose we add `__suppress_init_subclass__` as an named
> parameter to
> `type.__new__` - what would happen with this argument in
> `__init_subclass__` ? Would it show
> up in the kwargs? Otherwise it would be the _only_ kwarg popped out and
> not passed
> to __init_subclass__, being an inconvenient exception.
>
> Having an overridable, separate, method in type to run __init_subclass__
> and __set_name__
> bypass these downsides.
>
> In time, Happy holidays everyone!
>
> js
> -><-
>
> On Fri, 25 Dec 2020 at 00:38, Ethan Furman <ethan@stoneleaf.us> wrote:
>
>> PEP 487 introduced __init_subclass__ and __set_name__, and both of those
>> were wins for the common cases of metaclass usage.
>>
>> Unfortunately, the implementation of PEP 487 with regards to
>> __init_subclass__ has made the writing of correct
>> metaclasses significantly harder, if not impossible.
>>
>> The cause is that when a metaclass calls type.__new__ to actually create
>> the class, type.__new__ calls the
>> __init_subclass__ methods of the new class' parents, passing it the newly
>> created, but incomplete, class. In code:
>>
>> ```
>> class Meta(type):
>> #
>> def __new__(mcls, name, bases, namespace, **kwds):
>> # create new class, which will call __init_subclass__ and
>> __set_name__
>> new_class = type.__new__(mcls, name, bases, namespace, **kwds)
>> # finish setting up class
>> new_class.some_attr = 9
>> ```
>>
>> As you can deduce, when the parent __init_subclass__ is called with the
>> new class, `some_attr` has not been added yet --
>> the new class is incomplete.
>>
>> For Enum, this means that __init_subclass__ doesn't have access to the
>> new Enum's members (they haven't beet added yet).
>>
>> For ABC, this means that __init_subclass__ doesn't have access to
>> __abstract_methods__ (it hasn't been created yet).
>>
>> Because Enum is pure Python code I was able to work around it:
>> - remove new __init_subclass__ (if it exists)
>> - insert dummy class with a no-op __init_subclass__
>> - call type.__new__
>> - save any actual __init_subclass__
>> - add back any new __init_subclass__
>> - rewrite the new class' __bases__, removing the no-op dummy class
>> - finish creating the class
>> - call the parent __init_subclass__ with the now complete Enum class
>>
>> I have not been able to work around the problem for ABC.
>>
>> Two possible solutions I can think of:
>>
>> - pass a keyword argument to type.__new__ that suppresses the call to
>> __init_subclass__; and
>> - provide a way to invoke new class' parent's __init_subclass__ before
>> returning it
>>
>> or
>>
>> - instead of type.__new__ doing that work, have type.__init__ do it.
>>
>> Thoughts?
>>
>> --
>> ~Ethan~
>> _______________________________________________
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-leave@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/ZMRRNSFSLJZDGGZ66CFCYQBINU62CDNX/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>

Let me see if I can unpack this.

I observe that `type.__new__() ` is really the C function `type_new()` in
typeobject.c, and hence I will refer to it by that name.

I understand that `type_new()` is the only way to create type objects, and
it includes a call to `__init_subclass__()`.

In the source code of `type_new()`, calling `__init_subclass__()` is the
last thing it does before returning the newly created class object, so at
this point the class object is complete, *except* that any updates made by
the caller of `type_new()` after `type_new()` returns have not been made,
of course. (For example, `new_class.some_attr = 9` from Ethan's post, or
`__abstractmethods__`, which is set by update_abstractmethods() in abc.py.)

Now here's something that Ethan said that I don't follow:

> For Enum, this means that `__init_subclass__` doesn't have access to the
new Enum's members (they haven't been added yet)

I would presume that in an example like the following, the members *are*
set by the time `type_new()` is called. What am I missing?
```
class Color(enum.Enum):
RED = 1
GREEN = 2
BLUE = 4
```
Maybe the problem is that the members are still set to their "naive"
initial values (1, 2, 4) rather than to the corresponding enum values (e.g.
`<Color.Red: 1>`)?

Without a more elaborate use case I can't give that more than a shrug. This
is how the `__init_subclass__()` protocol is designed. If it's not to your
liking, you can recommend to your users that they use something else.

Nore that for ABC, if you add or change the abstraction status of some
class attributes, you can just call `update_abstractmethods()` and it will
update `__abstractmethods__` based on the new contents of the class. This
also sounds like no biggie to me.



On Mon, Dec 28, 2020 at 6:47 PM Joao S. O. Bueno <jsbueno@python.org.br>
wrote:

> For the record - the 3rd process that is currently un-customizable when
> creating a class, i.e. things that happen in an opaque way inside
> `type.__new__`,
> is the ABC class machinery. I could not recall it when
> writing the previous e-mail.
>
> Still - I think this might be very little disruptive, and yet provide
> metaclasses back with full customization power, including
> being able to address the problem brought up by Etham.
>
>
> js
> -><-
>
> On Fri, 25 Dec 2020 at 01:40, Joao S. O. Bueno <jsbueno@python.org.br>
> wrote:
>
>> Actually, there are a few steps that `type.__new__` perform that are not
>> customizable in metaclasses.
>>
>> I had sometimes thought about mailing this here, or Python ideas, but
>> could not
>> come up with a "real world" use case where the customization of those
>> would be meaningful.
>>
>> Let'me see if I recall all cases - two of them are the calls to
>> `__init_subclass__` and the descriptors `__set_name__` as you put it,
>> I think there is a third behavior that can't be separated from
>> `type.__new__` - but
>> I can't remember it now
>>
>>
>> Anyway, the "thing to do" that always occurred to me about it is to add
>> "soft" method slots
>> to `type` itself - so that `type.__new__` would call those on the
>> corresponding initialization phases.
>>
>> Since these are to be run only when classes are created, their impact
>> should be negligible.
>>
>> In other words, have `type` implement methods like
>> `__run_init_subclass__`, `__run_descriptor_setname__`,
>> (and one for the other task I can't remember now). So, all metaclass code
>> written up to today remains valid,
>> and these behaviors become properly customizable.
>>
>> Adding keyword parameters to `type.__new__`, IMHO, besides a little bit
>> fishy as we are talking of
>> arguments to change the behavior of the method, would themselves compete
>> and have to be
>> filtered out, or otherwise special-cased in the `__init_subclass__`
>> method itself.
>> I mean - let's suppose we add `__suppress_init_subclass__` as an named
>> parameter to
>> `type.__new__` - what would happen with this argument in
>> `__init_subclass__` ? Would it show
>> up in the kwargs? Otherwise it would be the _only_ kwarg popped out and
>> not passed
>> to __init_subclass__, being an inconvenient exception.
>>
>> Having an overridable, separate, method in type to run __init_subclass__
>> and __set_name__
>> bypass these downsides.
>>
>> In time, Happy holidays everyone!
>>
>> js
>> -><-
>>
>> On Fri, 25 Dec 2020 at 00:38, Ethan Furman <ethan@stoneleaf.us> wrote:
>>
>>> PEP 487 introduced __init_subclass__ and __set_name__, and both of those
>>> were wins for the common cases of metaclass usage.
>>>
>>> Unfortunately, the implementation of PEP 487 with regards to
>>> __init_subclass__ has made the writing of correct
>>> metaclasses significantly harder, if not impossible.
>>>
>>> The cause is that when a metaclass calls type.__new__ to actually create
>>> the class, type.__new__ calls the
>>> __init_subclass__ methods of the new class' parents, passing it the
>>> newly created, but incomplete, class. In code:
>>>
>>> ```
>>> class Meta(type):
>>> #
>>> def __new__(mcls, name, bases, namespace, **kwds):
>>> # create new class, which will call __init_subclass__ and
>>> __set_name__
>>> new_class = type.__new__(mcls, name, bases, namespace, **kwds)
>>> # finish setting up class
>>> new_class.some_attr = 9
>>> ```
>>>
>>> As you can deduce, when the parent __init_subclass__ is called with the
>>> new class, `some_attr` has not been added yet --
>>> the new class is incomplete.
>>>
>>> For Enum, this means that __init_subclass__ doesn't have access to the
>>> new Enum's members (they haven't beet added yet).
>>>
>>> For ABC, this means that __init_subclass__ doesn't have access to
>>> __abstract_methods__ (it hasn't been created yet).
>>>
>>> Because Enum is pure Python code I was able to work around it:
>>> - remove new __init_subclass__ (if it exists)
>>> - insert dummy class with a no-op __init_subclass__
>>> - call type.__new__
>>> - save any actual __init_subclass__
>>> - add back any new __init_subclass__
>>> - rewrite the new class' __bases__, removing the no-op dummy class
>>> - finish creating the class
>>> - call the parent __init_subclass__ with the now complete Enum class
>>>
>>> I have not been able to work around the problem for ABC.
>>>
>>> Two possible solutions I can think of:
>>>
>>> - pass a keyword argument to type.__new__ that suppresses the call to
>>> __init_subclass__; and
>>> - provide a way to invoke new class' parent's __init_subclass__ before
>>> returning it
>>>
>>> or
>>>
>>> - instead of type.__new__ doing that work, have type.__init__ do it.
>>>
>>> Thoughts?
>>>
>>> --
>>> ~Ethan~
>>> _______________________________________________
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-leave@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/ZMRRNSFSLJZDGGZ66CFCYQBINU62CDNX/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/NUOMIKVCJMS6S5UUSPDIGD2HD3L56N2F/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Issue #42775: https://bugs.python.org/issue42775
PR #23986: https://github.com/python/cpython/pull/23986
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EQCFWHIOM43L6IH64OFEHP7WTOC2IC6F/
Code of Conduct: http://python.org/psf/codeofconduct/

On 12/28/20 9:31 PM, Guido van Rossum wrote:

> Let me see if I can unpack this.
>
> I observe that `type.__new__() ` is really the C function `type_new()` in typeobject.c, and hence I will refer to it by
> that name.
>
> I understand that `type_new()` is the only way to create type objects, and it includes a call to `__init_subclass__()`.
>
> In the source code of `type_new()`, calling `__init_subclass__()` is the last thing it does before returning the newly
> created class object, so at this point the class object is complete, *except* that any updates made by the caller of
> `type_new()` after `type_new()` returns have not been made, of course. (For example, `new_class.some_attr = 9` from
> Ethan's post, or `__abstractmethods__`, which is set by update_abstractmethods() in abc.py.)

This is really the heart of the issue. A major reason to write a custom metaclass is to be able to modify the returned
class before giving it back to the user -- so even though `type_new` is complete the new class could easily not be
complete, and calling `__init_subclass__` and `__set_name__` from `type_new` is premature.

> Now here's something that Ethan said that I don't follow:
>
>> For Enum, this means that `__init_subclass__` doesn't have access to the new Enum's members (they haven't been added yet)
>
> I would presume that in an example like the following, the members *are* set by the time `type_new()` is called. What am
> I missing?
> ```
> class Color(enum.Enum):
>     RED = 1
>     GREEN = 2
>     BLUE = 4
> ```
> Maybe the problem is that the members are still set to their "naive" initial values (1, 2, 4) rather than to the
> corresponding enum values (e.g. `<Color.Red: 1>`)?

Before `type_new()` is called all the (future) members are removed from `namespace`, so they are not present when
`__init_subclass__` is called. Even if they were left in as `{'RED': 1, `GREEN`: 2, `BLUE`: 4}` it would not be
possible for an `__init_subclass__` to customize the members further, or record them in custom data structures, or run
validation code on them, or etc.

> Without a more elaborate use case I can't give that more than a shrug. This is how the `__init_subclass__()` protocol is
> designed.

The `__init_subclass__` and `__set_name__` protocols are intended to be run before a new type is finished, but creating
a new type has three major steps:

- `__prepare__` to get the namespace
- `__new__` to get the memory and data structures
- `__init__` for any final polishing

We can easily move the calls from `type_new()` to `type_init`.

> Note that for ABC, if you add or change the abstraction status of some class attributes, you can just call
> `update_abstractmethods()` and it will update `__abstractmethods__` based on the new contents of the class. This also
> sounds like no biggie to me.

The issue tracker sample code that fails (from #35815):

```
import abc

class Base(abc.ABC):
#
def __init_subclass__(cls, **kwargs):
instance = cls()
print(f"Created instance of {cls} easily: {instance}")
#
@abc.abstractmethod
def do_something(self):
pass

class Derived(Base):
pass
```

And the output:

`Created instance of <class '__main__.Derived'> easily: <__main__.Derived object at 0x10a6dd6a0>`

If `Base` had been completed before the call to `__init_subclass__`, then `Derived` would have raised an error -- and it
does raise an error with the patch I have submitted on Github.

--
~Ethan~
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HG5SHTY76LKYQS7OY5CXH6TYMUGK7O5L/
Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Dec 28, 2020 at 10:24 PM Ethan Furman <ethan@stoneleaf.us> wrote:

> On 12/28/20 9:31 PM, Guido van Rossum wrote:
>
> > Let me see if I can unpack this.
> >
> > I observe that `type.__new__() ` is really the C function `type_new()`
> in typeobject.c, and hence I will refer to it by
> > that name.
> >
> > I understand that `type_new()` is the only way to create type objects,
> and it includes a call to `__init_subclass__()`.
> >
> > In the source code of `type_new()`, calling `__init_subclass__()` is the
> last thing it does before returning the newly
> > created class object, so at this point the class object is complete,
> *except* that any updates made by the caller of
> > `type_new()` after `type_new()` returns have not been made, of course.
> (For example, `new_class.some_attr = 9` from
> > Ethan's post, or `__abstractmethods__`, which is set by
> update_abstractmethods() in abc.py.)
>
> This is really the heart of the issue. A major reason to write a custom
> metaclass is to be able to modify the returned
> class before giving it back to the user -- so even though `type_new` is
> complete the new class could easily not be
> complete, and calling `__init_subclass__` and `__set_name__` from
> `type_new` is premature.
>
> > Now here's something that Ethan said that I don't follow:
> >
> >> For Enum, this means that `__init_subclass__` doesn't have access to
> the new Enum's members (they haven't been added yet)
> >
> > I would presume that in an example like the following, the members *are*
> set by the time `type_new()` is called. What am
> > I missing?
> > ```
> > class Color(enum.Enum):
> > RED = 1
> > GREEN = 2
> > BLUE = 4
> > ```
> > Maybe the problem is that the members are still set to their "naive"
> initial values (1, 2, 4) rather than to the
> > corresponding enum values (e.g. `<Color.Red: 1>`)?
>
> Before `type_new()` is called all the (future) members are removed from
> `namespace`, so they are not present when
> `__init_subclass__` is called. Even if they were left in as `{'RED': 1,
> `GREEN`: 2, `BLUE`: 4}` it would not be
> possible for an `__init_subclass__` to customize the members further, or
> record them in custom data structures, or run
> validation code on them, or etc.
>
> > Without a more elaborate use case I can't give that more than a shrug.
> This is how the `__init_subclass__()` protocol is
> > designed.
>
> The `__init_subclass__` and `__set_name__` protocols are intended to be
> run before a new type is finished, but creating
> a new type has three major steps:
>
> - `__prepare__` to get the namespace
> - `__new__` to get the memory and data structures
> - `__init__` for any final polishing
>
> We can easily move the calls from `type_new()` to `type_init`.
>

No, we can't. There is a window where the subclass is initialized after
`typing_new()` returned before `__init__` starts, and you propose to move
the subclass after that window. There may be code that depends on the class
being initialized at that point, and you will break that code.

Honestly I disapprove of the shenanigans you're committing in enum.py, and
if those are in in the 3.10 (master) branch I recommend that you take them
out. It just looks too fragile and obscure.


> > Note that for ABC, if you add or change the abstraction status of some
> class attributes, you can just call
> > `update_abstractmethods()` and it will update `__abstractmethods__`
> based on the new contents of the class. This also
> > sounds like no biggie to me.
>
> The issue tracker sample code that fails (from #35815):
>
> ```
> import abc
>
> class Base(abc.ABC):
> #
> def __init_subclass__(cls, **kwargs):
> instance = cls()
> print(f"Created instance of {cls} easily: {instance}")
> #
> @abc.abstractmethod
> def do_something(self):
> pass
>
> class Derived(Base):
> pass
> ```
>
> And the output:
>
> `Created instance of <class '__main__.Derived'> easily: <__main__.Derived
> object at 0x10a6dd6a0>`
>
> If `Base` had been completed before the call to `__init_subclass__`, then
> `Derived` would have raised an error -- and it
> does raise an error with the patch I have submitted on Github.
>

Again, big shrug. A static checker will catch that.

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On 12/29/20 8:59 AM, Guido van Rossum wrote:
> On Mon, Dec 28, 2020 at 10:24 PM Ethan Furman wrote:

>> The `__init_subclass__` and `__set_name__` protocols are intended to be run before a new type is finished, but creating
>> a new type has three major steps:
>>
>> - `__prepare__` to get the namespace
>> - `__new__` to get the memory and data structures
>> - `__init__` for any final polishing
>>
>> We can easily move the calls from `type_new()` to `type_init`.
>
> No, we can't. There is a window where the subclass is initialized after `typing_new()` returned before `__init__`
> starts, and you propose to move the subclass after that window. There may be code that depends on the class being
> initialized at that point, and you will break that code.

True, there will be a few custom metaclasses that need to move some code from their `__new__` to `__init__` instead, and
a few that need to add an `__init__` to consume any keyword arguments that don't need to get passed to
`__init_subclass__`. That seems like a small price to pay to be able to write custom metaclasses that are able to fully
participate in the `__set_name__` and `__init_subclass__` protocols.

Just in the stdlib we have two custom metaclasses that would start working correctly with this change (for the
`__init_subclass__` case).

> Honestly I disapprove of the shenanigans you're committing in enum.py, and if those are in in the 3.10 (master) branch I
> recommend that you take them out. It just looks too fragile and obscure.

Trust me, I don't like them either. But I like even less that a custom `__init_subclass__` for an Enum would fail if it
tried to do anything with the members. That's one of the reasons why I would like to see this fix put in (the
shenanigans would be unnecessary then).

From the point of view of a metaclass author, the current behavior feels buggy. In theory, `__init_subclass__` and
`__set_name__` are supposed to be called after a class is created, and yet they are being called somewhere in the middle
of my metaclass' `__new__`.

Looked at another way, `__init_subclass__` should be receiving a `cls` that is ready for use, i.e. done and fully
constructed and complete. But if that class is being created by a custom metaclass, then the thing that is being given
to `__init_subclass__` could easily be only partial.

--
~Ethan~
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7WPNGL267FDGN6WWCHZQUAXWVBTUJOMN/
Code of Conduct: http://python.org/psf/codeofconduct/

I need to think about this more.

Technically the class *is* created at the point `__init_subclass__` is
called. What the metaclass does after that point is embellishment.

Most metaclasses will have sufficient control for their needs because they
can manipulate the contents of the namespace that's passed to type_new() --
I understand that doesn't work for Enum because the instances have a
reference to the class in them.

I suspect the classes that you claim are "buggy" are fine in practice, even
if you can construct a counterexample.

All in all I just worry about the backward compatibility here: one could
easily imagine a metaclass whose `__new__` relies on the effect of the
`__init_subclass__` called by type_new(), and moving the call would break
that. So it's probably 6 bugs fixed, half a dozen new bugs created. In such
a case, the status quo should win.

--Guido

On Tue, Dec 29, 2020 at 10:44 AM Ethan Furman <ethan@stoneleaf.us> wrote:

> On 12/29/20 8:59 AM, Guido van Rossum wrote:
> > On Mon, Dec 28, 2020 at 10:24 PM Ethan Furman wrote:
>
> >> The `__init_subclass__` and `__set_name__` protocols are intended to be
> run before a new type is finished, but creating
> >> a new type has three major steps:
> >>
> >> - `__prepare__` to get the namespace
> >> - `__new__` to get the memory and data structures
> >> - `__init__` for any final polishing
> >>
> >> We can easily move the calls from `type_new()` to `type_init`.
> >
> > No, we can't. There is a window where the subclass is initialized after
> `typing_new()` returned before `__init__`
> > starts, and you propose to move the subclass after that window. There
> may be code that depends on the class being
> > initialized at that point, and you will break that code.
>
> True, there will be a few custom metaclasses that need to move some code
> from their `__new__` to `__init__` instead, and
> a few that need to add an `__init__` to consume any keyword arguments that
> don't need to get passed to
> `__init_subclass__`. That seems like a small price to pay to be able to
> write custom metaclasses that are able to fully
> participate in the `__set_name__` and `__init_subclass__` protocols.
>
> Just in the stdlib we have two custom metaclasses that would start working
> correctly with this change (for the
> `__init_subclass__` case).
>
> > Honestly I disapprove of the shenanigans you're committing in enum.py,
> and if those are in in the 3.10 (master) branch I
> > recommend that you take them out. It just looks too fragile and obscure.
>
> Trust me, I don't like them either. But I like even less that a custom
> `__init_subclass__` for an Enum would fail if it
> tried to do anything with the members. That's one of the reasons why I
> would like to see this fix put in (the
> shenanigans would be unnecessary then).
>
> From the point of view of a metaclass author, the current behavior feels
> buggy. In theory, `__init_subclass__` and
> `__set_name__` are supposed to be called after a class is created, and yet
> they are being called somewhere in the middle
> of my metaclass' `__new__`.
>
> Looked at another way, `__init_subclass__` should be receiving a `cls`
> that is ready for use, i.e. done and fully
> constructed and complete. But if that class is being created by a custom
> metaclass, then the thing that is being given
> to `__init_subclass__` could easily be only partial.
>
> --
> ~Ethan~
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/7WPNGL267FDGN6WWCHZQUAXWVBTUJOMN/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On 12/29/2020 10:00 PM, Guido van Rossum wrote:
> I need to think about this more.

Both techniques could coexist.

> Technically the class *is* created at the point `__init_subclass__` is
> called. What the metaclass does after that point is embellishment.
>
> Most metaclasses will have sufficient control for their needs because
> they can manipulate the contents of the namespace that's passed to
> type_new() -- I understand that doesn't work for Enum because the
> instances have a reference to the class in them.
>
> I suspect the classes that you claim are "buggy" are fine in practice,
> even if you can construct a counterexample.
>
> All in all I just worry about the backward compatibility here: one
> could easily imagine a metaclass whose `__new__` relies on the effect
> of the `__init_subclass__` called by type_new(), and moving the call
> would break that. So it's probably 6 bugs fixed, half a dozen new bugs
> created. In such a case, the status quo should win.
>
> --Guido
>
> On Tue, Dec 29, 2020 at 10:44 AM Ethan Furman <ethan@stoneleaf.us
> <mailto:ethan@stoneleaf.us>> wrote:
>
> On 12/29/20 8:59 AM, Guido van Rossum wrote:
> > On Mon, Dec 28, 2020 at 10:24 PM Ethan Furman wrote:
>
> >> The `__init_subclass__` and `__set_name__` protocols are
> intended to be run before a new type is finished, but creating
> >> a new type has three major steps:
> >>
> >> - `__prepare__` to get the namespace
> >> - `__new__` to get the memory and data structures
> >> - `__init__` for any final polishing
> >>
> >> We can easily move the calls from `type_new()` to `type_init`.
> >
> > No, we can't. There is a window where the subclass is
> initialized after `typing_new()` returned before `__init__`
> > starts, and you propose to move the subclass after that window.
> There may be code that depends on the class being
> > initialized at that point, and you will break that code.
>
> True, there will be a few custom metaclasses that need to move
> some code from their `__new__` to `__init__` instead, and
> a few that need to add an `__init__` to consume any keyword
> arguments that don't need to get passed to
> `__init_subclass__`.  That seems like a small price to pay to be
> able to write custom metaclasses that are able to fully
> participate in the `__set_name__` and `__init_subclass__` protocols.
>
> Just in the stdlib we have two custom metaclasses that would start
> working correctly with this change (for the
> `__init_subclass__` case).
>
> > Honestly I disapprove of the shenanigans you're committing in
> enum.py, and if those are in in the 3.10 (master) branch I
> > recommend that you take them out. It just looks too fragile and
> obscure.
>
> Trust me, I don't like them either.  But I like even less that a
> custom `__init_subclass__` for an Enum would fail if it
> tried to do anything with the members.  That's one of the reasons
> why I would like to see this fix put in (the
> shenanigans would be unnecessary then).
>
>  From the point of view of a metaclass author, the current
> behavior feels buggy.  In theory, `__init_subclass__` and
> `__set_name__` are supposed to be called after a class is created,
> and yet they are being called somewhere in the middle
> of my metaclass' `__new__`.
>
> Looked at another way, `__init_subclass__` should be receiving a
> `cls` that is ready for use, i.e. done and fully
> constructed and complete. But if that class is being created by a
> custom metaclass, then the thing that is being given
> to `__init_subclass__` could easily be only partial.
>
> --
> ~Ethan~
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> <mailto:python-dev@python.org>
> To unsubscribe send an email to python-dev-leave@python.org
> <mailto:python-dev-leave@python.org>
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> <https://mail.python.org/mailman3/lists/python-dev.python.org/>
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/7WPNGL267FDGN6WWCHZQUAXWVBTUJOMN/
> <https://mail.python.org/archives/list/python-dev@python.org/message/7WPNGL267FDGN6WWCHZQUAXWVBTUJOMN/>
> Code of Conduct: http://python.org/psf/codeofconduct/
> <http://python.org/psf/codeofconduct/>
>
>
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> /Pronouns: he/him //(why is my pronoun here?)/
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AD2ORFW6VCYQNIKCW2DZ3F5KVADDYYZC/
> Code of Conduct: http://python.org/psf/codeofconduct/

On Wed, Dec 30, 2020, 1:05 AM Guido van Rossum <guido@python.org> wrote:

> I need to think about this more.
>
> Technically the class *is* created at the point `__init_subclass__` is
> called. What the metaclass does after that point is embellishment.
>
> Most metaclasses will have sufficient control for their needs because they
> can manipulate the contents of the namespace that's passed to type_new() --
> I understand that doesn't work for Enum because the instances have a
> reference to the class in them.
>
> I suspect the classes that you claim are "buggy" are fine in practice,
> even if you can construct a counterexample.
>
> All in all I just worry about the backward compatibility here: one could
> easily imagine a metaclass whose `__new__` relies on the effect of the
> `__init_subclass__` called by type_new(), and moving the call would break
> that. So it's probably 6 bugs fixed, half a dozen new bugs created. In such
> a case, the status quo should win.
>

I admit I don't know anything about the metaclass internals, but if
backward compatibility is an issue, couldn't we create a new dunder instead
of moving the existing one? Say, call it __init_subclass_2__ (and bikeshed
the name later), and document that as being called at the point to which
it's currently proposed to move the calling of __init_subclass__.

With that, there is zero breakage of existing code, and code that can
benefit from the new behavior simply uses a different dunder.
__init_subclass__ could potentially be deprecated in the future if it's
determined that it has no benefit over the new dunder, but doesn't need to
deprecated immediately.

>

On Wed, 30 Dec 2020 at 04:38, Ethan Furman <ethan@stoneleaf.us> wrote:
> > No, we can't. There is a window where the subclass is initialized after `typing_new()` returned before `__init__`
> > starts, and you propose to move the subclass after that window. There may be code that depends on the class being
> > initialized at that point, and you will break that code.
>
> True, there will be a few custom metaclasses that need to move some code from their `__new__` to `__init__` instead, and
> a few that need to add an `__init__` to consume any keyword arguments that don't need to get passed to
> `__init_subclass__`. That seems like a small price to pay to be able to write custom metaclasses that are able to fully
> participate in the `__set_name__` and `__init_subclass__` protocols.

From https://www.python.org/dev/peps/pep-0487/#implementation-details:

"""As a third option, all the work could have been done in
type.__init__. Most metaclasses do their work in __new__, as this is
recommended by the documentation. Many metaclasses modify their
arguments before they pass them over to super().__new__. For
compatibility with those kind of classes, the hooks should be called
from __new__."""

That isn't the clearest way of saying it, but the intent was to make
sure that metaclass __new__ implementations continued to see fully
initialised class objects as soon as they returned from
type.__new__(), with type.__init__() continuing to be essentially a
no-op.

If a metaclass wants to modify the attributes on the type in a way
that is visible to __init_subclass__ and __set_name__, it needs to
modify the namespace dict passed to type().__new__, not modify the
type after it has already been created:

```
class Meta(type):
#
def __new__(mcls, name, bases, namespace, **kwds):
# create new class, which will call __init_subclass__ and __set_name__
new_namespace = namespace.copy()
# Customize class contents
new_namespace["some_attr"] = 9
return type.__new__(mcls, name, bases, new_namespace, **kwds)
```

If you need to access the class being defined from __init_subclass__,
it gets passed in, and if you need to access it from __set_name__,
then it gets passed in there as well.

If you need to access the class being defined from elsewhere, then the
`__class__` cell used by zero-arg super is already populated before
any of those hooks get called.

As a CPython implementation detail, that cell is actually passed
through to type().__new__ as `__classcell__` in the namespace. If that
was elevated to a language feature rather than a CPython
implementation detail, then metaclasses could write:

cls_cell = namespace["__classcell__"]
@property
def defining_class(obj):
return cls_cell.get()
namespace[defining_class.fget.__name__] = defining_class

Cheers,
Nick.

--
Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QWVL2UPV2MCJJXTO3TOANH5YSR2GEDBP/
Code of Conduct: http://python.org/psf/codeofconduct/

On 1/5/21 4:22 AM, Nick Coghlan wrote:
> On Wed, 30 Dec 2020 at 04:38, Ethan Furman wrote:

>> ... there will be a few custom metaclasses that need to move some code
>> from their `__new__` to `__init__` instead, and a few that need to add
>> an `__init__` to consume any keyword arguments that don't need to get
>> passed to `__init_subclass__`. That seems like a small price to pay to
>> be able to write custom metaclasses that are able to fully
>> participate in the `__set_name__` and `__init_subclass__` protocols.
>
> If a metaclass wants to modify the attributes on the type in a way
> that is visible to __init_subclass__ and __set_name__, it needs to
> modify the namespace dict passed to type().__new__, not modify the
> type after it has already been created.

And if that's not possible? (More on that later.)

> If you need to access the class being defined from elsewhere, then the
> `__class__` cell used by zero-arg super is already populated before
> any of those hooks get called.

If the custom metaclass hasn't finished its work, I don't see how that is helpful.

> As a CPython implementation detail, that cell is actually passed
> through to type().__new__ as `__classcell__` in the namespace. If that
> was elevated to a language feature rather than a CPython
> implementation detail, then metaclasses could write:
>
> cls_cell = namespace["__classcell__"]
> @property
> def defining_class(obj):
> return cls_cell.get()
> namespace[defining_class.fget.__name__] = defining_class

I have two concrete examples of custom metaclasses currently broken by calling `__init_subclass__` from `type.__new__`:

- `ABCMeta`

- `Enum`

Perhaps `ABCMeta` can be easily fixed -- it is calling the function that records all abstract methods, etc., after the
`type.__new__` call; can it be called before?

`Enum` cannot be fixed: it must call `type.__new__` to get the new class object, and it must have the new class object
in order to create the members as instances of that `Enum` class -- which means that anyone trying to use
`__init_subclass__` to do anything with an `Enum`'s members is doomed to failure because those members don't exist yet.

Possible work-arounds for `Enum`:

- call `__init_subclass__` twice - once from `type.__new__`, and then again
by `EnumMeta` after the members have been added

* problem: authors of `__init_subclass__` have to be aware that they will
get called twice

- do horribly brittle mro rewrites in `EnumMeta` so the `type.__new__` call
is a no-op, then `EnumMeta` calls `__init_subclass__` when the members
have been created

* problem: horribly brittle

------------------------------------------------------------------------------

While I would prefer to see the calls to `__init_subclass__` and `__set_name__` (and any other such enhancements) moved
to `type.__init__`, another possibility would be to add a class attribute/keyword, `_pep487_to_init`, which if present
causes `type.__new__` to skip those calls and `type.__init__` to make them. Then the only headaches would be when
metaclasses with different desires were combined.

I am also open to other solutions that would allow `EnumMeta` and its users to be able to use `__init_subclass__` with a
fully created class.

--
~Ethan~
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NIIQPQO3DLDTMV6PJQNGM3ODUQHJBYSQ/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Jan 5, 2021 at 8:50 AM Ethan Furman <ethan@stoneleaf.us> wrote:

> [...]
> Perhaps `ABCMeta` can be easily fixed -- it is calling the function that
> records all abstract methods, etc., after the
> `type.__new__` call; can it be called before?
>

Why do you keep insisting that this is "broken"? Yes, you have identified a
way that an abstract class can be instantiated *by its own constructor*
despite being abstract. I think that's not something to get worked up
about. As the doctor in the old joke said, "then don't do that." Or perhaps
this is one of those situations where we're all still considered to be
consenting adults -- surely abstract classes and methods are meant as a way
to help users do the right thing, not as an absolute guarantee.

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On 1/5/21 10:25 AM, Guido van Rossum wrote:
> On Tue, Jan 5, 2021 at 8:50 AM Ethan Furman wrote:

>> [...]
>> Perhaps `ABCMeta` can be easily fixed -- it is calling the function that
>> records all abstract methods, etc., after the `type.__new__` call; can
>> it be called before?
>
> Why do you keep insisting that this is "broken"?

Because `__init_subclass__` is getting a new class object that is not ready to be gotten.

> Yes, you have identified a way that an abstract class can be instantiated
> *by its own constructor* despite being abstract.

It doesn't feel that way to me. Referring to the code below [1], the constructor for `MyPlugin` is `abc.ABCMeta`, but
`PluginBase.__init_subclass__` is getting `MyPlugin` before `abc.ABCMeta` has finished doing its job.

> I think that's not something to get worked up about. As the doctor in the
> old joke said, "then don't do that." Or perhaps this is one of those
> situations where we're all still considered to be consenting adults --
> surely abstract classes and methods are meant as a way to help users do
> the right thing, not as an absolute guarantee.

According to the documentation:
> This module provides abstract base classes that can be used to test whether
> a class provides a particular interface; for example, whether it is hashable
> or whether it is a mapping.

Do we need to add a warning to that?

> except inside `__init_subclass__` as the class has not been fully initialized.

For that matter, if we are not going to address this issue with code, we should definitely update the
`__init_subclass__` and `__set_name__` documentation:

> Note: custom metaclasses must have all changes needed already in the class
> namespace before calling `type.__new__`, as `type.__new__` will invoke
> `__init_subclass__` and `__set_name__` before returning. Any modifications
> made to the class after that point will not be available to those two methods.
>
> Note: because of the above, any `__init_subclass__` used with `Enum` classes will
> not see any members as members are added after the call to `type.__new__`.
>
> Note: because of the above, any `__init_subclass__` used with `ABC` classes will
> have to determine for itself if the class is abstract or concrete as that is
> determined after the call to `type.__new__`.

As I said before, perhaps `_abc_init` can be called before `type.__new__`, which would make that last note unnecessary.

That would not solve the problem for `Enum`, though, nor for any other metaclass that cannot complete its customization
before calling `type.__new__`.

--
~Ethan~


[1]
class PluginBase(abc.ABC):

# base class for plugins
# must override `do_something`

def __init_subclass__(cls, **kwargs):

instance = cls()
print(f"Created instance of {cls} easily: {instance}")


@abc.abstractmethod
def do_something(self):
pass


class MyPlugin(PluginBase):
pass



[2] https://docs.python.org/3/library/collections.abc.html
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KE3ZJEDFMZT4TXTTHXMVGEZSLX2HDUSZ/
Code of Conduct: http://python.org/psf/codeofconduct/

Both EnumMeta and ABCMeta should probably be relying on __set_name__ for
their per-member set up work these days, rather than deferring that work
until after __new__ returns.

Cheers,
Nick.

On 1/7/21 4:56 AM, Nick Coghlan wrote:

> Both EnumMeta and ABCMeta should probably be relying on `__set_name__`
> for their per-member set up work these days, rather than deferring
> that work until after __new__ returns.

And here I was thinking that `__set_name__` was for, well, setting the name. ;-)

But it does work.

--
~Ethan~
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/57NF3L53RJPJGUCSFIG7KPKYDLTCUCZL/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, 12 Jan 2021, 11:55 am Ethan Furman, <ethan@stoneleaf.us> wrote:

> On 1/7/21 4:56 AM, Nick Coghlan wrote:
>
> > Both EnumMeta and ABCMeta should probably be relying on `__set_name__`
> > for their per-member set up work these days, rather than deferring
> > that work until after __new__ returns.
>
> And here I was thinking that `__set_name__` was for, well, setting the
> name. ;-)
>


Yeah, __set_owner__ was also discussed as a possible name, since the method
notifies the descriptor of both the defining class and the attribute name
within that class.

We decided we preferred __set_name__ out of the two short options.



> But it does work.
>

Good to hear :)

Cheers,
Nick.



>