Mailing List Archive

Multiple inheritance and a broken super() chain
Hello.

The legacy code I'm working with uses a classic diamond inheritance. Let me
call the classes *Top*, *Left*, *Right*, and *Bottom*.
This is a trivial textbook example. The classes were written in the
pre-super() era, so all of them initialized their parents and Bottom
initialized both Left and Right in this order.

The result was expected: *Top* was initialized twice:

Top.__init__() Left.__init__() Top.__init__() Right.__init__()
Bottom.__init__()

Now I replaced all parent init calls with *super()*. After this, Top was
initialized only once.

Top.__init__() Right.__init__() Left.__init__() Bottom.__init__()

But at this point, I freaked out. The code is complex and I don't have the
time to examine its inner workings. And before, everything worked correctly
even though Top was initialized twice. So I decided to break the superclass
chain and use super() only in classes inheriting from a single parent. My
intent was to keep the original behavior but use super() where possible to
make the code more readable.

class Top:
def __init__(self):
print("Top.__init__()")

class Left(Top):
def __init__(self):
super().__init__()
print("Left.__init__()")

class Right(Top):
def __init__(self):
super().__init__()
print("Right.__init__()")

class Bottom(Left, Right):
def __init__(self):
Left.__init__(self) # Here I'm calling both parents manually
Right.__init__(self)
print("Bottom.__init__()")

b = Bottom()


The result has surprised me:

Top.__init__() Right.__init__() Left.__init__() Top.__init__()
Right.__init__() Bottom.__init__()

Now, as I see it, from the super()'s point of view, there are two
inheritance chains, one starting at Left and the other at Right. But
*Right.__init__()* is called twice. What's going on here?

Thanks,
Peter
--
https://mail.python.org/mailman/listinfo/python-list
Re: Multiple inheritance and a broken super() chain [ In reply to ]
On 7/3/23 1:38 PM, Peter Slížik via Python-list wrote:
> Hello.
>
> The legacy code I'm working with uses a classic diamond inheritance. Let me
> call the classes *Top*, *Left*, *Right*, and *Bottom*.
> This is a trivial textbook example. The classes were written in the
> pre-super() era, so all of them initialized their parents and Bottom
> initialized both Left and Right in this order.
>
> The result was expected: *Top* was initialized twice:
>
> Top.__init__() Left.__init__() Top.__init__() Right.__init__()
> Bottom.__init__()
>
> Now I replaced all parent init calls with *super()*. After this, Top was
> initialized only once.
>
> Top.__init__() Right.__init__() Left.__init__() Bottom.__init__()
>
> But at this point, I freaked out. The code is complex and I don't have the
> time to examine its inner workings. And before, everything worked correctly
> even though Top was initialized twice. So I decided to break the superclass
> chain and use super() only in classes inheriting from a single parent. My
> intent was to keep the original behavior but use super() where possible to
> make the code more readable.
>
> class Top:
> def __init__(self):
> print("Top.__init__()")
>
> class Left(Top):
> def __init__(self):
> super().__init__()
> print("Left.__init__()")
>
> class Right(Top):
> def __init__(self):
> super().__init__()
> print("Right.__init__()")
>
> class Bottom(Left, Right):
> def __init__(self):
> Left.__init__(self) # Here I'm calling both parents manually
> Right.__init__(self)
> print("Bottom.__init__()")
>
> b = Bottom()
>
>
> The result has surprised me:
>
> Top.__init__() Right.__init__() Left.__init__() Top.__init__()
> Right.__init__() Bottom.__init__()
>
> Now, as I see it, from the super()'s point of view, there are two
> inheritance chains, one starting at Left and the other at Right. But
> *Right.__init__()* is called twice. What's going on here?
>
> Thanks,
> Peter

Because the MRO from Bottom is [Bottom, Left, Right, Top] so super() in
Left is Right. It doesn't go to Top as the MRO knows that Right should
go to Top, so Left needs to go to Right to init everything, and then
Bottom messes things up by calling Right again.

--
Richard Damon

--
https://mail.python.org/mailman/listinfo/python-list
Re: Multiple inheritance and a broken super() chain [ In reply to ]
On 7/3/23 12:01, Richard Damon via Python-list wrote:
> On 7/3/23 1:38 PM, Peter Slížik via Python-list wrote:
>> Hello.
>>
>> The legacy code I'm working with uses a classic diamond inheritance.
>> Let me
>> call the classes *Top*, *Left*, *Right*, and *Bottom*.
>> This is a trivial textbook example. The classes were written in the
>> pre-super() era, so all of them initialized their parents and Bottom
>> initialized both Left and Right in this order.
>>
>> The result was expected: *Top* was initialized twice:
>>
>> Top.__init__() Left.__init__() Top.__init__() Right.__init__()
>> Bottom.__init__()
>>
>> Now I replaced all parent init calls with *super()*. After this, Top was
>> initialized only once.
>>
>> Top.__init__() Right.__init__() Left.__init__() Bottom.__init__()
>>
>> But at this point, I freaked out. The code is complex and I don't have
>> the
>> time to examine its inner workings. And before, everything worked
>> correctly
>> even though Top was initialized twice. So I decided to break the
>> superclass
>> chain and use super() only in classes inheriting from a single parent. My
>> intent was to keep the original behavior but use super() where
>> possible to
>> make the code more readable.
>>
>> class Top:
>> def __init__(self):
>> print("Top.__init__()")
>>
>> class Left(Top):
>> def __init__(self):
>> super().__init__()
>> print("Left.__init__()")
>>
>> class Right(Top):
>> def __init__(self):
>> super().__init__()
>> print("Right.__init__()")
>>
>> class Bottom(Left, Right):
>> def __init__(self):
>> Left.__init__(self) # Here I'm calling both parents manually
>> Right.__init__(self)
>> print("Bottom.__init__()")
>>
>> b = Bottom()
>>
>>
>> The result has surprised me:
>>
>> Top.__init__() Right.__init__() Left.__init__() Top.__init__()
>> Right.__init__() Bottom.__init__()
>>
>> Now, as I see it, from the super()'s point of view, there are two
>> inheritance chains, one starting at Left and the other at Right. But
>> *Right.__init__()* is called twice. What's going on here?
>>
>> Thanks,
>> Peter
>
> Because the MRO from Bottom is [Bottom, Left, Right, Top] so super() in
> Left is Right. It doesn't go to Top as the MRO knows that Right should
> go to Top, so Left needs to go to Right to init everything, and then
> Bottom messes things up by calling Right again.

And you can see this a little better in your toy example by using begin
*and* end prints in your initializers.

Also, you might find that because of the MRO, super() in your Bottom
class would actually give you what you want.

And if not sure, print out Bottom.__mro__



--
https://mail.python.org/mailman/listinfo/python-list
Re: Multiple inheritance and a broken super() chain [ In reply to ]
On Tue, 4 Jul 2023 at 03:39, Peter Slížik via Python-list
<python-list@python.org> wrote:
>
> Hello.
>
> The legacy code I'm working with uses a classic diamond inheritance. Let me
> call the classes *Top*, *Left*, *Right*, and *Bottom*.
> This is a trivial textbook example. The classes were written in the
> pre-super() era, so all of them initialized their parents and Bottom
> initialized both Left and Right in this order.
>
> The result was expected: *Top* was initialized twice:
>
> Top.__init__() Left.__init__() Top.__init__() Right.__init__()
> Bottom.__init__()

What happens when Top is initialized twice? This seems like a problem
waiting to happen, and when you moved to using super(), you more than
likely simplified things and fixed things.

There are not two instances of Top to be initialized, only one.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Multiple inheritance and a broken super() chain [ In reply to ]
On 7/3/23 12:13, Mats Wichmann via Python-list wrote:

To natter on a bit, and possibly muddy the waters even further...

> Now, as I see it, from the super()'s point of view, there are two
> inheritance chains, one starting at Left and the other at Right. But
> *Right.__init__()* is called twice.
No: each class has just a single inheritance chain, built up when the
class object is constructed, to avoid going insane. Yes, the chain for
Left and for Right are different, but you're not instantiating *either*
of those classes when you make a Bottom, so they don't matter here.
You're just filling out a Bottom: it looks for init, finds it, and so
would stop hunting - but then the super() call there sends it to the
next class object in the chain to look for the method you told it to
(also init), where it would stop since it found it, except you sent it
on with another super(), and so on. Python is a bit... different :)
(compared to other languages with class definitions)

--
https://mail.python.org/mailman/listinfo/python-list
Re: Multiple inheritance and a broken super() chain [ In reply to ]
>
> Also, you might find that because of the MRO, super() in your Bottom
> class would actually give you what you want.
>

I knew this, but I wanted to save myself some refactoring, as the legacy
code used different signatures for Left.__init__() and Right.__init__().

I realized the formatting of my code examples was completely removed; sorry
for that.

Best regards,
Peter
--
https://mail.python.org/mailman/listinfo/python-list
Re: Multiple inheritance and a broken super() chain [ In reply to ]
On Tue, 4 Jul 2023 at 22:06, Peter Slížik via Python-list
<python-list@python.org> wrote:
>
> >
> > Also, you might find that because of the MRO, super() in your Bottom
> > class would actually give you what you want.
> >
>
> I knew this, but I wanted to save myself some refactoring, as the legacy
> code used different signatures for Left.__init__() and Right.__init__().

This sounds like a mess that I would not touch. Unless something needs
to be fixed, I wouldn't switch these to use super() - I'd leave them
using explicit parent calls.

However, I would acknowledge somewhere that this will cause
Top.__init__() to be called twice on the same object.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Multiple inheritance and a broken super() chain [ In reply to ]
On 03/07/2023 19:39, Chris Angelico via Python-list wrote:
> On Tue, 4 Jul 2023 at 03:39, Peter Slížik via Python-list
>> The legacy code I'm working with uses a classic diamond inheritance.

> What happens when Top is initialized twice? This seems like a problem
> waiting to happen, and when you moved to using super(), you more than
> likely simplified things and fixed things.

Slightly off topic but I wonder how many real world problems
people have experienced having the top of a diamond initialized
twice? The reason I ask is that I ran a maintenance team for
about 5 years (early 1990s) working on various OOP projects using MI;
in Lisp Flavors, C++(*) and a homebrew variant of C that supported MI.
In that time I don't recall ever having problems with top objects
being initialized twice (apart from redundant execution of code
of course).

In most cases the top object was so abstract that its init()/constructor
was only doing generic type stuff or opening database sessions/networks
etc which got lost and tidied up by garbage collectors.

So I'm curious about how big this "big problem with MI" is in
practice. I'm sure there are scenarios where it has bitten folks
but it never (or very rarely) occurred in our projects. (Of
course, being maintenance programmers, the problems may have
been ironed out before the code ever reached us! But that
wasn't usually the case...)

(*) C++ is the odd one out because it doesn't have GC, but then
neither does it have an Object superclass so very often MI in C++
does not involve creating diamonds! And especially if the MI
style is mixin based.

--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


--
https://mail.python.org/mailman/listinfo/python-list
Re: Multiple inheritance and a broken super() chain [ In reply to ]
On 5/07/23 10:33 am, Alan Gauld wrote:
> (*) C++ is the odd one out because it doesn't have GC, but then
> neither does it have an Object superclass so very often MI in C++
> does not involve creating diamonds! And especially if the MI
> style is mixin based.

Even if all your mixins have empty constructors, in C++ there
is still a diamond problem if they have any data members, because
you end up with multiple copies of them.

But C++ has the concept of virtual base classes, which avoids the
diamond problem, albeit at the expense of making you explicitly
call all the base class constructors in your ancestry.

--
Greg





--
https://mail.python.org/mailman/listinfo/python-list
Re: Multiple inheritance and a broken super() chain [ In reply to ]
On Wed, 5 Jul 2023 at 08:35, Alan Gauld via Python-list
<python-list@python.org> wrote:
>
> On 03/07/2023 19:39, Chris Angelico via Python-list wrote:
> > On Tue, 4 Jul 2023 at 03:39, Peter Slížik via Python-list
> >> The legacy code I'm working with uses a classic diamond inheritance.
>
> > What happens when Top is initialized twice? This seems like a problem
> > waiting to happen, and when you moved to using super(), you more than
> > likely simplified things and fixed things.
>
> Slightly off topic but I wonder how many real world problems
> people have experienced having the top of a diamond initialized
> twice? The reason I ask is that I ran a maintenance team for
> about 5 years (early 1990s) working on various OOP projects using MI;
> in Lisp Flavors, C++(*) and a homebrew variant of C that supported MI.
> In that time I don't recall ever having problems with top objects
> being initialized twice (apart from redundant execution of code
> of course).

It's important to distinguish between diamond inheritance and what the
OP seemed to expect, which was independent hierarchies. (In C++ terms,
that's virtual inheritance and .... the unnamed default type of MI.
Non-virtual inheritance??) With independent hierarchies, the object is
composed of two subobjects, each with its own regular
single-inheritance tree, and unless you need to call a method on the
duplicated grandparent from the second parent (in which case you have
to cast before calling), it's perfectly natural to treat them
separately. But with virtual inheritance - as is always the case in
Python - there is only one object.

Whether it's a problem depends entirely on what the initializer does.
If it's idempotent and doesn't depend on any arguments that it didn't
already get, we're fine! But if it does something like this:

class Example:
def __init__(self):
self.button = Some_GUI_Library.Button("Example")
self.button.add_to_window()

then calling init twice will create a second button.

It's easy enough to design a specification that is safe against double
initialization; but then, it's also not that hard to design a
specification that's safe against super().__init__() and odd
hierarchies.

Since the OP didn't show us any of the code, I had to mention the
possibility here. Of course, it's entirely possible that it isn't
actually a problem.

> In most cases the top object was so abstract that its init()/constructor
> was only doing generic type stuff or opening database sessions/networks
> etc which got lost and tidied up by garbage collectors.

Remember though that diamond inheritance doesn't always happen at the
top-level object.

> So I'm curious about how big this "big problem with MI" is in
> practice. I'm sure there are scenarios where it has bitten folks
> but it never (or very rarely) occurred in our projects. (Of
> course, being maintenance programmers, the problems may have
> been ironed out before the code ever reached us! But that
> wasn't usually the case...)

Who said it's a big problem with MI? Diamond inheritance is in general
a problem to be solved, but calling __init__ twice is a separate
concern and usually only an issue when you try to treat MRO-based MI
as if it were composition-based MI.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Multiple inheritance and a broken super() chain [ In reply to ]
On Wed, 5 Jul 2023 at 10:31, Greg Ewing via Python-list
<python-list@python.org> wrote:
>
> On 5/07/23 10:33 am, Alan Gauld wrote:
> > (*) C++ is the odd one out because it doesn't have GC, but then
> > neither does it have an Object superclass so very often MI in C++
> > does not involve creating diamonds! And especially if the MI
> > style is mixin based.
>
> Even if all your mixins have empty constructors, in C++ there
> is still a diamond problem if they have any data members, because
> you end up with multiple copies of them.
>
> But C++ has the concept of virtual base classes, which avoids the
> diamond problem, albeit at the expense of making you explicitly
> call all the base class constructors in your ancestry.

Yeah, non-virtual MI in C++ is basically composition with a shorthand
for calling non-conflicting methods or accessing non-conflicting data
members. Point of random interest: Pike actually allows that sort of
"composition MI" but will give you back an array of all parents when
you seek a superclass's method, giving a very elegant syntax for MI.

inherit Thing1;
inherit Thing2;

void method() {
::method(); //this is actually calling an array of two methods
}

Python's way of doing it requires that every class choose to cooperate
in the MI and then be aware that they are all operating on the same
object. Pike's and C++'s can sometimes be used as composition in
disguise, but in general, MI does require proper cooperation.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Multiple inheritance and a broken super() chain [ In reply to ]
On 05/07/2023 01:27, Chris Angelico via Python-list wrote:

>> So I'm curious about how big this "big problem with MI" is in
>
> Who said it's a big problem with MI?

I think it's a very common perception, particularly with
newer programmers who have never used it in anger. Any
time anyone discusses MI it seems somebody will jump in
and warn about diamonds etc. As a result many steer clear
of MI, which is a shame.

My personal experience of MI is that used appropriately
it is a powerful and useful tool. But it must be used
in a true is-a type relationship and not just as a kind
of cheap reuse mechanism - that's when problems start.

Also, mixin style MI is particularly powerful but the
protocol between mixin and "subclass" needs to be carefully
designed and documented. Like any powerful tool you need
to understand the costs of use as well as the potential
benefits.

--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


--
https://mail.python.org/mailman/listinfo/python-list