Mailing List Archive

On Thu, 22 Oct 2020 at 18:31, Julio Di Egidio <julio@diegidio.name> wrote:

> why
> only if there are abstract methods defined in an ABC
> class is instantiation disallowed?
>

Not sure because I never tried or needed, but if no @abstractsomething in A
is defined and your B class is a subclass of A, B should be an abstract
class, not a concrete class.
--
https://mail.python.org/mailman/listinfo/python-list

On Thu, 22 Oct 2020 at 22:09, Marco Sulla <Marco.Sulla.Python@gmail.com>
wrote:

> Not sure because I never tried or needed, but if no @abstractsomething in
> A is defined and your B class is a subclass of A, B should be an abstract
> class, not a concrete class.
>

Now I'm sure:

>>> from abc import ABC, abstractmethod
>>> class A(ABC): pass
...
>>> class B(A):
... @abstractmethod
... def hello(self):
... print("hello")
...
>>> B()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class B with abstract methods hello

>
>
--
https://mail.python.org/mailman/listinfo/python-list

On 10/22/20 9:25 AM, Julio Di Egidio wrote:

> Now, I do read in the docs that that is as intended,
> but I am not understanding the rationale of it: why
> only if there are abstract methods defined in an ABC
> class is instantiation disallowed? IOW, why isn't
> subclassing from ABC enough?

Let's say you subclass from ABC:

class Abstract(ABC):
pass

Then you subclass from that:

class Concrete(Abstract):
pass

Then subclass from that:

class MoreConcrete(Concrete):
pass

If you do a

issubclass(<any of the above classes>, ABC)

you'll get

True

The idea behind abstract classes is the prevention of creating non-functional instances, which means if any abstract
methods, properties, etc., are present in an abstract class, then it's instances will not be fully functional;
contrariwise, if there are no abstract anythings in the class, then it is functional and there's no reason not to allow
it to be created.

Put another way: if ABC is anywhere in a class' parentage, then it is "abstract" -- the only way to tell if
instantiating it is okay is by the presence/absence of abstract pieces in the class.

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

On Thursday, 22 October 2020 23:04:25 UTC+2, Ethan Furman wrote:
> On 10/22/20 9:25 AM, Julio Di Egidio wrote:
>
> > Now, I do read in the docs that that is as intended,
> > but I am not understanding the rationale of it: why
> > only if there are abstract methods defined in an ABC
> > class is instantiation disallowed? IOW, why isn't
> > subclassing from ABC enough?
>
> Let's say you subclass from ABC:
>
> class Abstract(ABC):
> pass
>
> Then you subclass from that:
>
> class Concrete(Abstract):
> pass
>
> Then subclass from that:
>
> class MoreConcrete(Concrete):
> pass
>
> If you do a
>
> issubclass(<any of the above classes>, ABC)
>
> you'll get
>
> True

Ah, right, that's the point I was missing: how to tell the
compiler when a more derived class is *not* abstract...

I was indeed making the mistake of inheriting from ABC also
in the derived classes, and omitting it in the classes that
are eventually concrete, not realising that ABC isn't just
a keywork or a decorator, so it gets inherited all the way.

> The idea behind abstract classes is the prevention of creating
> non-functional instances

Well, in Python, not in any other OO language, where abstract
is just synonym with must-override (hence cannot instantiate),
no other constraints.

I am now thinking whether I could achieve the "standard"
behaviour via another approach, say with decorators, somehow
intercepting calls to __new__... maybe. Anyway, abstract
classes are the gist of most library code, and I remain a bit
puzzled by the behaviour implemented in Python: but I am not
complaining, I know it will take me some time before I actually
understand the language...

For now, thank you and Marco very much for your feedback,

Julio
--
https://mail.python.org/mailman/listinfo/python-list

On 23/10/20 2:13 pm, Julio Di Egidio wrote:
> I am now thinking whether I could achieve the "standard"
> behaviour via another approach, say with decorators, somehow
> intercepting calls to __new__... maybe.

I'm inclined to step back and ask -- why do you care about this?

Would it actually do any harm if someone instantiated your
base class? If not, then it's probably not worth going out
of your way to prevent it.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

On Friday, 23 October 2020 07:36:39 UTC+2, Greg Ewing wrote:
> On 23/10/20 2:13 pm, Julio Di Egidio wrote:
> > I am now thinking whether I could achieve the "standard"
> > behaviour via another approach, say with decorators, somehow
> > intercepting calls to __new__... maybe.
>
> I'm inclined to step back and ask -- why do you care about this?
>
> Would it actually do any harm if someone instantiated your
> base class? If not, then it's probably not worth going out
> of your way to prevent it.

This is the first little library I try to put together
in Python, and it was natural for me to hit it with all
the relevant decorations as well as type annotations in
order to expose *strict* contracts, plus hopefully have
intellisense work all over the place.

After several days of struggling, I am indeed finding that
impossible in Python, at least with the available tools,
and I am indeed going out of my way to just get some
half-decent results... But, if I give up on strict
contracts, I can as well give up on type annotations
and the whole lot, indeed why even subclass ABC? Which
is maybe too drastic, maybe not: it's the next thing I
am going to try, and see what I remain with. :)

Of course, any more hints welcome...

Julio
--
https://mail.python.org/mailman/listinfo/python-list

I'm a C++ programmer and Python programmer as well. Python classes are not exactly like C++ classes.

If you define a class where every method has an implementation, then it really isn't abstract. It can be instantiated. You can force it to be abstract by doing from abc import ABCMeta and declare class myclass(metaClass=ABCMeta). Otherwise, Python does not have a way to know that you intend the class to be abstract unless it contains an @abstractmethod that makes it actually abstract. Such a method must be overridden.

Usually, an Abstract Base Class defines an interface. You can make all the functions @abstractmethod, and separately make a another class that is based on your ABC and provides default implementations for all the functions. Other classes can be based on that class. I am not an authority on this so let me refer you to actual documentation:

See: https://docs.python.org/3/library/abc.html, that should help you.

-----Original Message-----
From: Julio Di Egidio <julio@diegidio.name>
Sent: Thursday, October 22, 2020 12:26 PM
To: python-list@python.org
Subject: Question on ABC classes

Hello guys,

I am professional programmer but quite new to Python, and I am trying to get the grips of some peculiarities of the language.

Here is a basic question: if I define an ABC class, I can still instantiate the class unless there are abstract methods defined in the class.

(In the typical OO language the class would be not instantiable, period, since it's "abstract". But this is not so in Python, to the point that, also for uniformity, I am feeling compelled to define an @abstractmethod __init__ in my ABC classes, whether they need one or not, and whether there are other abstract methods in the class or not.)

Now, I do read in the docs that that is as intended, but I am not understanding the rationale of it: why only if there are abstract methods defined in an ABC class is instantiation disallowed? IOW, why isn't subclassing from ABC enough?

Thanks for any enlightenment,

Julio

--
https://mail.python.org/mailman/listinfo/python-list

On 2020-10-22 23:35:21 -0700, Julio Di Egidio wrote:
> On Friday, 23 October 2020 07:36:39 UTC+2, Greg Ewing wrote:
> > On 23/10/20 2:13 pm, Julio Di Egidio wrote:
> > > I am now thinking whether I could achieve the "standard"
> > > behaviour via another approach, say with decorators, somehow
> > > intercepting calls to __new__... maybe.
> >
> > I'm inclined to step back and ask -- why do you care about this?
> >
> > Would it actually do any harm if someone instantiated your
> > base class? If not, then it's probably not worth going out
> > of your way to prevent it.
>
> This is the first little library I try to put together
> in Python, and it was natural for me to hit it with all
> the relevant decorations as well as type annotations in
> order to expose *strict* contracts, plus hopefully have
> intellisense work all over the place.

I think you are trying to use Python in a way contrary to its nature.
Python is a dynamically typed language. Its variables don't have types,
only its objects. And basically everything can be changed at runtime ...

It now has type annotations, but they are ignored both by the compiler
and at run-time. They are only for the benefit of linting tools like
mypy and editor features like intellisense.

> and the whole lot, indeed why even subclass ABC?

Good question. In six years of programming Python, I've never used ABC.
But then I came from another dynamically typed language to Python.

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

Peter J. Holzer wrote at 2020-10-25 20:48 +0100:
>On 2020-10-22 23:35:21 -0700, Julio Di Egidio wrote:
> ...
>> and the whole lot, indeed why even subclass ABC?

You often have the case that a base class can implement
a lot of functionality based on a few methods defined
by derived classes.

An example is a mapping class (with methods such as keys, values, items,
iteration, subscription, ...). All those methods can be implemented
generically based on "__len__", "__iter__", "__getitem__" and
"__getitem__".

In those cases, you can decorate the base methods with
`abstractmethod`.


Formerly, you had `raise NotImplementedError` in the body of
those methods. If a derived class forgot to implement a required
method, the exception was raised as soon as the method was called.
With `abstractmethod` you see the problem earlier.
--
https://mail.python.org/mailman/listinfo/python-list

On Sunday, 25 October 2020 20:55:26 UTC+1, Peter J. Holzer wrote:
> On 2020-10-22 23:35:21 -0700, Julio Di Egidio wrote:
> > On Friday, 23 October 2020 07:36:39 UTC+2, Greg Ewing wrote:
> > > On 23/10/20 2:13 pm, Julio Di Egidio wrote:
> > > > I am now thinking whether I could achieve the "standard"
> > > > behaviour via another approach, say with decorators, somehow
> > > > intercepting calls to __new__... maybe.
> > >
> > > I'm inclined to step back and ask -- why do you care about this?
> > >
> > > Would it actually do any harm if someone instantiated your
> > > base class? If not, then it's probably not worth going out
> > > of your way to prevent it.
> >
> > This is the first little library I try to put together
> > in Python, and it was natural for me to hit it with all
> > the relevant decorations as well as type annotations in
> > order to expose *strict* contracts, plus hopefully have
> > intellisense work all over the place.
>
> I think you are trying to use Python in a way contrary to its nature.
> Python is a dynamically typed language. Its variables don't have types,
> only its objects. And basically everything can be changed at runtime ...

Consider this example:

def abs(x):
return math.sqrt(x.re**2 + x.im**2)

That of course fails if we pass an object not of the correct
(duck) type. But the exception is raised by math.sqrt, which,
properly speaking, beside considerations on usability, is a
private detail of the implementation of abs.

The point is more about what the function semantically
represents and to which point the implementation fulfils
that promise/contract. What is that "abs" really supposed
to provide? Let's say: << 'abs', given a "complex number"
(duck-typed or not), returns its "real number" magnitude >>.
And that's it, not just the doc but really what *type* of
computation it is, and then the mathematics we can apply
to types.

And an effect is that, by constraining the domains (and
codomains) of functions, we make verification of correctness,
the need for defensive coding, as well as for plain testing,
less onerous by orders of magnitude.

So, at least for a function in a public interface, we *must*
add validation (and the doc string):

def abs(x):
"""...as above..."""
if not isinstance(x, ComplexNumber):
raise TypeError(...)
return math.sqrt(x.re**2 + x.im**2)

and then why not rather write:

@typechecked
def abs(x: ComplexNumber) -> RealNumber:
return math.sqrt(x.re**2 + x.im**2)

i.e. checked by the tools statically and/or at runtime,
with, for easy intellisense, the default doc string that
comes from the signature?

More generally, I do think dynamic typing leads to interesting
patterns of reuse, but I think "unconstrained" is a red herring:
ab initio there is a *requirement* to fulfil.

> It now has type annotations, but they are ignored both by the compiler
> and at run-time. They are only for the benefit of linting tools like
> mypy and editor features like intellisense.

For the chronicle, I have meanwhile switched from pylint to
mypy (in VSCode) and things have improved dramatically, from
configurability to intellisense to static type checking working
pretty well. I have meanwhile also realised that the built-in
solutions (the typing module and so on) as well as the tools
supporting them (including those for runtime validation) are
still quite young, so instead of venting frustration (and I
must say so far I think the typing module is a mess), as I
have done in the OP, I should be patient, rather possibly
give a hand...

> > and the whole lot, indeed why even subclass ABC?
>
> Good question. In six years of programming Python, I've never used ABC.
> But then I came from another dynamically typed language to Python.

Back in the day we had a laugh when intellisense was invented...
but now, with the eco-info-system booming with everything and the
contrary of everything (to put it charitably), I think it's about
(formal, as much as possible) verifiability and correctness.

Anyway. my 2c. Sorry for the long post and thanks for the feedback.

Julio
--
https://mail.python.org/mailman/listinfo/python-list

On Fri, Oct 30, 2020 at 1:06 PM Julio Di Egidio <julio@diegidio.name> wrote:
>
> On Sunday, 25 October 2020 20:55:26 UTC+1, Peter J. Holzer wrote:
> > I think you are trying to use Python in a way contrary to its nature.
> > Python is a dynamically typed language. Its variables don't have types,
> > only its objects. And basically everything can be changed at runtime ...
>
> Consider this example:
>
> def abs(x):
> return math.sqrt(x.re**2 + x.im**2)
>
> That of course fails if we pass an object not of the correct
> (duck) type. But the exception is raised by math.sqrt, which,
> properly speaking, beside considerations on usability, is a
> private detail of the implementation of abs.

Not sure what sorts of errors you're expecting, but for the most part,
if it's not a complex number, you should get an AttributeError before
you get into the sqrt. (Although I'm not sure why you're looking for
".re" and ".im" - typo? Python's complex type has ".real" and ".imag",
and you can accept any numeric type that way - even
fractions.Fraction.) It's highly unlikely that you'd have real and
imaginary parts that can be squared and summed, but the result can't
be squirted.

But here's the thing: even if that IS the case, most Python
programmers are fine with getting an AttributeError stating exactly
what the problem is, rather than a TypeError complaining that it has
to be the exact type some other programmer intended. A good example is
dictionaries and dict-like objects; if your code type checks for a
dict, I have to give you a dict, but if you just try to do whichever
operations you need, I can create something more dynamic and give you
that instead.

Python doesn't push for heavy type checking because it's almost
certainly not necessary in the majority of cases. Just use the thing
as you expect it to be, and if there's a problem, your caller can
handle it. That's why we have call stacks.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

On Friday, 30 October 2020 05:09:34 UTC+1, Chris Angelico wrote:
> On Fri, Oct 30, 2020 at 1:06 PM Julio Di Egidio <julio@diegidio.name> wrote:
> > On Sunday, 25 October 2020 20:55:26 UTC+1, Peter J. Holzer wrote:
> > > I think you are trying to use Python in a way contrary to its nature.
> > > Python is a dynamically typed language. Its variables don't have types,
> > > only its objects. And basically everything can be changed at runtime ...
> >
> > Consider this example:
> >
> > def abs(x):
> > return math.sqrt(x.re**2 + x.im**2)
> >
> > That of course fails if we pass an object not of the correct
> > (duck) type. But the exception is raised by math.sqrt, which,
> > properly speaking, beside considerations on usability, is a
> > private detail of the implementation of abs.
>
> Not sure what sorts of errors you're expecting, but for the most part,
> if it's not a complex number, you should get an AttributeError before
> you get into the sqrt.
<snip>

Yes, sorry, I was glossing over an example that was meant
to be just general. I could have written something like:

def abs(x):
...bunch of possibly complex operations...

and that that is completely opaque both to the user of the
function as well as, in my opinion, to its author (code must
speak to speak for itself!).

Not to mention, from the point of view of formal verification,
this is the corresponding annotated version, and it is in fact
worse than useless:

def abs(x: Any) -> Any:
...some code here...

Useless as in plain incorrect: functions written in a totally
unconstrained style are rather pretty much guaranteed not to
accept arbitrary input... and, what's really worse, to be
unsound on part of the input that they do accept: read
undefined behaviour.

> But here's the thing: even if that IS the case, most Python
> programmers are fine with getting an AttributeError stating exactly
> what the problem is, rather than a TypeError complaining that it has
> to be the exact type some other programmer intended. A good example is
> dictionaries and dict-like objects; if your code type checks for a
> dict, I have to give you a dict, but if you just try to do whichever
> operations you need, I can create something more dynamic and give you
> that instead.

I did say "duck-typing or not". I was not talking of restricting
dynamism (python protocols are an implementation of duck typing
and can be used in typing annotations, even generic protocols),
the point is making intentions explicit: to the benefit of the
user of that code, but also to that of the author, as even the
author has to have some "plan" in mind. As well as, eventually,
to the various tools and checkers.

> Python doesn't push for heavy type checking because it's almost
> certainly not necessary in the majority of cases. Just use the thing
> as you expect it to be, and if there's a problem, your caller can
> handle it. That's why we have call stacks.

I am learning Pandas and I can rather assure you that it is an
absolute pain as well as loss of productivity that, whenever I
misuse a function (in spite of reading the docs), I indeed get
a massive stack trace down to the private core, and have to
figure out, sometimes by looking at the source code, what I
actually did wrong and what I should do instead. To the point
that, since I want to become a proficient user, I am ending up
reverse engineering the whole thing... Now, as I said up-thread,
I am not complaining as the whole ecosystem is pretty young,
but here the point is: "by my book" that code is simply called
not production level.

Julio
--
https://mail.python.org/mailman/listinfo/python-list

On Fri, Oct 30, 2020 at 5:16 PM Julio Di Egidio <julio@diegidio.name> wrote:
> Not to mention, from the point of view of formal verification,
> this is the corresponding annotated version, and it is in fact
> worse than useless:
>
> def abs(x: Any) -> Any:
> ...some code here...
>

Useless because, in the absence of annotations or other information
saying otherwise, "Any" is exactly what you'd get. There's no point
whatsoever in annotating that it ... is exactly what it would be
without annotations. Unless you're trying for some sort of rule "all
functions must be annotated", or something, it's best to just not
bother.

> Useless as in plain incorrect: functions written in a totally
> unconstrained style are rather pretty much guaranteed not to
> accept arbitrary input... and, what's really worse, to be
> unsound on part of the input that they do accept: read
> undefined behaviour.

Not quite true. You're assuming that the annotation is the only thing
which determines what a function will accept. Consider that the
function had been written thus:

def abs(x):
"""Calculate the magnitude of a (possibly complex) number"""
return (x.real ** 2 + x.imag ** 2) ** 0.5

The docstring clearly states that it expects a number. But ANY number
will do, so long as it has .real and .imag attributes. Python's core
types all have those; the numpy int, float, and complex types all have
those; any well-behaved numerical type in Python will have these
attributes.

In fact, if you were to annotate this as "x: numbers.Complex", it
would give exactly zero more information. This function requires a
number with real and imag attributes - that's all. If anything, such
an annotation would confuse people into thinking that it *has* to
receive a complex number, which isn't true.

There is nothing undefined here. If you give it (say) a string, you
get a nice easy AttributeError saying that it needs to have a .real
attribute.

> > Python doesn't push for heavy type checking because it's almost
> > certainly not necessary in the majority of cases. Just use the thing
> > as you expect it to be, and if there's a problem, your caller can
> > handle it. That's why we have call stacks.
>
> I am learning Pandas and I can rather assure you that it is an
> absolute pain as well as loss of productivity that, whenever I
> misuse a function (in spite of reading the docs), I indeed get
> a massive stack trace down to the private core, and have to
> figure out, sometimes by looking at the source code, what I
> actually did wrong and what I should do instead. To the point
> that, since I want to become a proficient user, I am ending up
> reverse engineering the whole thing... Now, as I said up-thread,
> I am not complaining as the whole ecosystem is pretty young,
> but here the point is: "by my book" that code is simply called
> not production level.
>

Part of the art of reading exception tracebacks rapidly is locating
which part is in your code and which part is library code. Most of the
time, the bug is at the boundary between those two. There are tools
that can help you with this, or you can develop the skill of quickly
eyeballing a raw traceback and finding the part that's relevant to
you. I don't think this is a consequence of the ecosystem being young
(Pandas has been around for twelve years now, according to Wikipedia),
but more that the Python community doesn't like to destroy good
debugging information and the ability to fully duck-type.

Solve the correct problem. If tracebacks are too hard to read, solve
the traceback problem, don't try to force everything through type
checks.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list