Mailing List Archive

On Tue, 10 May 2022 at 16:31, Barney Stratford
<barney_stratford@fastmail.fm> wrote:
>
> Hello all.
>
> It occurs to me that creating threads in Python is more awkward than it needs to be. Every time I want to start a new thread, I end up writing the same thing over and over again:
>
> def target(*args, **kwds):
> ...
> t = threading.Thread(target = target, args = <something>, kwargs= <something>)
> t.start()
>
> This becomes especially repetitive when calling a target function that only makes sense when run in a new thread, such as a timer.
>
> In my own code, I’ve taken to decorating functions that always run in new threads. Then I can call the function using the usual function call syntax, and have the new thread returned to me. With the decorator, the code reads instead:
>
> @threaded
> def target(*args, **kwds):
> …
> t = target(*args, **kwds)
>
> This has a couple of advantages. I don’t have to import the threading module all over my code. I can use the neater syntax of function calls. The function’s definition makes it clear it’s returning a new thread since it’s decorated. It gets the plumbing out of the way so I can concentrate more on what my code does and less in how it does it.
>
> It feels like the right place for this decorator is the standard library, so I’ve created PR #91878 for it. @rhettinger suggests that this is a bit premature, and that we should discuss it here first. Thoughts?

I like this. I'd probably use it if it existed. Your example isn't
particularly compelling, though - calling the function "target" makes
sense when it's the target of the thread, but not so much when it's
the thread itself. Could you give a motivating example based on real
code, rather than just the "this is what the syntax would look like"
as you did above? Ideally showing what it would look like with and
without the decorator?

Things I *think* i get, but I'd like to see clearly in an example, include:

1. Does t = target(...) start the thread? I think it does.
2. Is it possible to create daemon threads? @threaded(daemon=True)
seems plausible. I suspect your PR doesn't have this, but it wouldn't
be difficult to add. But I don't actually know if it's worth adding,
TBH.
3. Can you create multiple threads for the same function? I assume t1,
t2, t3 = target(arg1), target(arg2), target(arg3) would work.

I'm probably being dense here - I'm pretty sure your proposal actually
is as simple as it looks (which is a good thing!) and I'm just
over-complicating things in my head.

Paul
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FVETRDHASCD4ZYJ5YQG77LTG7R6TLIZO/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, May 10, 2022 at 1:20 PM Paul Moore <p.f.moore@gmail.com> wrote:

> On Tue, 10 May 2022 at 16:31, Barney Stratford
> <barney_stratford@fastmail.fm> wrote:
> >
> > Hello all.
> >
> > It occurs to me that creating threads in Python is more awkward than it
> needs to be. Every time I want to start a new thread, I end up writing the
> same thing over and over again:
> >
> > def target(*args, **kwds):
> > ...
> > t = threading.Thread(target = target, args = <something>, kwargs=
> <something>)
> > t.start()
> >
> > This becomes especially repetitive when calling a target function that
> only makes sense when run in a new thread, such as a timer.
> >
> > In my own code, I’ve taken to decorating functions that always run in
> new threads. Then I can call the function using the usual function call
> syntax, and have the new thread returned to me. With the decorator, the
> code reads instead:
> >
> > @threaded
> > def target(*args, **kwds):
> > …
> > t = target(*args, **kwds)
> >
> > This has a couple of advantages. I don’t have to import the threading
> module all over my code. I can use the neater syntax of function calls. The
> function’s definition makes it clear it’s returning a new thread since it’s
> decorated. It gets the plumbing out of the way so I can concentrate more on
> what my code does and less in how it does it.
> >
> > It feels like the right place for this decorator is the standard
> library, so I’ve created PR #91878 for it. @rhettinger suggests that this
> is a bit premature, and that we should discuss it here first. Thoughts?
>
> I like this. I'd probably use it if it existed. Your example isn't
> particularly compelling, though - calling the function "target" makes
> sense when it's the target of the thread, but not so much when it's
> the thread itself. Could you give a motivating example based on real
> code, rather than just the "this is what the syntax would look like"
> as you did above? Ideally showing what it would look like with and
> without the decorator?
>
> Things I *think* i get, but I'd like to see clearly in an example, include:
>
> 1. Does t = target(...) start the thread? I think it does.
> 2. Is it possible to create daemon threads? @threaded(daemon=True)
> seems plausible. I suspect your PR doesn't have this, but it wouldn't
> be difficult to add. But I don't actually know if it's worth adding,
> TBH.
> 3. Can you create multiple threads for the same function? I assume t1,
> t2, t3 = target(arg1), target(arg2), target(arg3) would work.
>
> I'm probably being dense here - I'm pretty sure your proposal actually
> is as simple as it looks (which is a good thing!) and I'm just
> over-complicating things in my head.
>
> Things that come to mind that should be considered here:
1) not every two liner function should be added to the stdlib
and, on the other hand,
2) asyncio features `loop.run_in_executor`
which does essentially the same, with more care for what happens
in "real world" code, but is only usable in async code.

all in all it can be useful for quick and dirty code - and maybe
something more elaborate allowing control on whether to create
daemon threads, or maybe something that will deal with
concurrent.future.executors
as well, and allow access to the function call results

All in all, I find it a nice snippet, but one that is not aligned with
the trend Python has been taking over the last years,
facilitating the writing of "quick and dirty" scripts in contrast
with application code that will take care of all possible corner-cases.

As a good example of what I am talking about, we've seen the contrary
movement take place, long ago, when
"os.popen" was superseded by "subprocess.*": I think up to today people
had not catched up with the far more complicated API of the latter.

All in all, I'd eventually use this "@threaded" if it was there.



> Paul
>
> js
-><-

> 1. Does t = target(...) start the thread? I think it does.
I think it does too. In the commonest use case, immediately after creating a thread, you start it. And if you want to delay the thread but still use the decorator, then you can do that explicitly with some locks. In fact, it’s probably better to explicitly delay execution than have hidden requirements concerning the timing of thread creation and startup.

> 2. Is it possible to create daemon threads?
Not at the moment. I did think about this, but felt that simpler is better. Like you say, it’d be easy to add. In fact, I might just go ahead and add it to the PR in a bit. The simplest way to do it is probably to define a second decorator for daemonic threads.

> 3. Can you create multiple threads for the same function? I assume t1,
> t2, t3 = target(arg1), target(arg2), target(arg3) would work.
That’s exactly what I had in mind. Make it so that thread creation and function call look exactly alike. You can call a function as many times as you want with whatever args you want, and you can create threads as often as you want with whatever args you want.

There isn’t a single use case where the decorator is particularly compelling; rather, it’s syntactic sugar to hide the mechanism of thread creation so that code reads better. I could give some examples of where I used it recently if you want, but I don’t think it would be terribly illuminating. More useful might be to look at the problem from the opposite perspective: would anyone like to write

result = call_function(target = foo, args = (42,), kwargs = {“bar”: 60})

in preference to

result = foo(42, bar = 60)

Cheers,
Barney.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YESYSRUCR666HASBXYHO2SI6AI6T5MEW/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, May 10, 2022 at 10:34 AM Barney Stratford <
barney_stratford@fastmail.fm> wrote:

> > 1. Does t = target(...) start the thread? I think it does.
> I think it does too. In the commonest use case, immediately after creating
> a thread, you start it. And if you want to delay the thread but still use
> the decorator, then you can do that explicitly with some locks. In fact,
> it’s probably better to explicitly delay execution than have hidden
> requirements concerning the timing of thread creation and startup.
>
> > 2. Is it possible to create daemon threads?
> Not at the moment. I did think about this, but felt that simpler is
> better. Like you say, it’d be easy to add. In fact, I might just go ahead
> and add it to the PR in a bit. The simplest way to do it is probably to
> define a second decorator for daemonic threads.
>

If this is even to be added (i personally lean -1 on it), I suggest
intentionally not supporting daemon threads. We should not encourage them
to be used, they were a misfeature that in hindsight we should never have
created. Daemon threads can lead to very bad surprises upon interpreter
finalization - an unfixable problem given how daemon threads are defined to
behave.

> 3. Can you create multiple threads for the same function? I assume t1,
> > t2, t3 = target(arg1), target(arg2), target(arg3) would work.
> That’s exactly what I had in mind. Make it so that thread creation and
> function call look exactly alike. You can call a function as many times as
> you want with whatever args you want, and you can create threads as often
> as you want with whatever args you want.
>
> There isn’t a single use case where the decorator is particularly
> compelling; rather, it’s syntactic sugar to hide the mechanism of thread
> creation so that code reads better.
>

This is my take as well. I don't like calling code to hide the fact that a
thread is being spawned. Use this decorator and if you fail to give the
callable a name communicating that it spawns and returns a thread, you will
have surprised readers of the calling code.

A nicer design pattern is to explicitly manage threads. Use
concurrent.futures.ThreadPoolExecutor. Or use the async stuff that Joao
mentioned or similar libraries. I think we already provide decent batteries
with the threading APIs.

-gps

On Tue, 10 May 2022 16:12:13 +0100
Barney Stratford <barney_stratford@fastmail.fm> wrote:
> Hello all.
>
> It occurs to me that creating threads in Python is more awkward than it needs to be. Every time I want to start a new thread, I end up writing the same thing over and over again:
>
> def target(*args, **kwds):
> ...
> t = threading.Thread(target = target, args = <something>, kwargs= <something>)
> t.start()
>
> This becomes especially repetitive when calling a target function that only makes sense when run in a new thread, such as a timer.
>
> In my own code, I’ve taken to decorating functions that always run in new threads. Then I can call the function using the usual function call syntax, and have the new thread returned to me. With the decorator, the code reads instead:
>
> @threaded
> def target(*args, **kwds):
> …
> t = target(*args, **kwds)
>
> This has a couple of advantages. I don’t have to import the threading module all over my code. I can use the neater syntax of function calls. The function’s definition makes it clear it’s returning a new thread since it’s decorated. It gets the plumbing out of the way so I can concentrate more on what my code does and less in how it does it.
>
> It feels like the right place for this decorator is the standard library, so I’ve created PR #91878 for it. @rhettinger suggests that this is a bit premature, and that we should discuss it here first. Thoughts?

This seems like an attractive nuisance. Creating threads comes with its
own constraints and subtleties. I don't think it really helps users to
hide it behind a "regular" function call.

Like Greg I'm leaning towards -1 on this.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7XXVN7GOBWUFORMDLHQGLFIYZH23IXA2/
Code of Conduct: http://python.org/psf/codeofconduct/

It seems like the consensus is that this is a good idea, but it’s the wrong good idea. Should I cancel the PR or should we try to make it into a better good idea?

Cheers,
Barney.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Z4AWYWNHJGOSBBU6DFV3ZS26WWQIILMW/
Code of Conduct: http://python.org/psf/codeofconduct/

It’s definitely too early for a PR, so if you already have one (I didn’t
see one linked to this thread) please close it.

Then once we’ve bikeshedded the right good idea you can start a new PR.

On Thu, May 12, 2022 at 12:21 Barney Stratford <barney_stratford@fastmail.fm>
wrote:

> It seems like the consensus is that this is a good idea, but it’s the
> wrong good idea. Should I cancel the PR or should we try to make it into a
> better good idea?
>
> Cheers,
> Barney.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/Z4AWYWNHJGOSBBU6DFV3ZS26WWQIILMW/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
--Guido (mobile)

On 12May2022 20:17, Barney Stratford <barney_stratford@fastmail.fm> wrote:
>It seems like the consensus is that this is a good idea, but it’s the
>wrong good idea. Should I cancel the PR or should we try to make it
>into a better good idea?

Why not shift slightly? As remarked, having a function automatically
spawn threads can be confusing, because spawning a thread has
implications for both the thread code itself and for the person calling
the function. The caller might find that confusing or complex.

Personally, my approach has been a tiny shim function named "bg" in my
personal kit, to make it easy to spawn a regular function in a thread:

T = bg(func, args....) # returns a running Thread

An approach I've also seen is Celery's one of decorating a function with
attributes which can dispatch it as a task, roughly:

@task
def regular_function(.....)
...

and it can be dispatched by saying regular_function.defer(......) and
variations.

Some downsides to decorators include:
- only decorated functions can be kicked off "automatically as threads";
that could be a good thing too
- the decorator wires in the dispatch mechanism: your decorator spawns a
thread, the Celery @task queues to a Celery task queue, and so forth

So my personal inclination is to provide an easy to use shim for the
caller, not the function. Eg:

from cs.threads import bg
.....
T = bg(func, args.....)

or:

from cs.threads import bg as bg_thread
from cs.later import bg as bg_later
.....
T = bg_thread(func, args......) # run in a Thread
.....
R = bg_later(func, args.......) # hand off to a Later, get a Result for collection
......
bg = bg_later # specify the queuing system
......
... do stuff via bg(), the chosen queuing system ...

or whatever other queuing system you might be using. The idea here is to
make it easy to submit a function to any of several things rather than
decorating the function itself to submit to a now-hardwired thing.

Just things to consider....

Cheers,
Cameron Simpson <cs@cskk.id.au>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BX6J7BKEXMNQFARDKMKDSKAS7VTOVPIP/
Code of Conduct: http://python.org/psf/codeofconduct/

10.05.22 18:12, Barney Stratford ????:
> This has a couple of advantages. I don’t have to import the threading module all over my code. I can use the neater syntax of function calls. The function’s definition makes it clear it’s returning a new thread since it’s decorated. It gets the plumbing out of the way so I an concentrate more on what my code does and less in how it does it.

I do not see advantages. You definitely need to import the threading
module or other module depending on the threading module in which you
define the decorator. And you need to decorate the function. It will not
save you a line of code.

If you need to run a lot of functions in threads, note that a thread has
some starting cost. Consider using concurrent.futures.ThreadPoolExecutor.

If you only run few long living threads, the syntax of starting them
does not matter, in comparison with the rest of your code.

Also, it looks wrong to me to mix two different things: what code to
execute and how to run it. If we need a neater syntax, I would propose:

t = threading.start_thread(func, *args, **kwargs)

But I am not sure that it is worth to add such three-line function in
the stdlib.

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RBIAWP2J3NTYMLIS3W6CKX44G5QIBXAU/
Code of Conduct: http://python.org/psf/codeofconduct/

Thinking about it, it's actually a fine design to have a decorator that
turns a regular function into one that starts a thread -- from the caller's
POV it's no different than having a function that explicitly starts a
thread, and it could be a nice shorthand if you do this all the time. (You
have to document it in either case.)

That said, I don't think we need this 3-line decorator in the stdlib.

--Guido


On Thu, May 12, 2022 at 10:40 PM Serhiy Storchaka <storchaka@gmail.com>
wrote:

> 10.05.22 18:12, Barney Stratford ????:
> > This has a couple of advantages. I don’t have to import the threading
> module all over my code. I can use the neater syntax of function calls. The
> function’s definition makes it clear it’s returning a new thread since it’s
> decorated. It gets the plumbing out of the way so I an concentrate more on
> what my code does and less in how it does it.
>
> I do not see advantages. You definitely need to import the threading
> module or other module depending on the threading module in which you
> define the decorator. And you need to decorate the function. It will not
> save you a line of code.
>
> If you need to run a lot of functions in threads, note that a thread has
> some starting cost. Consider using concurrent.futures.ThreadPoolExecutor.
>
> If you only run few long living threads, the syntax of starting them
> does not matter, in comparison with the rest of your code.
>
> Also, it looks wrong to me to mix two different things: what code to
> execute and how to run it. If we need a neater syntax, I would propose:
>
> t = threading.start_thread(func, *args, **kwargs)
>
> But I am not sure that it is worth to add such three-line function in
> the stdlib.
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RBIAWP2J3NTYMLIS3W6CKX44G5QIBXAU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>