Mailing List Archive

Did you check the ThreadPoolExecutor or the ProcessPoolExecutor? They
won't give you atomic writes unless you add a Lock or a Condition, but
they will execute your code in another thread or process.

https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor
https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor

Keep in mind that Python's threads have a global interpreter lock
(GIL) that prevents full parallelism. Processes work as expected, but
require IPC and pickable objects in and out.

--
Diego Souza
Wespa Intelligent Systems
Rio de Janeiro - Brasil


On Mon, Sep 26, 2022 at 1:01 PM <python-list-request@python.org> wrote:
> From: Axy <axy@declassed.art>
> To: Python List <python-list@python.org>
> Date: Mon, 26 Sep 2022 12:17:47 +0100
> Subject: Asynchronous execution of synchronous functions
> Hi there,
>
> is there a library to call functions in context of a thread? For
> example, as in asyncsqlite which has a thread and a queue I mean has
> anyone generalized such an approach already?
>
> If not, I'll do it myself, no problem.
>
> It's a kind of tiny stuff, like atomicwrites, which is quite difficult
> to dig out with modern search engines that have already rolled down to
> hell a decade ago.
>
> Axy.
--
https://mail.python.org/mailman/listinfo/python-list

On 2022-09-26, Stefan Ram <ram@zedat.fu-berlin.de> wrote:
> So, I wanted to try to download all pages in parallel with
> processes to avoid any GIL effect, while I don't understand
> what the GIL actuall is. But processes didn't work here, so
> I tried threads. This worked and now the total run time is
> down to about 50 seconds.

Downloading things from the network is *extremely* I/O-bound.
So, as you have discovered, the GIL is going to make essentially
no difference whatsoever.
--
https://mail.python.org/mailman/listinfo/python-list

> Did you check the ThreadPoolExecutor or the ProcessPoolExecutor? They
> won't give you atomic writes unless you add a Lock or a Condition, but
> they will execute your code in another thread or process.

Yes, I did, but they are too complicated to use. I'd like something for
humans, such as

asynchronizer = InThreadExecutor()

result = await asynchronizer.run(myfunc, myargs, mykwargs)


and I almost implemented that borrowing code from asyncsqlite (doves fly
slowly, electric mail is even slower :)), but...

> Keep in mind that Python's threads have a global interpreter lock
> (GIL) that prevents full parallelism. Processes work as expected, but
> require IPC and pickable objects in and out.

yes, that became a problem.

So, I revoke my question. Went out to redesign the whole approach.

Thanks for reply!

Axy.
--
https://mail.python.org/mailman/listinfo/python-list

On Tue, Sep 27, 2022 at 3:19 AM Axy via Python-list
<python-list@python.org> wrote:
>
> > Did you check the ThreadPoolExecutor or the ProcessPoolExecutor? They
> > won't give you atomic writes unless you add a Lock or a Condition, but
> > they will execute your code in another thread or process.
>
> Yes, I did, but they are too complicated to use. I'd like something for
> humans, such as
>
> asynchronizer = InThreadExecutor()
>
> result = await asynchronizer.run(myfunc, myargs, mykwargs)
>

There are two convenient APIs in asyncio. asyncio.to_thread and run_in_executor.

to_thread creates and destroy a dedicated thread for the function.
run_in_executor uses threadpool owned by loop.

https://docs.python.org/3/library/asyncio-task.html?highlight=to_thread#asyncio.to_thread
https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor

>
> and I almost implemented that borrowing code from asyncsqlite (doves fly
> slowly, electric mail is even slower :)), but...
>
> > Keep in mind that Python's threads have a global interpreter lock
> > (GIL) that prevents full parallelism. Processes work as expected, but
> > require IPC and pickable objects in and out.
>
> yes, that became a problem.
>
> So, I revoke my question. Went out to redesign the whole approach.
>
> Thanks for reply!
>
> Axy.
> --
> https://mail.python.org/mailman/listinfo/python-list



--
Inada Naoki <songofacandy@gmail.com>
--
https://mail.python.org/mailman/listinfo/python-list