perl5-porters:
Thank you everyone for your thoughts. :-)
On 7/24/21 10:48 PM, Oodler 577 via perl5-porters wrote:> * David
Christensen <dpchrist@holgerdanske.com> [2021-07-24 16:08:58 -0700]:
>> Whatever is done, I have been wanting a Perl concurrency mailing list
>> for many years.
> Regarding the discussion of "threads" - I'd love to have one.
I'm fine talking about threading, concurrency, etc., here. It is useful
to have archives, so a cc: model does not appeal.
On 7/25/21 4:10 AM, Leon Timmermans wrote:
> On Sun, Jul 25, 2021 at 1:09 AM David Christensen wrote:
>> For example, I have discovered that threads is not fully compatible
>> with Data::Dumper and Capture::Tiny.
>
> The former sounds like a bug.
https://github.com/Perl/perl5/issues/19014 > The latter deals with a process-global resource (standard IO), so
> will inherently have some funkiness.
The following Capture::Tiny documentation led directly to my previous
"self-fulfilling prophecy" statement:
2021-08-01 15:51:56 dpchrist@dipsy ~
$ perldoc Capture::Tiny | head -n 262 | tail -n 4
Using threads
Filehandles are global. Mixing up I/O and captures in different threads
without coordination is going to cause problems. Besides, threads are
officially discouraged.
Threading was a core Perl feature. CPAN modules were supposed to
support threading. Then threads were declared "officially discouraged".
What was supposed to replace threads?
Now my choices are 1) make Capture::Tiny thread-safe, or 2) write a
sawed-off thread-safe replacement. Madness:
http://winestockwebdesign.com/Essays/Lisp_Curse.html =~ s/lisp/perl/ig
> IMO what we need is to research other threading models on top of the
> primitives that we have (and if needed extend those primitives).
> threads::lite was my first attempt at that, though I've started a
> successor.
AIUI threads and threads::shared were the canonical Perl threading
primitives, as demonstrated by their inclusion in the Camel book. I am
aware of the following documentation, which describes the threading
model they implement:
"Programming Perl" [1], pages 1035-1036, 1078.
https://metacpan.org/pod/Thread https://metacpan.org/pod/threads https://metacpan.org/pod/threads::shared https://metacpan.org/dist/perl/view/pod/perlthrtut.pod "Network Programming with Perl" [2] identifies additional primitives and
non-primitives that can relate to concurrency.
On 7/25/21 8:26 AM, Oodler 577 via perl5-porters wrote:
> The ideas of "practical" concurrency is way more important to
> perl/Perl's future than virtually any other issue.
Which feature is most important is debatable.
I would put "Conceptual Integrity" and "Programming Systems Product" at
the top of the list, per Brooks [3].
AIUI threads were pushed to the forefront ~20 years ago, when multi-core
and hyper-threading microprocessors became commodity items. As always,
user expectations have risen since then. And, as always, developers are
expected to create software that can make full use of the latest
hardware. So, if Perl wants developers, Perl must give developers the
tools they need to meet user expectations.
But, threading is just one way to achieve concurrency on a single
computer. The traditional Unix way is fork/ exec. I can simulate
concurrency in a single thread by invoking short, non-blocking
subroutines in a tight loop (e.g. polling, asynchronous I/O, select),
but that approach is brittle; my application becomes a scheduler.
Perhaps I could write a Perl program, launch it into the background with
a 1-argument system() or qx(), and communicate with it over non-blocking
pipes or sockets. Are there any other core Perl mechanisms to achieve
concurrency within a single computer?
Are there any core Perl mechanisms to achieve real concurrency on a
network of computers?
> ... openmp ...
https://www.openmp.org/ https://www.mpi-forum.org/ Perl modules/ distributions for these could be useful.
> Off the top of my head there are many topics ripe for discussion:
>
> * ithreads
> * forking based things
> * async
> * "real" threads via OpenMP/pthreads bindings
> * perl vs Perl memory models
> * atomicity/sequential consistency
I'm happy to discuss any or all of them. :-)
On 7/28/21 8:26 AM, Ovid wrote:
> Give me:
>
> - Concurrency
> - OOP
> - Signatures
> - Some way of defining enforceable types
>
> With that, Perl can come out of the gate swinging.
I would like:
- Conceptual Integrity
- Programming Systems Product -- functional specification, design/
construction specification, test/ validation specification, etc., with
everything documented and under configuration management
- Multi-paradigm -- imperative, functional, object-oriented,
declarative, meta-programming (includes signatures and types)
- Multi-Platform -- add Android and iOS
- Memory safe/ garbage collected
- Secure
- Concurrent/ Distributed/ Networked
- Macros
- Programs can be compiled and statically linked into stand-alone
executeables
On 7/28/21 10:32 AM, shmem wrote:> Last time I looked there was MCE
which somehow looks meaningful.
https://metacpan.org/dist/MCE/view/lib/MCE.pod I saw MCE. And POE. And Coro. And PDL. I put my effort into threads
and threads::shared years ago (pre-discouraged) because they looked like
the foundation of threading in Perl, they were the most similar to my
previous C/ assembly experiences, and I thought I could build useful
stuff with them. These reasons have proven to be mostly correct. The
problems I have now are dealing with non-thread-safe code throughout the
rest of Perl and the knowledge that threads are on the chopping block.
On 7/28/21 11:05 AM, Oodler 577 via perl5-porters wrote:
> What we need is a door into the sharing of memory; if not among
> threads (not my goal), surely amoung fork'd processes. Coupled with
> atomic synchronization primatives (which we have in some form via
> sysopen), this would provide us a path to utilize forks+shared memory.
I read about shared memory on Unix via C years ago, but have not needed
or used it. STFW I see existing modules/ distributions that do shared
memory. You don't see anything you like?
https://metacpan.org/search?q=shared+memory On 7/28/21 8:59 PM, Yuki Kimoto wrote:
> The word "concurrency" is always ambiguous when people talk about
> "concurrency".
https://en.wikipedia.org/wiki/Concurrency_(computer_science)
> I hear the this conversation "Perl doesn't have concurrency, so Perl
> can't use CPUs in parallel"
>
> But, I know the pre-fork model Perl web applications are using CPUs in
> parallel.
>
> I think we need to properly classify concurrency(fork, thread, I/O,
> CPU(OpenMP), GPU, coroutine, async/await syntax) before we say we want
> to add concurrency.
Perl has several forms of concurrency. Unfortunately, a
high-visibility, core form -- threading -- has not worked out and is now
unmaintained.
On 7/28/21 10:12 PM, Oodler 577 wrote:
> Typically, OpenMP is accomplished at low level with pthreads.
AIUI POSIX Threads (pthreads) are not supported on all of the platforms
that support Perl. So, Perl was required invented its own
multi-platform threading model.
I suspect OpenMP will similarly lose to multi-platform. Can OpenMP be
put into a Perl module that works on the relevant subset of platforms?
For that matter, can POSIX Threads be put into a Perl module that works
on the relevant subset of platforms?
Have these been tried already (see next)?
On 7/29/21 10:34 AM, Leon Timmermans wrote:
> In practice, all ideas have either been already done, or are very
> difficult (or even impossible) to do.
Please let us know if you see the discussion headed that way.
On 7/29/21 10:34 AM, Leon Timmermans wrote:
> I agree [concurrency is] important, that's why I'm actually working on
> the issue. To be precise, an ithreads based (so can actually use
> multiple processors) continuation passing style (same model Go uses
> for its channels) module.
https://en.wikipedia.org/wiki/Continuation-passing_style STFW again, I see existing modules/ distributions with continuations.
You do not see anything you like?
https://metacpan.org/search?size=20&q=continuation Can continuations be abstracted out, become a Perl primitive, and used
throughout?
On 7/29/21 10:47 AM, Oodler 577 wrote:
> I am doing 2 things:
>
> 1. continuing to develop the idea of the "perl" memory model
> (uniprocess) and the Perl memory model (semantically, admits
> concurrency). See Sub::Genius.
>
> 2. approaching concurrency from the shared memory perspective by
> exploring various ideas in order to determine how OpenMP can be most
> naturally leveraged on an SMP machine. See OpenMP::Environment and
> Alien::OpenMP.
https://metacpan.org/pod/Sub::Genius https://metacpan.org/pod/OpenMP::Environment https://metacpan.org/pod/Alien::OpenMP I have been searching for a general model of concurrency for many years.
Flow-based programming (FBP) has caught my interest:
https://en.wikipedia.org/wiki/Flow-based_programming STFW I did not see anything I liked:
https://metacpan.org/search?q=flow https://metacpan.org/search?q=fbp I have built a simplified FBP library in Perl using threads and a
re-write of Thread::Queue (with multiple readers and writers, and
coordinated open and close). It can run FBP programs concurrently on
multi-core and/or hyper-threaded processors. It should be able to run a
FBP program concurrently on multi-processor computers (when I have such
a machine for testing). I am now researching extensions so that it can
run a FBP program concurrently on a network of multiple computers.
David
References:
[1] Tom Christiansen, et al, 2012, "Programming Perl", 4 e,
https://www.oreilly.com/library/view/programming-perl-4th/9781449321451/ [2] Lincoln D. Stein, 2000, "Network Programming with Perl",
https://www.oreilly.com/library/view/network-programming-with/0201615711/ [3] Frederick P. Brooks, 1996, "Mythical Man-Month, The: Essays on
Software Engineering, Anniversary Edition," 2 e.,
https://www.pearson.com/us/higher-education/program/Brooks-Mythical-Man-Month-The-Essays-on-Software-Engineering-Anniversary-Edition-2nd-Edition/PGM172844.html