In case it's of use, I wrote a couple of scripts to run a command, killing it if it took more than a specified timeout, and reporting how long it took to run:
Update: Using this node nowadays for links on Threading and Concurrency.
Perl Documentation
CPAN
Perl monk marioroy has produced some excellent CPAN modules in this domain:
- MCE by marioroy (Many-Core Engine for Perl providing parallel processing capabilities - can be configured to use processes or threads (via use_threads parameter), by default MCE spawns child processes on UNIX platforms and threads on Windows)
- MCE::Grep by marioroy
- Mutex by marioroy
- MCE::Shared by marioroy
June 2024: From chatterbox, we learn that marioroy is planning MCE2:
- MCE2 will not be compatible with MCE, hence new name.
- MCE was a playground, learning parallel.
- Before MCE2, I learned parallel in other languages, OpenMP, Python Multiprocessing, ...
- OpenMP parallel lacks OMP->next and OMP->last. MCE has this.
- Python Semaphore bogs down the more workers. It lacks a limiter in the design.
See also:
Event-driven Programming
Event-driven programming is an attractive alternative to threads, especially when using scripting languages.
For Perl, Mojolicious seems to be the most popular nowadays, as described at:
See also:
Chunking
As noted by marioroy, chunking complements event-driven programming.
Some further chunking examples from marioroy:
See also: event-driven references in the Wikipedia References section below.
TLS/SSL Related
On CPAN:
Some Perl Monks Threading and Concurrency Nodes
- IPC::Open, Parallel::ForkManager, or Sockets::IO for parallelizing? by mldvx4 (Sep 2023) - replies mention Mojolicious, Mojo::Promise, Net::Async::HTTP and Future (by Paul Evans), LWP::Parallel, MCE, Thread::Queue, Forking::Amazing, ...
- Perl and Threading by scorpion7 (Nov 2023) - panic: del_backref, *svp=0 phase=DESTRUCT refcnt=3 during global destruction; choroba reply recommends Thread::Queue
Windows and Strawberry Perl
- Strawberry Perl (wikipedia)
- Mingw-w64 (wikipedia) - gcc-based C/C++ compiler used by Strawberry Perl (Note: MSYS2 ("minimal system 2") is a software distribution and a development platform for Microsoft Windows, based on Mingw-w64 and Cygwin ... MSYS2 and its bash environment is used by Git)
- Cygwin (wikipedia) - a Unix-like environment and command-line interface for Microsoft Windows (uses cross-compilers from Mingw-w64 project)
- How is fork() implemented in Cygwin? - Cygwin fork() essentially works like a non-copy on write version of fork() (like old Unix versions used to do). Because of this it can be a little slow. In most cases, you are better off using the spawn family of calls if possible.
- Strawberry Perl (official site)
- Strawberry Perl (github)
Parallel::ForkManager
Promises and Futures
- Future by Paul Evans
- Promises by Yanick Champoux (compares to Scala Promises and Futures)
- C++ std::promise
- C++ std::future - future and promise enable transfer of a value between two tasks without explicit use of a lock (the "system" implements the transfer efficiently); when a task wants to pass a value to another, it puts the value into a promise; the system makes that value appear in the corresponding future; promise provides simple put operations to match future's get
Signals
C++ Multi-Threading
- NVIDIA HPC SDK Version 24.3 Documentation (nvc++ is a C++17 compiler for NVIDIA GPUs and AMD, Intel, OpenPOWER, and Arm CPUs. It invokes the C++ compiler, assembler, and linker for the target processors with options derived from its command line arguments. nvc++ supports ISO C++17, supports GPU and multicore CPU programming with C++17 parallel algorithms, OpenACC, and OpenMP)
- The three compilers used by marioroy in Long List is Long are g++, clang++, nvc++. I also used Google abseil library.
Unit Testing Concurrent Code
Test-driven development, a practice enabling developers
to detect bugs early by incorporating unit testing into the development process, has become
wide-spread, but it has only been effective for programs with a single thread of control.
The order of operations in different threads is essentially non-deterministic, making it
more complicated to reason about program properties in concurrent programs than in
single-threaded programs.
-- from a recent PhD proposal to develop a concurrent testing framework
See the "Testing Concurrent Software References" section below for more references
in this active area of research.
Though I haven't used any of these tools yet, I'd be interested to hear from folks who have
or who have general advice and tips on how to troubleshoot and fix complex concurrency-related bugs.
In particular, I'm not aware of any Perl-based concurrent testing frameworks.
In practice, the most effective, if crude, method I've found for dealing
with nasty concurrency bugs is good tracing code at just the right places combined
with understanding and reasoning about the code, performing experiments,
and "thinking like a detective".
One especially useful experiment (mentioned in
Clean Code)
is to add "jiggle points" at critical places in your concurrent code and have
the jiggle point either do nothing, yield, or sleep for a short interval.
There are more sophisticated tools available, for example IBM's ConTest,
that use this approach to flush out bugs in concurrent code.
Testing Concurrent Software References
Wikipedia References
See Also
Updated: Extra references added long after the original reply was made.