No amount of syntax will give me SMP-multithreading in Perl

Do you mean perl 5.005 threads?

IIRC, the conclusion we got from there is that it is not doable without rewamping the runtime completely.

Perl internal data structures were never designed for atomic access so locks need to be used to serialize their access. You could do as Python, have a global lock (bye bye SMP!) or require the programmer to lock by hand (that was the 5.005 approach), but that didn't work. Most modules failed to work in multithreading environments, usually because of the perl internals becoming corrupted... XS code was even worse.

Perl 5.6 threads was a compromise where everything is cloned so that there is no interference between threads and whenever you want to share something, you have to say it explicitly and an intermediate (and quite inefficient) layer that does the locking and serializes access is set up by the runtime.

So, in conclusion, do you want SMP in Perl 5? then write a new runtime! and forget about compatibility with any XS module!!!

Would it be worth the trouble? IMO, well, maybe yes. There are so many things in Perl 5 that need fixing that could only be fixed by using a new runtime!!!

But anyway, who is gonna do it?

Update: oops, s/5\.6/5.005/g; s/5\.8/5.6/g;. I knew I had it wrong because I can still remember my excitment trying the threads support with LWP and getting SIGSEGVs all the time when perl 5.005 was released 22 years and two days ago! Just one of those things than randomly stick in your mind!

> Perl 5.8 threads was a compromise where everything is cloned so that there is no interference between threads and whenever you want to share something, you have to say it explicitly and an intermediate (and quite inefficient) layer that does the locking and serializes access is set up by the runtime.

I'm by far no expert here, but I remember seeing benchmarks where the Perl approach actually did beat Python's GIL in terms of performance.

But benchmarks are like statistics, the author often only measures his own prejudices ... ;-)

Cheers Rolf
_{(addicted to the Perl Programming Language :)
Wikisyntax for the Monastery}

I'm by far no expert here, but I remember seeing benchmarks where the Perl approach actually did beat Python's GIL in terms of performance.

Different trade-offs: Python thread creation is cheap, but then they don't run in parallel except when performing I/O operations. Data is shared and there is no overhead for using threads.

Perl on the other hand, has a very costly thread creation, both in terms of CPU and memory. But then they run in parallel using all the available CPUs. Data is only shared between threads when explicitly requested by the programmer and accessing the shared data has an important overhead.

A purist could pop in and say to you "no, I want Perl to just be P.E.R.L. and only that, no fancy additions."

I don't agree with that view but we should not forget what Perl was created for.

Asking for shared memory alone is vague. Asking for *efficient* shared memory is much more accurate. Only then we can see who is "swimming naked" ("You only find out who is swimming naked when the tide goes out" hehe, and let the source remain nameless) when it comes to comparing to other systems. Although I do not dispute that Perl's shared memory is inefficient.

For the realist, if a solution must involve Perl, perhaps it should not be as fine-grained as to require sharing the million variables of a parallel algorithm. A coarse-grain implementation could be more suitable for Perl (being further away from hardware, compared to C, and this is the reason we all enjoy other perks!). That is, limit the amount of what is shared or access it in larger chunks of raw data where the inefficient lock mechanism does ot matter much. On the other hand, MMAPing a raw-memory buffer and sharing it, is easy but each thread interpreting it as a Perl nested data structure or object will take some efficiency away. Interpreting it as 100 such data structures in 100 threads it will be much slower. And then locking each of these.

Additionally, there is the other route to harnessing state-of-the-art computer science developments: by using extenral libraries. For example the excellent implementation of AI::MXNet by Sergey Kolychev. Which is still a well-kept secret (I think? Anybody using it? Although regularly update. Thanks Sergey). Also see his relevant post http://blogs.perl.org/users/sergey_kolychev/2017/02/machine-learning-in-perl.html and my brief introduction to using it for very simple Dataflow programming Dataflow programming on CPU and GPU using AI::MXNet even on a GPU.

These are great doors opened from within Perl. Fair enough, not what exactly you are asking for: transparent shmem-smp. But just write your algorithm as a Graph and let MXnet worry about parallelising it or even having the GPU solve it.

Perhaps setting a concrete programming challenge can show the distance between practice and my theoretical blabber above?

bw, bliako

I was not being entirely serious in my comment (see the smiley at the end). That said, I'll still give it a go to answer your statements

No amount of syntax will give me SMP-multithreading in Perl (preemptively: any type of async is not even close to multithreading)

use threads;? Unless I'm missing something, which is likely here since the answer seems so simple.

On top of that, in order to write the above I would need a proper, marshal-less, shared memory implementation.

I'm not sure what you mean by unmarshalled shared memory, but then I'm not a full-time programmer, so this is probably my fault. There is PerlIO::mmap, the SysV shared memory modules, or a bunch of mmap modules on CPAN that might be usable for a generic blob of memory. I would guess that at least one of these would be usable, if not necessarily efficient or pleasant.

Of course, it would be crazy to try to literally translate between two different programming languages, much like literally translating between spoken languages, so YMMV.