in reply to parallel programming and MCE - hubris and the frame

Hi Discipulus, you give me more credit than I deserve, in as far as being an expert on threads and forks. I'm just a hacker. :-)

However, to continue sharing my observations, there is one thing you overlook in the whole discussion. It's the problem of i/o to the disk or network when writing programs for parallel processing. I/O is very slow compared to how fast the cpu runs, and unless you are running an 8-core Xeon machine, with a T1 internet line, most of the cpu time in the program is wasted in waiting for the i/o to complete. That fact makes parallel processing a tricky proposition because you may waste more resources trying to fork and spawn threads than you gain in overall program efficiency. Newer software designs like node.js and Perl6 are designed to address this issue. The general solution is to run single threaded "event-loop" programs which handle the i/o with "promises" and callbacks. This is essentially how gui code, like Tk or Gtk+ runs. Node.js is just a single threaded event-loop program which listens on a socket .

There is a simple explanation of this at asynchronous Perl6 and the author of Perl6's MoarVM, has an easy to watch video on youtube -> Worthington on Parallel programs in Perl6

Of course, frameworks like MCE, parallel forks, and spawning worker threads are still very useful in certain circumstances, but in general, the software of the future will be event-loop based solutions.

Finally, most of us, have single core or a few core computers. On machines like these, parallel processing is usually just an exercise in coding, and dosn't really speed anything up.

I'm not really a human, but I play one on earth. ..... an animated JAPH
  • Comment on Re: parallel programming and MCE - hubris and the frame

Replies are listed 'Best First'.
Re^2: parallel programming and MCE - hubris and the frame
by 1nickt (Canon) on Apr 10, 2017 at 12:10 UTC

    Finally, most of us, have single core or a few core computers. On machines like these, parallel processing is usually just an exercise in coding, and dosn't really speed anything up.

    Surely this depends on the task?

    Parallel::ForkManager has long given the simple example of a script to download content from many websites. Here obviously the program is not bound by disk I/O but by network response, and parallelization results in many-fold speed increases. I am using MCE for similar tasks and "dosn't really speed anything up" could not be further from my experience.

    I have switched recently from using Parallel::ForkManager to using MCE, mostly because of wanting the extra features MCE provides such as easy shared data structures with MCE::Shared, the fact that the author is very committed to supporting all major Oses, and the fact that the author is very quick to support and help.

    Yes, I am a little mindful of the dangers of committing to a "framework", but as brother Discipulus says, we all use frameworks. Some have not been a good choice over my years in this work, but the vast majority of the ones I have chosen to use, after careful research and experimentation, have been reliable and powerful and useful. Perhaps Perl's prioritization of not breaking backwards compatibility is to thank for some of this, as well as the character of Perl developers and the Perl community.

    Overall, MCE has been a great choice for me, your basic common-or-garden journeyman Perl developer (I'm not surprised you get different opinions from computer scientists like BrowserUk and zentara).


    The way forward always starts with a minimal test.
[reply]
      Hi, buried in my first reply was "Of course, frameworks like MCE, parallel forks, and spawning worker threads are still very useful in certain circumstances,", so I stand corrected and you are correct.

      But I will have to agree with BrowserUk, that complex frameworks take alot of time to master, and often require the expert guidance of the module author, as shown in Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run. Without the author's guidance, I never would have been able to work that problem out. Also, I use Linux solely, and don't really care if stuff runs properly on Windows or not. I will use what is simple and easy for linux. I do applaud MCE's author for his effort though, it appears to be a great module.

      I'm not really a human, but I play one on earth. ..... an animated JAPH
[reply]

In Section Meditations