kcott has asked for the wisdom of the Perl Monks concerning the following question:

G'day All,

I haven't use MCE previously and decided to get my head around it. I believe some of my current personal projects could benefit from it; there are also some $work applications.

I ran the very first example code in "MCE::Core - SYNOPSIS" and that worked fine.

[Note: In the following, "lcores" refers to "logical CPU cores"; MCE doco uses the same abbreviation.]

I decided that it would be useful to allow users to specify what percentage of lcores were used by an application. For instance, at any given time, 50% might be allowed for one MCE process, 25% for another MCE process, leaving the remaining 25% for other general use on the computer.

My first thought was that "max_workers => 'auto'" was unsuitable for this; even with its arithmetic modifiers (see "MCE::Core - MCE->new()" and "MCE::Util - get_ncpu()" for more about that). It uses an arbitrary eight or all lcores, whichever is the smallest: not useful for determining a percentage of lcores. So, I came up with the following proof-of-concept (POC) code: 

#!/usr/bin/env perl

use 5.016;
use warnings;

use MCE;

my @percentages = qw{0 20 25 33.333 40 50 60 66.667 75 80 100 200};

for my $specified_percentage (@percentages) {
    say "Specified percentage of logical cores to use: $specified_perc
+entage%";
    my $modified_percentage
        = $specified_percentage <= 0   ? 1
        : $specified_percentage >= 100 ? 100
        :                                $specified_percentage;
    say "Modified percentage of logical cores to use: $modified_percen
+tage%";
    say 'Total number of logical cores ', MCE::Util::get_ncpu();
    my $max_workers
        = int(0.5 + MCE::Util::get_ncpu() * $modified_percentage / 100
+) || 1;
    say "Maximum workers to use: $max_workers";

    my @wids;

    my $mce = MCE::->new(
        max_workers => $max_workers,
        gather => \@wids,
        user_func => sub {
            my ($mce) = @_;
            MCE->gather($mce->wid);
        },
    );

    $mce->run();

    say "Worker ID(s): @{[sort { $a <=> $b } @wids]}";
}

[download]

[Note: The use 5.016 is $work related: the minimum version to use. 5.012 will cover strict and say; MCE only requires 5.008001.]

The output from this looked fine. I've put it inside the spoiler. 

Specified percentage of logical cores to use: 0%
Modified percentage of logical cores to use: 1%
Total number of logical cores 12
Maximum workers to use: 1
Worker ID(s): 1
Specified percentage of logical cores to use: 20%
Modified percentage of logical cores to use: 20%
Total number of logical cores 12
Maximum workers to use: 2
Worker ID(s): 1 2
Specified percentage of logical cores to use: 25%
Modified percentage of logical cores to use: 25%
Total number of logical cores 12
Maximum workers to use: 3
Worker ID(s): 1 2 3
Specified percentage of logical cores to use: 33.333%
Modified percentage of logical cores to use: 33.333%
Total number of logical cores 12
Maximum workers to use: 4
Worker ID(s): 1 2 3 4
Specified percentage of logical cores to use: 40%
Modified percentage of logical cores to use: 40%
Total number of logical cores 12
Maximum workers to use: 5
Worker ID(s): 1 2 3 4 5
Specified percentage of logical cores to use: 50%
Modified percentage of logical cores to use: 50%
Total number of logical cores 12
Maximum workers to use: 6
Worker ID(s): 1 2 3 4 5 6
Specified percentage of logical cores to use: 60%
Modified percentage of logical cores to use: 60%
Total number of logical cores 12
Maximum workers to use: 7
Worker ID(s): 1 2 3 4 5 6 7
Specified percentage of logical cores to use: 66.667%
Modified percentage of logical cores to use: 66.667%
Total number of logical cores 12
Maximum workers to use: 8
Worker ID(s): 1 2 3 4 5 6 7 8
Specified percentage of logical cores to use: 75%
Modified percentage of logical cores to use: 75%
Total number of logical cores 12
Maximum workers to use: 9
Worker ID(s): 1 2 3 4 5 6 7 8 9
Specified percentage of logical cores to use: 80%
Modified percentage of logical cores to use: 80%
Total number of logical cores 12
Maximum workers to use: 10
Worker ID(s): 1 2 3 4 5 6 7 8 9 10
Specified percentage of logical cores to use: 100%
Modified percentage of logical cores to use: 100%
Total number of logical cores 12
Maximum workers to use: 12
Worker ID(s): 1 2 3 4 5 6 7 8 9 10 11 12
Specified percentage of logical cores to use: 200%
Modified percentage of logical cores to use: 100%
Total number of logical cores 12
Maximum workers to use: 12
Worker ID(s): 1 2 3 4 5 6 7 8 9 10 11 12

[download]

The percentage needed to be capped at 100. If I comment out ": $specified_percentage >= 100 ? 100", the last block of output, which was 

Specified percentage of logical cores to use: 200%
Modified percentage of logical cores to use: 100%
Total number of logical cores 12
Maximum workers to use: 12
Worker ID(s): 1 2 3 4 5 6 7 8 9 10 11 12

[download]

becomes

Specified percentage of logical cores to use: 200%
Modified percentage of logical cores to use: 200%
Total number of logical cores 12
Maximum workers to use: 24
Worker ID(s): 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
+ 23 24

[download]

I could put a version of that code into a module with a function returning $max_workers.

I decided to revisit 'auto' and see if I could come up with something better. Here's that POC solution: 

#!/usr/bin/env perl

use 5.016;
use warnings;

use MCE;

my $total_logical_cores = MCE::Util::get_ncpu();
my $auto_max_workers;

{
    my $mce_temp = MCE::->new(max_workers => 'auto');
    $auto_max_workers = $mce_temp->max_workers();
    $mce_temp->shutdown;
}

my $auto_to_ncpu_mult = $total_logical_cores / $auto_max_workers;
my @percentages = qw{0 20 25 33.333 40 50 60 66.667 75 80 100 200};

for my $percentage (@percentages) {
    say "Specified percentage of logical cores to use: $percentage%";
    $percentage = 1 if $percentage <= 0;
    say "Modified percentage of logical cores to use: $percentage%";
    say 'Total number of logical cores ', MCE::Util::get_ncpu();
    my $auto_max_workers = 'auto*' . $auto_to_ncpu_mult * $percentage 
+/ 100;
    say "Auto max_workers: $auto_max_workers";

    my @wids;

    my $mce = MCE::->new(
        max_workers => $auto_max_workers,
        gather => \@wids,
        user_func => sub {
            my ($mce) = @_;
            MCE->gather($mce->wid);
        },
    );

    $mce->run();

    say "Worker ID(s): @{[sort { $a <=> $b } @wids]}";
}

[download]

The output looks fine; again, it's in the spoiler. 

Specified percentage of logical cores to use: 0%
Modified percentage of logical cores to use: 1%
Total number of logical cores 12
Auto max_workers: auto*0.015
Worker ID(s): 1
Specified percentage of logical cores to use: 20%
Modified percentage of logical cores to use: 20%
Total number of logical cores 12
Auto max_workers: auto*0.3
Worker ID(s): 1 2
Specified percentage of logical cores to use: 25%
Modified percentage of logical cores to use: 25%
Total number of logical cores 12
Auto max_workers: auto*0.375
Worker ID(s): 1 2 3
Specified percentage of logical cores to use: 33.333%
Modified percentage of logical cores to use: 33.333%
Total number of logical cores 12
Auto max_workers: auto*0.499995
Worker ID(s): 1 2 3 4
Specified percentage of logical cores to use: 40%
Modified percentage of logical cores to use: 40%
Total number of logical cores 12
Auto max_workers: auto*0.6
Worker ID(s): 1 2 3 4 5
Specified percentage of logical cores to use: 50%
Modified percentage of logical cores to use: 50%
Total number of logical cores 12
Auto max_workers: auto*0.75
Worker ID(s): 1 2 3 4 5 6
Specified percentage of logical cores to use: 60%
Modified percentage of logical cores to use: 60%
Total number of logical cores 12
Auto max_workers: auto*0.9
Worker ID(s): 1 2 3 4 5 6 7
Specified percentage of logical cores to use: 66.667%
Modified percentage of logical cores to use: 66.667%
Total number of logical cores 12
Auto max_workers: auto*1.000005
Worker ID(s): 1 2 3 4 5 6 7 8
Specified percentage of logical cores to use: 75%
Modified percentage of logical cores to use: 75%
Total number of logical cores 12
Auto max_workers: auto*1.125
Worker ID(s): 1 2 3 4 5 6 7 8 9
Specified percentage of logical cores to use: 80%
Modified percentage of logical cores to use: 80%
Total number of logical cores 12
Auto max_workers: auto*1.2
Worker ID(s): 1 2 3 4 5 6 7 8 9 10
Specified percentage of logical cores to use: 100%
Modified percentage of logical cores to use: 100%
Total number of logical cores 12
Auto max_workers: auto*1.5
Worker ID(s): 1 2 3 4 5 6 7 8 9 10 11 12
Specified percentage of logical cores to use: 200%
Modified percentage of logical cores to use: 200%
Total number of logical cores 12
Auto max_workers: auto*3
Worker ID(s): 1 2 3 4 5 6 7 8 9 10 11 12

[download]

Because 'auto' caps the maximum number of workers at the maximum number of lcores, I didn't need to do anything special with that. Here's the last block of output for comparison with what I showed above. 

Specified percentage of logical cores to use: 200%
Modified percentage of logical cores to use: 200%
Total number of logical cores 12
Auto max_workers: auto*3
Worker ID(s): 1 2 3 4 5 6 7 8 9 10 11 12

[download]

However, I did need to set the minimum percentage. If I comment out "$percentage = 1 if $percentage <= 0;, the first block of output, which was 

Specified percentage of logical cores to use: 0%
Modified percentage of logical cores to use: 1%
Total number of logical cores 12
Auto max_workers: auto*0.015
Worker ID(s): 1

[download]

becomes

Specified percentage of logical cores to use: 0%
Modified percentage of logical cores to use: 0%
Total number of logical cores 12
Auto max_workers: auto*0
Worker ID(s): 1 2 3 4 5 6 7 8

[download]

Curious: 'auto*0' eq 'auto*1' — is that a bug?

I could also put a version of that code into a module with a function returning $auto_max_workers.

So, finally the questions :-)

— Ken

Replies are listed 'Best First'.
Re: MCE - max_workers
by marioroy (Prior) on Nov 15, 2021 at 15:29 UTC

    Hi Ken,

    That's a great idea. I can add a regex check for %; i.e. max_workers => '25%' and set the value accordingly. It will clamp to 1 minimum and MCE::Util::get_ncpu() maximum.

    With regards to max_workers => 'auto', that limits to 8 max including on machines with more than 8 logical cores. It was a safety mechanism at the time to not have MCE consume more cores for heavy IO related tasks. Basically, I did not want for MCE to be the reason why a server went down.

    auto*0 is not a bug. The minimum number of workers is 1.

[reply]

      G'day Mario,

      "That's a great idea."

      Thanks for the positive feedback.

      "I can add a regex check for % ..."

      That would be great. Having it in the back-end (e.g. an MCE class): 

      use MCE;
...
max_workers => "$percent%"

      [download]

      rather than in an intermediary module (e.g. a Kens::MCE::Util class): 

      use MCE;
use Kens::MCE::Util 'get_max_workers';
...
max_workers => get_max_workers($percent)

      [download]

      would certainly simplify matters.

      "auto*0 is not a bug."

      Thanks for the clarification.

      — Ken

[reply]
[d/l]
[select]

        Hi Ken,

        MCE 1.875 was released moments ago. Folks may specify a percentage for max_workers. I enjoyed writing the MCE::Flow test in 03_max_workers.t.

        use strict;
use warnings;
use Test::More;

BEGIN {
   use_ok 'MCE';
   use_ok 'MCE::Flow';
}

{
   no warnings 'redefine';
   sub MCE::Util::get_ncpu { return 16; }
}

# ... lots more tests not shown here

{
   MCE::Flow::init(max_workers => [1, '25%']);

   my @res;
   mce_flow { gather => \@res },
       sub { MCE->gather('a'.MCE->task_wid()); },  # 1 worker
       sub { MCE->gather('b'.MCE->task_wid()); };  # 4 workers

   @res = sort @res;
   is("@res", "a1 b1 b2 b3 b4", "check that MCE::Flow ran with 5 worke
+rs");

   MCE::Flow->finish();
}

done_testing;

        [download]

        Thanks,

[reply]
[d/l]
Re: MCE - max_workers [Benchmark]
by kcott (Archbishop) on Jul 01, 2021 at 20:55 UTC

    I stripped both pieces of code down to the bare minimum of calculating just $max_workers (1st solution) and $auto_max_workers (2nd solution). I ran some benchmarks on these last night.

    The "$max_workers (1st solution)" ran about 15 times faster than "$auto_max_workers (2nd solution)". This is not overly surprising: calculating $max_workers is mostly just arithmetic; calculating $auto_max_workers involves instantiating a temporary object and then some arithmetic.

    In the absence of other advice, "$max_workers (1st solution)" will end up in Kens::MCE::Util (or whatever I decide to call it). I've implemented nothing as yet; however, expected usage would probably be something like: 

    ...
use Kens::MCE::Util 'get_max_workers';
...
my $percent; # value from user input, config file, etc.
...
my $mce = MCE::->new(
    max_workers => get_max_workers($percent),
    ...
);
...

    [download]

    — Ken

[reply]
[d/l]
[select]
Re: MCE - max_workers
by karlgoethebier (Abbot) on Jul 01, 2021 at 05:48 UTC
    «…specify what percentage…»

    Hi kcott, for the moment I don’t understand why you bother yourself with this percentage stuff. May be that this has something to do with my bad Denglish. AKA that I didn’t understand your problem right. But anyway: Why don’t you hand over the number of CPU‘s as an int param and good is? Best regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

[reply]

      G'day Karl,

      Let's say I have three clients (users of my MCE application): A has 4 lcores; B has 8 lcores; C has 16 lcores.

      Let's further say that each client only wants to use half of their processing power for my application, so: A wants to use 2 lcores; B wants to use 4 lcores; C wants to use 8 lcores.

      If I simply use an "int param", as you suggest, they'll get the same value for max_workers. For instance, if I used max_workers => 6: A would be using 150%; B would be using 75%; C would be using 37.5%. (Compare what happened, in the first half of my OP, where I didn't cap the value and, at 200%, my computer with 12 lcores got 24 workers.)

      Furthermore, clients' requirements may change depending on a number of factors. Perhaps start a big job on a Friday afternoon, using all available processing power, and let it run over the weekend; in another scenario, run jobs mid-week, during a busy part of the day, using only a fraction of available processing power.

      Both of my solutions calculate the percentages correctly. What I didn't know was if there was a better solution; and, if not, which of those two solutions fellow monks would consider to be better (keeping in mind, of course, that better might vary under different conditions).

      — Ken

[reply]
[d/l]
[select]

        May be that I still miss the point. But it is like you go to your favorite restaurant: It has n cooks. As a guest AKA client you can’t influence this number to get your diner faster. You have to wait. You get in a queue. Same thing as if you take this Magic Bus

        «The Crux of the Biscuit is the Apostrophe»

[reply]

Back to Seekers of Perl Wisdom