in reply to Re^10: PDL and srand puzzle
in thread PDL and srand puzzle

This indicates that random() generates fewer random bits than rand(). Right ??

I don't think so. Looking at the produced values was a surprise.

Took the threads loop and ran it with 16 pids and 1024 random() calls. Used "%.64f" "%.72f" as format for the generated numbers. In most cases, different runs produce identical results.

<Update>
For completeness, here is the script.

#!/usr/bin/perl use v5.030; use threads; use threads::shared; use PDL; BEGIN { $PDL::no_clone_skip_warning = 1; } my $lock : shared = 0; PDL::srand(3); # PDL 2.087 for my $tid (1..16) { threads->create(sub { my $output = ""; for (1..1024) { my $r = random(); $output .= sprintf "%.72f\n", $r; } lock $lock; print $output; }); } $_->join for threads->list;
</Update>

When a run produces a different result, I see two scenarios:

$ ./pdl-rand.pl | sort >sort.16384.1 $ ./pdl-rand.pl | sort >sort.16384.2 $ ./pdl-rand.pl | sort >sort.16384.3 $ ./pdl-rand.pl | sort >sort.16384.4 $ sort -u sort.16384.1 |wc -l 16384 $ sort -u sort.16384.2 |wc -l 16383 $ sort -u sort.16384.3 |wc -l 16382 $ sort -u sort.16384.4 |wc -l 16383 $ diff -U 1 sort.16384.1 sort.16384.2 --- sort.16384.1 2024-06-08 09:45:09.466739327 +0200 +++ sort.16384.2 2024-06-08 09:50:24.203356598 +0200 @@ -12562,2 +12562,3 @@ 0.7698795891537290048134423159353900700807571411132812500000000000000 +00000 +0.7698795891537290048134423159353900700807571411132812500000000000000 +00000 0.7699350058887169945265327442029956728219985961914062500000000000000 +00000 @@ -14417,3 +14418,2 @@ 0.8818249357757870221519169717794284224510192871093750000000000000000 +00000 -0.8818467386417550013533173114410601556301116943359375000000000000000 +00000 0.8818802323789649566521120505058206617832183837890625000000000000000 +00000 $ diff -U 1 sort.16384.1 sort.16384.3 --- sort.16384.1 2024-06-08 09:45:09.466739327 +0200 +++ sort.16384.3 2024-06-08 09:52:32.837243809 +0200 @@ -13113,2 +13113,3 @@ 0.8005079965106629558135864499490708112716674804687500000000000000000 +00000 +0.8005079965106629558135864499490708112716674804687500000000000000000 +00000 0.8007896602698270083209308722871355712413787841796875000000000000000 +00000 @@ -13686,3 +13687,2 @@ 0.8361851909041919661547126452205702662467956542968750000000000000000 +00000 -0.8361968170640460273901339860458392649888992309570312500000000000000 +00000 0.8364519442006479454931877626222558319568634033203125000000000000000 +00000 @@ -14417,3 +14417,2 @@ 0.8818249357757870221519169717794284224510192871093750000000000000000 +00000 -0.8818467386417550013533173114410601556301116943359375000000000000000 +00000 0.8818802323789649566521120505058206617832183837890625000000000000000 +00000 @@ -16095,2 +16094,3 @@ 0.9834468387403769717991508514387533068656921386718750000000000000000 +00000 +0.9834468387403769717991508514387533068656921386718750000000000000000 +00000 0.9837232694387190168328061190550215542316436767578125000000000000000 +00000 $ diff -U 1 sort.16384.1 sort.16384.4|head -12 --- sort.16384.1 2024-06-08 09:45:09.466739327 +0200 +++ sort.16384.4 2024-06-08 09:58:27.630459409 +0200 @@ -1,154 +1,161 @@ -0.0000798345195128779969639606917120033813262125477194786071777343750 +00000 +0.0000273717076871849010000561919220274376129964366555213928222656250 +00000 +0.0000550196742872266023331902229376311197484028525650501251220703125 +00000 0.0001113510803039119965475445273028753945254720747470855712890625000 +00000 -0.0001282989599132870110840404231922207145544234663248062133789062500 +00000 0.0002144808569095600127036443938166598854877520352602005004882812500 +00000 -0.0003265905868672370191731213484587215134524740278720855712890625000 +00000 0.0003745585362475710206227319520877472314168699085712432861328125000 +00000 -0.0004606498956905009760365299342765865731053054332733154296875000000 +00000 $ diff -U 0 sort.16384.1 sort.16384.4|grep -v '@@'|wc -l 19698

I have no idea what's going on here but maybe someone else has.

Update: Fixed manually modified file names in diff output.

Greetings,
🐻

$gryYup$d0ylprbpriprrYpkJl2xyl~rzg??P~5lp2hyl0p$

Replies are listed 'Best First'.
Re^12: PDL and srand puzzle - predictability summary
by marioroy (Prior) on Jun 08, 2024 at 09:53 UTC

    Predictability Summary:

    non-threads

    # Processes - CORE CORE::srand(N); CORE::rand(); - PDL CORE::srand(N); # This also, for MCE predictable results PDL::srand(N); # PDL v1.062 ~ v1.089 PDL::srandom(N); # PDL v1.089_01 PDL->random; # MCE v1.895 - Math::Prime::Util CORE::srand(N); # This also, for MCE predictable results Math::Prime::Util::srand(N); Math::Prime::Util::drand(); Math::Prime::Util::irand64(); - Math::Random CORE::srand(N); # This also, for MCE predictable results Math::Random::random_set_seed(N, N); Math::Random::random_normal(); Math::Random::random_uniform(); - Math::Random::MT::Auto CORE::srand(N); # This also, for MCE predictable results Math::Random::MT::Auto::set_seed(N); Math::Random::MT::Auto::rand();

    threads

    # Threads (matching non-threads output), requires MCE v1.894 - CORE CORE::srand(N); CORE::rand(); - Math::Prime::Util CORE::srand(N); # This also, for MCE predictable results Math::Prime::Util::srand(N); Math::Prime::Util::drand(); Math::Prime::Util::irand64(); - Math::Random::MT::Auto CORE::srand(N); # This also, for MCE predictable results Math::Random::MT::Auto::set_seed(N); Math::Random::MT::Auto::rand(); - Not possible PDL or Math::Random

    Running threads, I'm unable to generate predictable results using PDL or Math::Random (random_normal/random_uniform), regardless seeding the generator per each thread.

    See the complete demonstration here. Run Perl with -Mthreads to spin threads.

      Running threads, I'm unable to generate predictable results using PDL

      Depends on the number of samples. I have the impression there is a somewhat rare condition triggering the bug. The more calls, the more probable its appearance.

      Investigated several loop sizes and the number of identical results for several runs:

      loops predictable result 512 100% 1024 60% 2048 20% 4096 0%

      Greetings,
      🐻

      $gryYup$d0ylprbpriprrYpkJl2xyl~rzg??P~5lp2hyl0p$

        This is a non-thread version. This exposes a PDL bug. That is PDL::srandom has no effect and must call CORE::srand(N) instead for predictable output. The relay block guarantees that worker1 outputs first, followed by worker2, and so on.

        Edit 1: MCE checks for PDL::Primitive->can('srand'), but missed checking PDL::Primitive->can('srandom'). Resolved in MCE v1.894 and MCE::Shared v1.889.

        Edit 2: MCE configures an internal seed. It turns out that MCE may not know the srand or setter used by the application. Releasing MCE 1.895 and MCE::Shared 1.890. I updated the demonstration to process a sequence of numbers (lesser memory consumption). See also, Predictability Summary.

        Loops 3276800

        #!/usr/bin/perl use v5.030; use PDL; use MCE 1.895; CORE::srand(3); # This also, for MCE predictable results # MCE sets internal seed = CORE::random() PDL::srandom(3); # PDL::srand(3) v1.062 ~ v1.089 MCE->new( use_threads => 0, # Ensure non-threads on Windows max_workers => 16, init_relay => 0, user_func => sub { my $output = ""; for (1..3276800) { my $r = random(); $output .= sprintf "%.72f\n", $r; } MCE::relay { print $output; }; } )->run;
        $ perl pdl-rand-mce.pl | wc -l 52428800 $ perl pdl-rand-mce.pl | cksum 3755051732 3932160000 $ perl pdl-rand-mce.pl | cksum 3755051732 3932160000 $ perl pdl-rand-mce.pl | cksum 3755051732 3932160000 $ perl pdl-rand-mce.pl | LC_ALL=C sort -u | wc -l $ perl pdl-rand-mce.pl | LC_ALL=C mcesort -j16 -u | wc -l $ perl pdl-rand-mce.pl | LC_ALL=C parsort --parallel=16 -u | wc -l 52428799

        Loops 3276800 Iterate Piddle

        #!/usr/bin/perl use v5.030; use PDL; use MCE 1.895; CORE::srand(3); # This also, for MCE predictable results # MCE sets internal seed = CORE::random() PDL::srandom(3); # PDL::srand(3) v1.062 ~ v1.089 MCE->new( use_threads => 0, # Ensure non-threads on Windows max_workers => 16, init_relay => 0, user_func => sub { my $output = ""; my $pdl = PDL->random(3276800); foreach (0 .. $pdl->nelem - 1) { my $r = $pdl->at($_); $output .= sprintf "%.72f\n", $r; } MCE::relay { print $output; }; } )->run;
        $ perl pdl-rand-mce2.pl | wc -l 52428800 $ perl pdl-rand-mce2.pl | cksum 1425016579 3932160000 $ perl pdl-rand-mce2.pl | cksum 1425016579 3932160000 $ perl pdl-rand-mce2.pl | cksum 1425016579 3932160000 $ perl pdl-rand-mce2.pl | LC_ALL=C sort -u | wc -l $ perl pdl-rand-mce2.pl | LC_ALL=C mcesort -j16 -u | wc -l $ perl pdl-rand-mce2.pl | LC_ALL=C parsort --parallel=16 -u | wc -l 52428799

        The parallel mcesort program is found at GitHub Gist. Another option is GNU parallel parsort.