Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re^2: Script exponentially slower as number of files to process increases

by xnous (Sexton)
on Jan 28, 2023 at 01:08 UTC ( #11149963=note: print w/replies, xml ) Need Help??


in reply to Re: Script exponentially slower as number of files to process increases
in thread Script exponentially slower as number of files to process increases

You nailed it! So, if I understand correctly, the crucial difference is that my original script forked a process for every single file in queue, while yours only for $maxforks, right?

The documentation on fork() says it "does a fork(2) system call to create a new process running the same program at the same point" and "great care has gone into making it extremely efficient (for example, using copy-on-write technology on data pages)", so I assumed an array with a few hundred thousand elements wouldn't pose a problem. Obviously, I was wrong.

I benchmarked the 3 proposed updates from choroba, marioroy (who had also been of great help in my previous question) and kikuchiyo. The tests ran in a loop utilizing 8, 16, 32, etc up to 4096 threads over 3457, 34570 and 345700 files and the results are very interesting and demonstrate the strong and weak points of each implementation. The test box was a mobile 4th-gen, 4c/8t i7 with 32GB of RAM. All tests ran on tmpfs.

I created some graphs for easier interpretation of the results but it appears I can't link to imgur; is there an "allowed" image posting site I can link to?

All said, I'd like to thank every one of you, your insight and wisdom have fully answered my questions.

  • Comment on Re^2: Script exponentially slower as number of files to process increases

Replies are listed 'Best First'.
Re^3: Script exponentially slower as number of files to process increases
by marioroy (Parson) on Jan 28, 2023 at 05:24 UTC
    the results are very interesting...

    The kikuchiyou.pl script not involving IPC should be first. It's a great solution. If memory is plentiful, why not consume a little memory for input data before spawning.

    The tests ran in a loop utilizing 8, 16, 32, etc up to 4096 threads...

    It's interesting seeing one attempting 4,000+ workers for a use-case that is somewhat CPU-bound. On another note, what coolness witnessing the operating system coping with this. For example, the threads and MCE::Child solutions involve IPC; e.g. workers entering a critical section -- who's next to read input from the queue or channel.

    On my system, the kikuchiyo.pl script exits early running 512 workers. I modified the script to figure why that is.

    --- kikuchiyo1.pl 2023-01-27 23:31:34.592261566 -0600 +++ kikuchiyo2.pl 2023-01-27 23:31:12.488762580 -0600 @@ -92,12 +92,16 @@ for my $worker_id (0..$maxforks-1) { if (my $pid = fork) { ++$forkcount; + } elsif (!defined $pid) { + warn "fork failed for worker_id $worker_id\n"; } else { for my $i (0..$#{$batched_data[$worker_id]}) { my $infile = $batched_data[$worker_id][$i]; my $subdir = $worker_id + 1; - open my $IN, '<', $infile or exit(0); - open my $OUT, '>', "$tempdir/$subdir/text-$i" or exit(0); + open my $IN, '<', $infile + or die "[$worker_id] open error: infile"; + open my $OUT, '>', "$tempdir/$subdir/text-$i" + or die "[$worker_id] open error: outfile\n"; while (<$IN>) { tr/-!"#%&()*',.\/:;?@\[\\\]_{}><^)(|/ /; # no punc +t " s/^/ /;

    Possibly a ulimit issue -n issue. My open-files ulimit is 1024. Edit: It has nothing to do with ulimit as the number of open files ulimit is per process. Workers 256+ exit early.

    [256] open error: outfile [257] open error: outfile [258] open error: outfile ... [511] open error: outfile

    The threads and MCE::Child solutions pass for 512 workers. Again, regex is used here -- kinda CPU bound. Is running more workers than the number of logical CPU cores improving performance? There are golden CPU samples out there.

      First of all, thank you for your explanations and the work you put in suggesting an alternative.

      Is running more workers than the number of logical CPU cores improving performance? There are golden CPU samples out there.

      I don't know what to say, other than try to benignly bypass the perlmonks' filter and show somehow the graphs (if a janitor would be kind enough to URLify these, thanks). It does seem that in certain scenarios the first part of your statement holds true.

    • First test, https://i.imgur.com/CpclI9L.png - 3457 files
    • Second test, https://i.imgur.com/cDi41fC.png 34570 files
    • Third test, https://i.imgur.com/yNZokCx.png - 345700 files, due to long run times, I omitted the clearly slower Threads::Queue solution.
    • Final test, https://i.imgur.com/2NVovHx.png - same load as in #3, but only for fork() which proved to be the fastest, across the 512-4096 range in 128-step increments, trying to find the sweet spot.
    • You need to copy/paste the links by hand, but it's worth the trouble. The gain of fork() from 256 to 512 processes is almost unbelievable, while the performance of the other implementations is practically linear.

      EDIT: But of course it is, it's due to workers exiting early.

      I also tested your updated script but it showed no tangible improvement on my setup.

        Certainly, there's an anomaly. Do you know what is causing the beautiful script to suddenly leap from 110 seconds down to 50 seconds? Unfortunately, half of the workers exit due to open file error. They go unnoticed due to no warning messages.

        Applying the changes here may enlighten you as to why. Another verification is running ls -l data.dat. Is the file size smaller than expected?

Re^3: Script exponentially slower as number of files to process increases
by marioroy (Parson) on Jan 28, 2023 at 08:45 UTC

    This topic presented an opportunity for me to brush up on MCE, particularly the input iterator. Notice the parallel routine to process files, the input iterator run by the manager process, and MCE::Loop to launch parallel into action. The $chunk_id is used to compute $i, matching the data orderly. Moreover, MCE->last to leave the loop entirely (including further input data) due to open file error.

    Update: I had met for the input iterator to obtain data after workers have spawned. Now, workers consume little memory. Hence, the reason making this demonstration.

    Sample run from a 32-core, 64-threads box:

    I captured the "real" via the UNIX time command. We're dealing with big data. The UNIX time command is helpful to capture the overall time. This includes loading modules, running, and reaping workers. Not to forget, global cleanup in Perl.

    The total time includes post-processing, reading temp-files, writing to data.dat, and unlinking.

      A chunking engine allows one to specify a smaller chunk size for more possibilities. What we can do is have workers append to a local string variable. Upon completing the chunk, workers write the buffer to the target file. This works out quite well, further reducing the overall time. This rid of the time taken before and after W.R.T. temp dirs/files creation, reading, and cleanup.

      Why smaller chunk size? Workers append to a local string variable for this demonstration. A greater chunk size also means greater memory consumption, potentially.

      Making this possible is MCE::Relay (for orderly) or MCE::Mutex (unorderly). Not seeing all CPU cores at 100% utilization? That's because workers now cooperate orderly or serially when appending to the target file. Please no need to spin 4,000+ workers or beyond the physical limit of the box. That's really unnecessary.

      Look at the time saved. Merging results is now handled by workers versus post-processing, previously taking 31.510 seconds.

      # MCE::Relay $ time ../mario-mceiter2.pl Parsing 345701 files maxforks: 64 chunksize: 50 regex: 13.628399 real 0m13.670s user 11m11.484s sys 0m 4.739s # MCE::Mutex $ time ../mario-mceiter2.pl Parsing 345701 files maxforks: 64 chunksize: 50 regex: 12.646211 real 0m12.689s user 11m28.535s sys 0m 4.827s

      mario-mceiter2.pl

        I got my hands on a bare metal EPYC 16/32 server and fired up the scripts on it. Its sweet spot for kikuchiyo's script was between 768 and 864 forks, running for 3.2 seconds on 60K files. Above that it steadily worsened. Your mceiter2.pl script took around 7.3 seconds for every fork count between 32-128, dropping slowly afterwards. So, the CPU architecture indeed plays a major role in how (and how many) threads are handled.

        Oh, by the way, the initial MCE script somehow OOM'ed my 32GB laptop, killing Xorg in the process, but I was cocky enough to have kept a 300-tab firefox session open during testing...

        EDIT: After the $subdir overflow bug discovery in kikuchiyo's script, both his and marioroy's scripts perform roughly the same. Apologies are in order for not double-checking the results before posting triumphant benchmarks with 2048 threads on a 4-core machine...

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11149963]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (1)
As of 2023-03-31 02:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Which type of climate do you prefer to live in?






    Results (74 votes). Check out past polls.

    Notices?