Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Is there any cache mechanism for running perl script

by DillonYu (Novice)
on Apr 09, 2017 at 07:54 UTC ( [id://1187512]=perlquestion: print w/replies, xml ) Need Help??

DillonYu has asked for the wisdom of the Perl Monks concerning the following question:

hello every one !
Recently I write a perl script which not long, but it has an array named "@arr_abundant" which contain nearly 30000 elements.
And I make two "for loop" on it on a section
So every time I re run this script, it runs very slowly.

Is there any one can tell me any clues about deal with this problem? such as any cache mechanism. so I can re run it fast for debug?

Really appreciate your help!!

my @arr_abundant; for (12 .. 28123 ){ ### Evaluating [===| ] % done push @arr_abundant, $_ if prob23_type_of_number($_) eq "abundant" + ; } my %hash; # get no repeated abundant number foreach my $i (@arr_abundant){ ### Evaluating [===| ] % done foreach my $j (@arr_abundant) { my $positive = $i + $j; next if $positive > 28123; $hash{$positive} ++; } } # get the sum of positive integers cannot be written as ... my $sum_total; my @arr_sum_two_abundant_numbers=sort {$a <=> $b} keys %hash ; my $n = 1; foreach my $key (@arr_sum_two_abundant_numbers){ for ( my $i=$n ; $i < $key ; $i++) { $sum_total += $i; } $n = $key +1; } print "$sum_total\n";

Replies are listed 'Best First'.
Re: Is there any cache mechanism for running perl script
by haukex (Archbishop) on Apr 09, 2017 at 09:33 UTC
    such as any cache mechanism. so I can re run it fast for debug?

    Yes, there are several ways to do this. Which data structure would you like to cache? Since in my tests, using the assumption I showed below, I was able to regenerate @arr_abundant very quickly, I'm guessing it would make sense to cache @arr_sum_two_abundant_numbers? Since both are simple arrays of integers, you could write either of them to a file, each integer on a line. Or, you could use one of the many serialization formats like JSON, YAML, XML, etc. There's also the core module Storable, which reads and writes Perl data structures in a binary format, but it does have one possibly very important caveat, which is that such files are often not compatible between different versions of the module. If you're just using it for temporary caching on one machine, it's fine, but it's not the right solution for long-term storage or sharing between machines. Here, I've shown both the "plain file" method (using Path::Class to simplify the reading/writing, although it could easily be done in plain Perl), as well as Storable, commented out.

    use warnings; use strict; use Math::Prime::Util::GMP qw/sigma/; use Path::Class qw/file/; #OR: use Storable qw/store retrieve/; my $cache_file = '/tmp/abundant_sums_cache'; my @arr_sum_two_abundant_numbers; if (-e $cache_file) { @arr_sum_two_abundant_numbers = file($cache_file)->slurp(chomp=>1) +; #OR: @arr_sum_two_abundant_numbers = @{ retrieve($cache_file) }; } if (!@arr_sum_two_abundant_numbers) { print "regenerating \@arr_sum_two_abundant_numbers...\n"; my @arr_abundant = grep { sigma($_)>2*$_ } 12 .. 28123; my %hash; foreach my $i (@arr_abundant) { foreach my $j (@arr_abundant) { my $positive = $i + $j; next if $positive > 28123; $hash{$positive}++; } } @arr_sum_two_abundant_numbers = sort {$a <=> $b} keys %hash; file($cache_file)->spew_lines(\@arr_sum_two_abundant_numbers); #OR: store(\@arr_sum_two_abundant_numbers, $cache_file); } else { print "loaded \@arr_sum_two_abundant_numbers from $cache_file\n +" } my $sum_total; my $n = 1; foreach my $key (@arr_sum_two_abundant_numbers) { for ( my $i=$n ; $i < $key ; $i++) { $sum_total += $i; } $n = $key +1; } print "$sum_total\n";

    Because your question seems to be more about the caching, and because you don't show your data structures, explain what prob23_type_of_number is (no Google results other than this node*), or show the expected output, I haven't yet spent a lot of time on understanding your algorithm or whether it can be optimized. However, I did notice one thing:

    an array named "@arr_abundant" which contain nearly 30000 elements.

    I am guessing/assuming these are Abundant numbers (OEIS A005101), then as far as I can tell there are 6965 such numbers from 12 to 28123:

    use Math::Prime::Util::GMP qw/sigma/; print grep( { sigma($_)>2*$_ } 12 .. 28123 )."\n"; __END__ 6965

    Update: Updated wording, a few minor additions to explanation.

    * Update 2: Is this Project Euler Problem 23? Final Update: Since the output of my code matches that of other solutions I found, I guess it is :-)

      Thx a lot!
Re: Is there any cache mechanism for running perl script
by huck (Prior) on Apr 09, 2017 at 09:42 UTC

    Here is another way, using JSON to make your storage files

    use strict; use warnings; use JSON; my $usedeltatime=0; eval 'use time::DeltaTime;' if ($usedeltatime); my @bigarray; my %bighash; doit(0,0); doit(1,0); doit(1,1); sub doit { my $usecache1=shift; my $usecache2=shift; deltatime('all','set') if ($usedeltatime); if ($usecache2) {$usecache1=1;} if ($usecache1){ open (my $cache1,'<','json1.json') or die $@; my $save= do {local $/;<$cache1>}; close $cache1; @bigarray= @{decode_json($save)}; deltatime('all','stage','cache1r:'.$usecache1) if ($usedeltatime) +; } else { for my $ii (1..30000) { if (rand(100)>80) { push @bigarray,$ii; } } my $save = encode_json (\@bigarray); open (my $cache1,'>','json1.json') or die $@; print $cache1 $save; close $cache1; deltatime('all','stage','cache1w:'.$usecache1) if ($usedeltatime) +; } if ($usecache2){ open (my $cache2,'<','json2.json') or die $@; my $save= do {local $/;<$cache2>}; close $cache2; %bighash= %{decode_json($save)}; deltatime('all','stage','cache2r:'.$usecache2) if ($usedeltatime) +; } else { foreach my $i (@bigarray){ foreach my $j (@bigarray) { my $positive = $i + $j; next if $positive > 28123; $bighash{$positive} ++; } } my $save = encode_json (\%bighash); open (my $cache2,'>','json2.json') or die $@; print $cache2 $save; close $cache2; deltatime('all','stage','cache2w:'.$usecache2) if ($usedeltatime); + } print 'array:'.scalar(@bigarray)."\n";; print 'hash :'.scalar(keys %bighash)."\n";; deltatime('all','done','cache1:'.$usecache1.'cache2:'.$usecache2) +if ($usedeltatime); } # dout
    I have a little routine to create delta-times between stages, you dont, so leave $usedeltatime=0; Others may show you different ways of timing, ive been using deltatime for almost 2 decades, so i used it now.
    d-time:000:00:00 000:00:00->all:cache1w:0 d-time:000:00:30 000:00:30->all:cache2w:0 array:5994 hash :28079 d-time:000:00:00 000:00:30->all:cache1:0cache2:0@2017-04-09-04-33-41 d-time:000:00:00 000:00:00->all:cache1r:1 d-time:000:00:29 000:00:29->all:cache2w:0 array:5994 hash :28079 d-time:000:00:00 000:00:29->all:cache1:1cache2:0@2017-04-09-04-34-10 d-time:000:00:00 000:00:00->all:cache1r:1 d-time:000:00:01 000:00:01->all:cache2r:1 array:5994 hash :28079 d-time:000:00:00 000:00:01->all:cache1:1cache2:1@2017-04-09-04-34-11
    With deltatime enabled, no caching used 29 seconds, caching just bigarray used 29 seconds, but caching both bigarray and bighash only used 1 second

    YMMV

      > I have a little routine to create delta-times between stages, you dont

      I've SuperSearched, cpanserched and googled and i found nothing; it is not something worth to share? ;)

      L*

      There are no rules, there are no thumbs..
      Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
      thx very much!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1187512]
Approved by marto
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others about the Monastery: (6)
As of 2024-04-19 11:10 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found