csalzer has asked for the wisdom of the Perl Monks concerning the following question:

I read some of the other postings here but didn't find anything that addresses my need. I am *very* remedial in perl but trying to learn quickly as my job description now requires it.

-I have a short script that uses List::Permutor to generate permutations of a series. Works great and prints to an outfile. -I also have a short script that open the .txt file containing all permutations, selects unique sequences from the list, and prints the unique sequences to a new outfile.

With small series this works great, but with long series I can't complete both steps because of my computer runs out of RAM (8GB). In fact, with 12! permutations, the computer runs sluggish just to open the outfile. I thought that if I could combine the two steps to run with only 1 outfile to worry about it might save me some memory.

I have posted my code below. Thanks for your help and time!


FIRST
---------------------
#!/usr/bin/perl<br> use strict; use warnings; use List::Permutor; my $OutFile = '>permutations.txt'; open(OUTFILE, $OutFile) or die "The file $OutFile could not be found.\ +n"; my $word = 'aabc'; my $perm = new List::Permutor split(//,$word); while (my @set = $perm->next) { #print @set for(my $i=0; $i<length($word)-1; $i=$i+1) { my $letter = $set[$i]; print OUTFILE "$letter,"; }; my $letter = $set[length($word)-1]; print OUTFILE "$letter\n"; } close (OUTFILE);

-----------------------------------------

SECOND
-----------------------

#!/usr/bin/perl use warnings; use strict; chdir('//Users/laptop/Desktop/') or die ("Sorry could not change to th +e perl/programs directory.\n"); my $AllPermutations = 'permutations.txt'; #INFILE open(ALLPERMUTATIONS, $AllPermutations) or die "File: $AllPermutations + failed to open: $!\n"; my $UniqPermutations = '>unique_permutations.txt'; #OUTFILE open(UNIQPERMUTATIONS, $UniqPermutations) or die "The file $UniqPermut +ations could not be found.\n"; my @AllPermutations = <ALLPERMUTATIONS>; my %seen = (); my @unique = grep { ! $seen{ $_ }++ } @AllPermutations; print "@unique"; my $str = "@unique"; print join(",",split(//,$str)); close(ALLPERMUTATIONS); close(UNIQPERMUTATIONS);
------------------------------------------------------

Replies are listed 'Best First'.
Re: Combine Two Scripts Into One
by toolic (Bishop) on Feb 23, 2011 at 20:48 UTC
    Instead of slurping the entire permutations.txt file (which is probably big) into your 2nd script, just loop over the lines of the file:
    my %seen; while (<ALLPERMUTATIONS>) { chomp; $seen{$_} = 1; } print "$_\n" for keys %seen;
Re: Combine Two Scripts Into One
by wind (Priest) on Feb 23, 2011 at 21:14 UTC
    To clean up the code of your first script a little bit, we get this:
    #!/usr/bin/perl use strict; use warnings; use List::Permutor; my $outfile = 'permutations.txt'; open my $fh, '>', $outfile or die "$outfile: $!"; my $word = 'aabc'; my $perm = new List::Permutor split(//,$word); while (my @set = $perm->next) { my $str = join '', @set; print $fh "$str\n"; } close $fh;
    And if we want to include a modified version of toolic's uniqueness check, we end up with this:
    my %seen; while (my @set = $perm->next) { my $str = join '', @set; print $fh "$str\n" if ! $seen{$str}++; }
Re: Combine Two Scripts Into One
by samarzone (Pilgrim) on Feb 24, 2011 at 10:47 UTC

    You are writing a file in first script and reading the same file in second script, so you must not be running these scripts in parallel. Why do you think that merging these two scripts into one, will solve your memory issues? IMHO the max memory taken by any script (or code block) will still be the same. In fact, in a single script, there might be chances that while executing the later half of the code, first half is also eating up some memory.

    Suggestions are already provided.

    --
    Regards
    - Samar