in reply to Re: filter a file using an exclusion list
in thread filter a file using an exclusion list

Thanks for the help - but It still isnt working...I have included print statements for debugging: 

#!/usr/bin/perl -w
use strict;
open(my $excludes,"<",$ARGV[0]) or die("could not open $ARGV[0]!");
open(my $large,"<",$ARGV[1]) or die("could not open the $ARGV[1]!");

my @excludes = <$excludes>;
chomp(@excludes);
my %stopwords = map { $_ => } <@excludes>;

while( <$large> ){
        my ($test,@temp) = split(/\s+/,$_);
        print "testing: $test\n";
        print $stopwords{$test} , "\n";
        #print $test, "\n";
}


close($excludes);
close($large);

[download]
I get this message for each line of $large: 
testing: 9999853
Use of uninitialized value in print at ./filter_exclusions.pl line 13,
+ <$large> line 1.

[download]
Is it something Im missing with my input files?

Replies are listed 'Best First'.
Re^3: filter a file using an exclusion list
by Cristoforo (Curate) on Feb 13, 2011 at 22:16 UTC
    There are some errors. The one that blows up is:

    my %stopwords = map { $_ => } <@excludes>;

    If you print %stopwords in Data::Dumper you would have had:

    $VAR1 = {
          '9999853' => '999986'
        };

    [download]

    And this is not what you want! In the map, you are failing to assign a value with each key. You could state that as:

    my %stopwords = map { $_ => 1} @excludes;

    Here I am assigning a value of '1'. It conveniently then tests true when you are looking for stopwords. I also removed the angle brackets around @excludes in your code, (that would be a glob, not what you want).

    Altogether, it could be solved like this:

    #!/usr/bin/perl
use strict;
use warnings;
use 5.012;
use Data::Dumper;

my $large =<<EOF;
9999853    5615    4    148656321
999986    5615    14    94873609
9999883    5615    4    860669
9999929    5615    4    73689618
9999931    5615    4    31286083
9999944    5615    4    148596445
999995    5615    10    78405504
9999963    5615    4    84291761
9999966    5615    4    5978256
9999979    5615    4    135953341
EOF

my $excludes =<<EOF;
9999853
999986
EOF


open my $fh1, "<", \$excludes or die $!;
my %stopwords = map {chomp; $_ => 1} <$fh1>;
close $fh1 or die $!;

open my $fh2, "<", \$large or die $!;
while( <$fh2> ){
        my ($test) = /^(\d+)/;
        
        # if $test is in the hash
        # then $stopwords{ $test } == 1 or true
        print unless $stopwords{ $test };
}
close $fh2 or die $!;

#print Dumper \%stopwords;

    [download]
[reply]
[d/l]
[select]
      The above code works well on sample data - For a 70,000 item exclusion list and 40GB of data to filter the script does not work - it is not filtering the exclusion list and outputs every line of the large file (even though I have gone through and used grep on some of the items and it finds them in the large file - I would use grep -v -f excludes.txt large.txt but that does not work either on the full data). Is there a maximum limit on the size of a perl hash? Any other reason it would not work on the full data?
[reply]
      ahh - dos2unix !! I think it will work now. Many thanks!
[reply]

In Section Seekers of Perl Wisdom