The above code works well on sample data - For a 70,000 item exclusion list and 40GB of data to filter the script does not work - it is not filtering the exclusion list and outputs every line of the large file (even though I have gone through and used grep on some of the items and it finds them in the large file - I would use grep -v -f excludes.txt large.txt but that does not work either on the full data). Is there a maximum limit on the size of a perl hash? Any other reason it would not work on the full data?