in reply to Re: Super fast file creation needed
in thread Super fast file creation needed

Whoa gnarly dude! That is totally way faster! Here's the details on what I am doing in case anyone spots something totally stupid on my part ...
#!/usr/local/bin/perl -w use strict; my %seen; my @logfiles = glob ( "*access_log" ); foreach my $logfile (@logfiles) { open ( MYLOG, ">>progress_safe" ); print MYLOG "Doing $logfile\n"; close ( MYLOG ); process_file( $logfile ); } sub process_file { my $fn = shift; open ( FH, "<$fn" ); while (<FH>) { chomp; s/\W/_/g; my $new_empty_file = substr( $_, 0, 200 ); my $target = "$new_empty_file"; if ( $seen{$target} && ($fn ne $seen{$target}) ) { $seen{$target} = "$seen{$target} and $fn both +have :$target\n"; open ( DUP, ">>dups_found_safe" ); print DUP "$seen{$target}"; close DUP; } else { $seen{$target} = "$fn"; } } }
... thanks Joost and Monks!

Replies are listed 'Best First'.
Re^3: Super fast file creation needed
by chromatic (Archbishop) on Oct 19, 2007 at 05:37 UTC

    Opening and closing your files for appending inside the loop is unnecessary. You won't get a huge speed boost by moving the opens out of the loops, but it will be an improvement.