Re-reading all your user data files once for each record in the log file is inefficient. It would be more efficient to build a hash of the data on individual users first, then process the log file, pulling user data from the hash.
The following is untested but should be close to working. It assumes you have sufficient memory to hold all the data in memory. If you don't, you might try tied data structures (e.g. Tie::File::AsHash), and pre-sorting the log file or post-sorting the output file, perhaps using the system sort utility.
use strict; use warnings; # Create a hash of arrays of data about individual users my %user_data; foreach my $file ( "Carlo_Arnold_2.232_Final.txt", "Sohaib_Ahmad_2.225_Final.txt" ) { open(my $fh, '<', $file) or die "$file: $!"; foreach (<$fh>) { chomp; foreach ( map { [ split(/:/, $_) ] } <$fh> ) { push(@{$user_data{$_->[0]}}, $_->[1]); } } close($fh); } # An input LOG file which holds information about different users. my $file_name = "QoEWeb_DB_Log.txt"; open(my $sw, '<', $file_name) or die "$file_name: $!"; #Will write the result into an OUTPUT file. open(my $file, '>', 'output.txt') or die "output.txt: $!"; foreach ( sort { $a->[8] <=> $b->[8] } map { chomp; [ split /:/ ] } <$sw +> ) { my ($field1, $field2, $key1, $field5) = @$_[2,4,8,6]; foreach (@{$user_data{$key1}}) { #Printing the desired result from both batch of input +files. print $file "$field1:$field2:$key1:$field5:$_"; } } close $file or die "output.txt: $!"; close $sw or die "$file_name: $!";
In reply to Re: a little problem with sorting my data
by ig
in thread a little problem with sorting my data
by aash08
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |