Re-reading all your user data files once for each record in the log file is inefficient. It would be more efficient to build a hash of the data on individual users first, then process the log file, pulling user data from the hash.

The following is untested but should be close to working. It assumes you have sufficient memory to hold all the data in memory. If you don't, you might try tied data structures (e.g. Tie::File::AsHash), and pre-sorting the log file or post-sorting the output file, perhaps using the system sort utility.

use strict; use warnings; # Create a hash of arrays of data about individual users my %user_data; foreach my $file ( "Carlo_Arnold_2.232_Final.txt", "Sohaib_Ahmad_2.225_Final.txt" ) { open(my $fh, '<', $file) or die "$file: $!"; foreach (<$fh>) { chomp; foreach ( map { [ split(/:/, $_) ] } <$fh> ) { push(@{$user_data{$_->[0]}}, $_->[1]); } } close($fh); } # An input LOG file which holds information about different users. my $file_name = "QoEWeb_DB_Log.txt"; open(my $sw, '<', $file_name) or die "$file_name: $!"; #Will write the result into an OUTPUT file. open(my $file, '>', 'output.txt') or die "output.txt: $!"; foreach ( sort { $a->[8] <=> $b->[8] } map { chomp; [ split /:/ ] } <$sw +> ) { my ($field1, $field2, $key1, $field5) = @$_[2,4,8,6]; foreach (@{$user_data{$key1}}) { #Printing the desired result from both batch of input +files. print $file "$field1:$field2:$key1:$field5:$_"; } } close $file or die "output.txt: $!"; close $sw or die "$file_name: $!";

In reply to Re: a little problem with sorting my data by ig
in thread a little problem with sorting my data by aash08

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.