I'm crunching some web logs. However we've got two boxes (one an IIS and the other an Apache server). In order for our analysis tool to give us info, the logs have to be combined and sorted.

So right now I've got a perl script which pulls the logs from each box, combines them into one giant log, and puts that in the watch directory of the analysis software.

The problem is we only do this once a month and the combing step takes a while. I'd like to let the user know that something is actually happening and give them an idea of the progress. Here's the section of the 

my $cmd = "cat /tmp/win.log /tmp/lnx.log | grep -v "/" . $nextmonth . 
+"/" | sort -k4.2 > /export/w3logs/" . $year . $month . ".log";
system($cmd);

[download]
The only way I can think of making a progress bar for this, is to pull it all into perl. The basic outline would be something like: 
open the log files and the output file.
read in each log file into a hash with the key based on the date of th
+e item
sort the hash keys and write the values to the output file

[download]

Now from previous experience I know the above outline will be a lot slower than how it is being done now. Also to note, we're talking about 2-4 Gb logfiles.

Is there a faster way of doing this or there some way to make a progress bar and still use the system call?

In reply to progress bar for a system command by xorl

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.