Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

comment on

( [id://3333] : superdoc . print w/replies, xml ) Need Help??
Looking for direction and opinions. Background: I need to monitor 100s of application log files on approx 100 Windows 2000/2003 servers. Frequency of monitoring ranges from hourly to daily. Standard functionality in Perl 5.8 is working great from a single centralized server using Windows UNC file pathing to other local servers and log files. Note: monitoring the logs from a centralized server saves me time by eliminating the need for change-management plans to add scripts or perl binaries to 100 validated production server.

Issue: I need to expand my monitoring to include numerous remote servers -- some accessed via slow or bandwidth-impaired links. My problem is not the large remote log file, per se, as only a few new lines are appended hourly or daily. Rather my approach for extracting the new lines from the large log files seems to suck. My current logic to get new lines is:

1) if file modification date has changed, open file, count number of lines and close. 2) if newly-obtained line count differs from last line count, reopen file. 3) read past and ignore old lines. 4) read new lines and analyze patterns. 5) persist new file line count and mod date for next analysis.

This dual read (once for line count, another to get the new lines) is where all my script CPU and wall time is spent and I could obviously try to combine steps 1 - 4 into a single journey through the file. However, before I do that I thought I would ask for suggestions. Is there a better way to periodically extract the new lines from a log file? Again, with the constraint that I not deploy any scripts or perl distros to the local or remote servers where the logs reside?

Thanks for sharing any ideas you may have.


In reply to Extract new lines from log file by smithers

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.