Are the files really rotated on the one minute intervals? Or do you just mean that you want to download the big file at one minute intervals, but it gets rotated less frequently (perhaps once per day)?

Assuming the later, you could remember the first line of a file, and the number of the line you last processed. When you fetch the file, you examine it. If it's the same as the line you saved, the file has not been rotated yet, so you know how many lines to skip without parsing - (that's the second value you remembered). If it's not the same, the file has been rotated and you restart the "lines already seen" counter to zero.

Having said that, fetching the webserver logs every minute seems extraordinarily wastefull. The traffic from transfering the logs would soon become a significant chunk of the total site traffic. If you need the up-to-the-minute information, you are much better off talking with whoever controls the server, and arranging for a small monitoring program to read the log file as it is written (perhaps using File::Tail), process as much information as possible, and forward that information to the script on your server.

In reply to Re: Downloading and parsing apache logs by matija
in thread Downloading and parsing apache logs by juanmarcosmoren

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.