Flame has asked for the wisdom of the Perl Monks concerning the following question:

I rather suspect this has already been asked before, however, I've been unable to determine where, so I'll just have to ask again. (Sorry...)

I've started using a custom logging system for a set of online utilitys. This system only uses one file, and does not rotate it. The logfile tends to look something like this:
1015295974;127.0.0.1;1;Viewed Log File
1015296127;127.0.0.1;1;TestJoin added to join queue
1015296134;127.0.0.1;1;Acessed the GuildMaster program.
(in case it matters, that's time,ip,ID,action)

[download]


With each line being a different record. I want to allow the webmaster to determine a max filesize for the log, and have the logfile remove lines from the top if it's too big, until it's the right size. Is there any simple means to take lines off of the top of a file without rewriting the entire file? Or better yet, is there any way to tell how big a file will be once a string is added to it?

Thank you


"Weird things happen, get used to it."

Flame ~ Lead Programmer: GMS

Replies are listed 'Best First'.
Re: Limit a filesize
by belg4mit (Prior) on Apr 09, 2002 at 00:55 UTC
    The easiest way may be the new module Tie::File. Stat the file, if it's too big or will be soon (say with what you are about to append), shift/splice the array, et voila! (I'd simply remove n entries each time, instead of calculating for exact size).

    You can predict size with a stat / -s and then adding length($message).

    --
    perl -pe "s/\b;([mnst])/'\1/mg"

[reply]
      I did a short experement with a text file I have, basicaly I told it to print out the size of the file, and then open it, read it and print the length... they didn't match. I'll admit, I'm currently stuck on Win32, but the file was saved in UNIX (I love TextPad). So I don't quite know what to make of that, or did I misunderstand what you were saying?


      "Weird things happen, get used to it."

      Flame ~ Lead Programmer: GMS

[reply]
        You did misunderstand, or I was unclear. First and foremost the size of a file is not the same as the size of it's contents (especially on some filesystems). But for all but the strictest of purposes it's close enough. What I suggested was something like: 
        #About to append $message, do we have enough room?
if( -s $file + length($message) > $maxSize ){
  #do magic here
}

        [download]

        --
        perl -pe "s/\b;([mnst])/'\1/mg"

[reply]
[d/l]
Re: Limit a filesize
by tachyon (Chancellor) on Apr 09, 2002 at 03:54 UTC

    To remove the top part of a file you need to rewrite it. You can only shorten a file using truncate(). The underlying reason for this relates to how files are stored on a disk. Anyway here is an efficient way to do it using seek(). The first line of the 'truncated' file will be of random length and incomplete. 

    my $logfile = 'c:/test.txt';
my $tmpfile = 'c:/tmp.txt';
my $maxsize = 100;
my $size = -s $logfile;
my $need_to_dump = $size - $maxsize;
if ($need_to_dump > 0 ) {
    open LOG, $logfile or die $!;
    open TMP, ">$tmpfile" or die $!;
    seek LOG, $need_to_dump, 0;
    print TMP $_ while <LOG>;
    close LOG;
    close TMP;
    unlink $logfile or die $!;
    rename $tmpfile, $logfile or die $!;
    unlink $tmpfile or warn "Can't unlink $tmpfile $!\n";
}

    [download]

    cheers

    tachyon

    s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

[reply]
[d/l]
no one's fielded these yet
by Fletch (Bishop) on Apr 09, 2002 at 02:34 UTC
    Is there any simple means to take lines off of the top of a file without rewriting the entire file? Or better yet, is there any way to tell how big a file will be once a string is added to it?

    Short answer the first: no. Longer answer, see: 

    Found in /usr/lib/perl5/5.6.1/pod/perlfaq5.pod
       How do I change one line in a file/delete a line in a
       file/insert a line in the middle of a file/append to the
       beginning of a file?

    [download]

    A file is an ordered stream of octets. Lines are just an illusion imposed by convention.

    As for the second, presuming you're not mucking about with the file pointer and seek()ing about you can use tell() to find out where you are. That and the ever useful + operator along with length( $your_line ) should let you know. If you don't have the file open, -s/stat() would be where to look.

    Update: Someone had mentioned -s/stat(). /me wanders off to get some Windex to clean the monitor and his glasses.

[reply]
[d/l]
[select]
      >A file is an ordered stream of octets. Lines are just an illusion imposed by convention.
      Lunch lines doubly so

      PS> That's very non UTF centric of you ;-)

      --
      perl -pe "s/\b;([mnst])/'\1/mg"

[reply]
Re: Limit a filesize
by Dogma (Pilgrim) on Apr 09, 2002 at 02:00 UTC
    How often are you planning on "pruning" this log? If you want to do this everytime an entry is added to the file you are going to add alot of overhead and disk accesses no matter how you do it. It maybe worth considering a different method if you are performance minded. Myself I would use log rotating because it is low cost and only has to be run once a day. It also makes it very easy to archive logs.

    Cheers,
    -Dogma

[reply]
      I'd consider rotating, but these logs aren't really the kind you'd want lasting copys of, mostly it's just to let the webmaster know where people tend to go, and tell if someone made it somewhere they shouldn't. It's not likely that most people using my program will ever look at the log, even if they bothered to turn it on, so I'd like to keep it from swamping them (this is a space issue as much as overhead problem)


      "Weird things happen, get used to it."

      Flame ~ Lead Programmer: GMS

[reply]
Shuffling data is _expensive_
by RMGir (Prior) on Apr 09, 2002 at 12:02 UTC
    Any solution where you prune the file dynamically is going to be horrifically expensive, I fear.

    If you want to limit space usage to, let's say, 1 meg, why not just set up a rotation among 4 250k files?

    That way, you always have at least 750k of history, and you're not constantly shuffling data around...

    Make that 5 files if you want to guarantee a meg of history, at the cost of possibly 250k extra disk usage. Not too painful.
    --
    Mike

[reply]
Re: Limit a filesize
by talexb (Chancellor) on Apr 09, 2002 at 13:39 UTC
    It sounds like the objective is to limit how big the log file (or files) get. In that case, I'd probably just write to a file named with the current date, then delete anything older than n days using a cron job.

    That way you have a little history that you run some analysis on if you like. It's not going to keep the log files down to specific size, but if you still want to do that, then I'd set the cron job up to delete daily log files until you get down below the maximum size for all existing logs.

    I believe that either of these alternatives would be more efficient then rewriting the entire log file after chopping off the front part.

    --t. alex

    "Here's the chocolates, and here's the flowers. Now how 'bout it, widder hen, will ya marry me?" --Foghorn Leghorn

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom