Limiting the size of log files...

SteveTheTechie has asked for the wisdom of the Perl Monks concerning the following question:

My Perl code produces some log files that can grow to be pretty large. Right now, I am manually deleting them occasionally, but what I would prefer is some automatic routine that I could call that would inspect the size of a log file and chop off a certain amount at the *beginning* of the file (oldest info) to keep it within desired size limits. In other words, I really do not want to delete the log file, just get rid of enough info at the beginning of the file to keep the size under control.

I am thinking that I will need two size values, the "trigger" file size and the chopped down size limit... For example, 1MB and 500KB, respectively. If I only use one value, then I will be running the routine too frequently... (using two values is a performance tweak)

I am thinking I will check the file size (not sure how to do this in Perl), detect if it is over the "trigger" size, then determine the difference between actual size and the chopped down size. I am thinking I would use that difference as a file seek value to locate the file pointer in the file being trimmed down. However, as I am not sure if I can delete on the front end of a file, I am thinking I will need to basically find the first end of line character after the seek point (text files--do not want partial lines), then open a new file and basically make a copy of the information from after the end of line into the new file. After that I can delete the old log file. Still debating on whether I should try to set the time stamp of the new log file to be the same as the old log file. (Don't know if Perl allows for manipulating file timestamps.)

So... I am looking for some feedback and input on my approach. I am sure others would probably benefit from this also. Am I making this too complicated? Is there a simpler approach?

Comment on Limiting the size of log files...

Replies are listed 'Best First'.
Re: Limiting the size of log files... by soonix (Chancellor) on Nov 03, 2013 at 13:19 UTC
In addition to ww's comment: the acto of transferring the data from the old log file to the new needs additional space, causes I/O and takes time. All three points could introduce more errors. So, log rotation, as recommended by Anonymonk seems to be a far better strategy.	[reply]
Re^2: Limiting the size of log files... by SteveTheTechie (Novice) on Nov 03, 2013 at 23:15 UTC
Good points. I will look at the module. If I think it is well written and well suited to the task, I will give it a go.	[reply]
Re: Limiting the size of log files... by Laurent_R (Canon) on Nov 03, 2013 at 10:36 UTC
I am thinking I will check the file size (not sure how to do this in Perl)... The `-s` file test operator will return the size of your file. You may also use the stat function. In general terms, the approach you are contemplated should work, but you should probably use a module such as the one already suggested.	[reply] [d/l]
Re^2: Limiting the size of log files... by SteveTheTechie (Novice) on Nov 03, 2013 at 22:47 UTC
Thanks!	[reply]
Re: Limiting the size of log files... by Anonymous Monk on Nov 03, 2013 at 10:04 UTC
log rotate, log rotate	[reply]
Re^2: Limiting the size of log files... by SteveTheTechie (Novice) on Nov 03, 2013 at 22:46 UTC
Thanks... Was not aware of this module. ;-) I had a hunch someone probably had dealt with this issue before...	[reply]
Re: Limiting the size of log files... by ww (Archbishop) on Nov 03, 2013 at 12:07 UTC
And if you implement your plan, how are you going to read the information at the beginning of the log when some error (grotesque -- or obscure -- or even just a momentary brain-glitch or bad install) produces so many log entries that you reach or exceed your "trigger size?" Granted, having a single program produce a MB of log entries probably not likely... but what if?	[reply]
Re^2: Limiting the size of log files... by SteveTheTechie (Novice) on Nov 03, 2013 at 23:10 UTC
"grotesque"... nice Halloween word... ;-) You make a good point--can we say large trigger point perhaps? I am working with Perl code that took over/"inherited" (not original developer), so I am trying to make it more manageable. What I inherited is, shall we say "grotesque", oh there is that word again... lol I am trying to bring some order and manageability to the chaos and madness that is currently in the code. All with being basically a Perl newbie. I currently have a test log file and some simple testing routines that I created. The code I inherited had 5 different test log files for no good reason, and a myriad of junk testing code that was sometimes active, sometimes commented out--I stripped that out and implemented my stuff that I can control much more easily. Yeah, I know that there are modules... While some of them look good, frankly some of them look like the developer had too much time on their hands. :-D Anyway, if there is an error log file somewhere on the server I am FTPing into, I have not yet found it. Also, the die's and miscellaneous error output's are not consistent in approach at all. I will probably have to create something around that, also. I also need to implement a consistent "fail strategy" (error exit) that will log error messages complete with date/time, context info (routine & caller), and will do a graceful shut down of MySQL (commit outstanding transactions) and do some sort of graceful exit with appropriate user message.	[reply]
Re: Limiting the size of log files... by Anonymous Monk on Nov 04, 2013 at 14:44 UTC
Most shops run a `logrotate` daemon which can rotate the files, keeping so-many recent ones on hand, compressing older ones, and so on. You simply add your files/directories to the list that this existing daemon is already processing. No Perl/programming required. Windows has a similar feature.	[reply]