How to rotate and compress log files

andreas1234567 has asked for the wisdom of the Perl Monks concerning the following question:

Honorable Monks,

I have used Log::Log4perl and Log::Dispatch::FileRotate for a long time, but have recently run into problems running short of disk space. I expected it to be simple to not only rotate but also compress the log files, noting that such functionality is included in the TODO.

Searching the intarwebs revealed first of all that I was not the first to think of the matter, secondly that it seems harder than I expected.

As I understand it the main challenge is to prevent a long running compression process to block the logging itself.

Adding that I operate mainly in Unix/Linux-land, how should I proceed?

--
No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]

Comment on How to rotate and compress log files

Replies are listed 'Best First'.
Re: How to rotate and compress log files by shmem (Chancellor) on Aug 12, 2010 at 19:19 UTC
Adding that I operate mainly in Unix/Linux-land, how should I proceed? with logrotate(8) (`man logrotate`) - that compresses stuff, too.	[reply] [d/l]
Re^2: How to rotate and compress log files by dont_you (Hermit) on Aug 12, 2010 at 23:40 UTC
Check the 'copytruncate' option of logrotate, it avoids the hassle of reopening log files after rotation (otherwise, long running processes must be signaled to do that).	[reply]
Re^2: How to rotate and compress log files by andreas1234567 (Vicar) on Aug 13, 2010 at 11:03 UTC
with logrotate(8) (man logrotate) - that compresses stuff, too. Am I correct to assume I should then simply log to a file without rotation using log4perl, then let `logrotate` handle both rotation and compression? -- No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]	[reply] [d/l]
Re^3: How to rotate and compress log files by shmem (Chancellor) on Aug 14, 2010 at 09:57 UTC
Yes. No point in re-inventing this wheel IMHO if your OS provides a solid solution. Remember: Laziness, Impatience and Hubris :-)	[reply]
Re: How to rotate and compress log files by afoken (Chancellor) on Aug 14, 2010 at 07:29 UTC
Hmmm, may be you want to have a look at multilog? Pipe your logging data into a properly configured multilog and let multilog handle writing log files and time stamps. Combine with the remainder of daemontools to create daemons with reliable logging. Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply]