moving old files

pbaumgar has asked for the wisdom of the Perl Monks concerning the following question:

Hello everyone: I'm working on a script that will go through my apache httpd log directory and gzip and move log files older than 24 hours to an archive directory. Since I'm new to perl programming I was hoping others could offer advice on "best practices" or "best way to do it" on the perl code I've written so far. Here's my script so far:

#!/usr/bin/perl -w
use strict;
use File::Copy;

my $log_dir = "/etc/httpd/logs/2006";
my $archive_dir = "/etc/httpd/logs/archive";

#open logs directory and get files older than 24 hours
opendir(LOGS, $log_dir) 
    or die "Directory could not be opened: $!";
my @files = grep { -M "$log_dir/$_" > 1 } readdir(LOGS);
closedir(LOGS)
    or die "Directory could not be closed: $!";

#check each file in logs directory
my @to_move;
foreach my $file (@files) {
    $file eq '.' || $file eq '..'
    or push @to_move, $file;
}

#perform file test; compress file; move file
foreach my $file (@to_move) {
    if (! -l $file)
    {
    chdir $log_dir
        or die "Can't chdir to $log_dir: $!";
    system "gzip $file";
    move "$file.gz", $archive_dir;
    } else {
    die "Can't move $file: $!";
    }
}
[download]

Comment on moving old files Download Code

Replies are listed 'Best First'.
Re: moving old files by Paladin (Vicar) on Apr 20, 2006 at 17:05 UTC
`#check each file in logs directory my @to_move; foreach my $file (@files) { $file eq '.' \|\| $file eq '..' or push @to_move, $file; }` [download] The code in this section is a bit hard to read. A more readable way would be something like: `#check each file in logs directory my @to_move; foreach my $file (@files) { next if $file =~ /^\.\.?$/; push @to_move, $file; }` [download] Or better yet, remove this part, and add it to the readdir earlier. `my @files = grep { -M "$log_dir/$_" > 1 && !/^\.\.?$/ } readdir(LOGS);` [download] A few other things: Use the LIST form of system when you are able to: `system "gzip", $file` `chdir` to the log dir once at the start of the script instead of once for each file processed In the `else` block you use `$!` but there is no guarantee that it will be set to a valid value before hand. You know at that point that the file is a symbolic link, so just print that. Nit Pick: You seem to use 2 different indent styles, one for the `foreach` and another for the `if` (You indent the body of one, but not the other). It usually helps to be consistant.	[reply] [d/l] [select]
Re: moving old files by rhesa (Vicar) on Apr 20, 2006 at 17:21 UTC
I'd use shell scripting for this, something like this: `log_dir = "/etc/httpd/logs/2006" arch_dir = "/etc/httpd/logs/archive" find $log_dir -mtime +0 -type f -exec gzip {} \; -exec mv {}".gz" $arc +h_dir \;` [download]	[reply] [d/l]
Re: moving old files by davidrw (Prior) on Apr 20, 2006 at 20:04 UTC
or use `logrotate` .. very nice & powerful (see man page)	[reply] [d/l]
Re: moving old files by brd (Acolyte) on Apr 21, 2006 at 00:19 UTC
It would be easier to use something like cronolog (http://www.cronolog.org). Logrotate would also work, after moving the files remember to HUP Apache though. Sorry this is not a Perl solution, but there is no reason to reinvent the wheel. :)	[reply]
Re^2: moving old files by superfrink (Curate) on Apr 21, 2006 at 01:59 UTC
++ brd. I was just going to recommend chronolog. Chronolog uses the Apache feature of logging to another process instead of to a file. This means you do not have to do: `- rename all of the log files - HUP apache - then archive the old log fles however you like` [download] The best part is apache does not stop reponding for a short while during the HUP. I used to admin for a company that ran several hundred virtual hosts per apache install. That HUP can be noticable. Chronolog takes care of creating and renaming files for you as time goes by. I use something like the following: `LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent} +i\"" combined ErrorLog "\|/usr/local/sbin/cronolog /log/web/domain.com/%Y-%m-%d_err +or.log" CustomLog "\|/usr/local/sbin/cronolog /log/web/domain.com/%Y-%m-%d_ac +cess.log" combined` [download] This setup gives me one access log and one error log per calendar day. I think you can setup log files per hour or minute if you want to.	[reply] [d/l] [select]
Re^3: moving old files by brd (Acolyte) on Apr 21, 2006 at 05:06 UTC
The only problem I have noticed with using Cronolog is that it adds two processes everytime it is used. If you have lots of virtual hosts and a cronolog for the ErrorLog & CustomLog that adds four processes per VirtualHost. Edit: It seems that only on Apache 1.3 does apache create two processes for a piped logging program. It forks a shell and then runs cronolog. I'm seeing this in an Apache 1.3 server: `13311 ?? I 0:00.00 /bin/sh -c /usr/local/sbin/cronolog /usr/log +s/httpd/%Y/%m/access_log 13314 ?? I 0:00.36 /usr/local/sbin/cronolog /usr/logs/httpd/%Y/ +%m/access_log` [download] vs. on Apache 2.0: `29394 ?? I 0:00.04 /usr/local/sbin/cronolog /usr/logs/httpd/%Y/ +%m/access_log` [download]	[reply] [d/l] [select]
Re^3: moving old files by pbaumgar (Acolyte) on May 17, 2006 at 19:33 UTC
Hello again, I've decided to use Cronolog for Apache logging. I have cronolog creating logs with like this: %m-%Y/access_log. So it will seperate all the logs my months. What is the best way to handle the month change in my original script? I'd like at the end of them month to gzip the entire directory and dump it in the archive directory. Do I use a file/directory test to check the timestamp on it? `my $log_dir = "/etc/httpd/logs/2006"; my $archive_dir = "/etc/httpd/logs/archive";` [download]	[reply] [d/l]