first off: always use strictures (use strict; use warnings;).

my $out = "/home/users/user/tmp/$progname_$today.dat";
[download]

should be

my $out = "/home/users/user/tmp/${progname}_$today.dat";
[download]

or $progname_ will be seen as the variable name rather than $progname. Strictures should also force you to think a little more about lifetime of variables and how you pass information around to different parts of your program.

call_dir($p=1)
...
sub call_dir {
    if ($p == 1)
...
<p>would be better as:</p>
<c>
call_dir(1)
...
sub call_dir {
    my ($p) = @_;

    if ($p == 1)
[download]

although passing an array into the find call would be even better:

my $main = "/home/users/user";
my @rootDirs = ("$main/tmp/", "$main/exe");
...
find ({wanted => \&data_for_path, follow=>1, follow_skip=>2}, @rootDir
+s);
[download]

and avoids the interesting mismatch between the calls to call_dir and the implementation (3 calls, 2 valid return values).

However, I'd be inclined to read and parse today and yesterday's files in parallel to detect differences. That avoids having any more than a few lines of data in memory at any given time, but may make the parsing a little more interesting.

If instead you read the files in parallel as suggested above, but load a complete directory of files at a time into two arrays of lines (one for today's files and one for yesterday's) you could then use Algorithm::Diff to do the heavy lifting in differencing the two file sets. That limits the data in memory to one directory worth of files (x 2 - one image for each day), but probably simplifies the diff parsing substantially.

How about breaking the problem down to three separate procedures:

1. Create a sorted list of all the directories of interest on each scan.
2. For each directory, create a separate sorted list for the symlinks and data files in that direcory.
3. To find differences between yesterday and today, use the standard unix/linux "diff" tool on consecutive directory lists, and on consecutive file lists for each directory.

File::Find will be good for the first step, though you might want to consider just using the available unix/linux tools:

  find /path/of_interest -type d | sort > /path/for_log_info/dirlist.y
+ymmdd
[download]

Using "diff" on two consecutive "dirlist" files will reveal the addition or removal of directories.

For step 2, I would do something like:

open( DLIST, "<", $dirlist );
while ( my $dir = <DLIST>) {
    chomp $dir;
    opendir D, $dir or do {
        warn "opendir failed on $dir: $!\n"; next;
    };
    ( my $file_list_name = $dir ) =~ tr{/}{%};
    open( FLIST, ">", "$log_path/$file_list_name.$today" )
        or die "cannot write to $log_path/$file_list_name.$today: $!\n
+";

    for my $file ( sort grep { !-d "$dir/$_" } readdir( D )) {
        # check for symlink vs. datafile
        # gather other stat info as needed,
        # print a nicely formatted line to FLIST
    }
    close FLIST;
    closedir D;
}
close DLIST;
[download]

GrandFather thanks for the advice on proper programming I cleaned up a lot of that code. I've become too lazy and that gave me a little kick. I'm interested in Algorithm::Diff but will need time to investigate.

graff that's just way too simple I'm incredibly impressed and I've already got a good start on it. I like it, and it seems incredibly fast, however I find a few caveats. I need to use find -follow due to my file structure(need to follow links pointing to other partitions), but I receive loads of errors when it finds null links. How can I manage this and output to a file or is there a different approach. Also, I had to use an egrep -v to filter out directories I don't want. Any other solution as I always try to stay away from using system commands within perl?

UPDATE: If I use find -type d -follow and it detects a change while scanning it dies and the rest of my program continues on. Thus leaving me with only a partial directory scan. I'm trying to work in File::Find, but any other suggestions?

If your "path of interest" contains symlinks to directories on other disk volumes or partitions, and if it turns out that some of the symlinks point to non-existent or unavailable paths (e.g. pointing to a volume that isn't currently mounted), one solution would be to stick with my 3-step procedure (and still don't use  -follow in the find command at step 1), but subdivide step 2 into a few separate steps:
• 2. For each directory, scan its immediate contents with readdir()

  • 2.a. Accumulate data files and symlinks in separate lists

  • 2.b. For each element of the symlink list, determine its target, determine whether the target exists, and if so determine whether it is a data file, a directory, or another symlink. The output listing for the directory currently being scanned should contain just this information about the symlinks.

  • 2.c. For each element of the data file list, get the other stat info you need and report this in the listing for the current directory.

Based on that treatment, you'll know whether symlinks have been added or removed from within the directory tree of interest, you'll know which ones are broken, and for the ones that work, you'll know what sort of thing they point to, and (comparing listings from consecutive days) whether there has been a change in their target path. That should be all you need to know about symlinks per se.

For the ones that work, you won't know from the symlink listing whether the content of the target path has changed (i.e. change of a data file or change of directory contents). But if the target is a datafile or directory within the current tree of interest, that information will be available elsewhere in your overall output.

And if the target is on some other disk volume/partition, running this same process on that volume (on the relevant directory tree of interest) will tell you what you want to know about that content.

There will be some serious work in keeping all this information organized and managed properly, to make sure that everything gets covered with (ideally) no redundancy -- e.g. a process that scans the output listings for a given directory tree and launches this same process on other volumes as needed to cover all the cross-volume symlinks. I'll leave that as an exercise. ;)

(Good luck with the case of "symlink points to symlink points to symlink...", and with cases of "relative" as opposed to "absolute" target paths. You may need to do the equivalent of bash's "pushd / popd" within your perl script to test for target existence of relative-path symlinks.)