chetanspecial has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,
i have written a recursive based directory indexing program which indexes all folders,subfolders and files(specifically for "D"). the problem is the program completes abruptly after a certain depth of folders.i cannot observe any specific point at which it stops.in D containing over 30000files and folders the program runs only upto 1000 files without any error being thrown. i have to make it clear that no errors are produced during execution of program 
@mainarray=("D:");
sub indexcurrent;
indexcurrent("D:");
writefile();

sub indexcurrent
{
   if(opendir(DIR,$_[0]))
   {
    local @current=readdir(DIR);
        for($i=0;$i<=$#current;$i++)
        {
            //this if is because the readdir function   
            //returns "." and ".." also as folders
            if(($current[$i] ne "..")&&($current[$i] ne "."))
            {
                $temp=$current[$i];
                $current[$i]="$_[0]\\$temp";  
\\Here i append the file name with current folder location
                push @mainarray,$current[$i];
                indexcurrent($current[$i]);
            }
        }
   }
}
//to write the array of indexed files into a file
sub writefile
{
    open(File,">E:\\project\\index.txt");
    print File join("\n",@mainarray);
    close (File);
}
print "complete";
<>

[download]
1.The writefile function displays over 1000 files instead of over 30000
2.I use push so that i can keep the current location and it does not matter whether i write the file in the beginning or the end as i can see the same progress.
3.Also i actually need to use mainarray to make a file search.that is why i use push.
This program has been also written in C++(dev) and has been compiled.It runs indexing the whole directory. I have used the same conditions and variables. I started it out as a perl program,then to find out the reason why it didnt work i tried it in c++. please help

Replies are listed 'Best First'.
Re: File Indexing program
by psini (Deacon) on Jun 24, 2008 at 19:18 UTC

    Have you used strict and warnings?

    At a first look I see two potentially fatal problems here:

    The use of local @current instead of my @current could work fine, but should be avoided and I'm not sure what can it do in a (deeply) recursive function.

    The same applies to $temp and @mainarray, but this is not fatal as the first is not reused after recursion and the latter is intended to be a global. But it is anyway bad practice.

    Bottom line: strict&warnings are your friends, (almost) always.

    $i is not declared lexical (my) but is a global, so every recursion corrupts the value of $i of the previous and pending executions

    Rule One: "Do not act incautiously when confronting a little bald wrinkly smiling man."

[reply]
[d/l]
[select]
      Thank you very much psini and that was fast. the only problem in the program was $i which was not localised.
      i must change $i to my $i.
      but i also observed that this was a lot slower than compared to a c++ program and took a lot of time.but anyway it worked perfectly. thanks again
[reply]

        No, allow me to repeat: $i as global was the main problem in your program, it prevented the program to work.

        But it is not the only problem: all the other things I mentioned are time bombs waiting for the first time you change something in the code to explode.

        Rule One: "Do not act incautiously when confronting a little bald wrinkly smiling man."

[reply]
Re: File Indexing program
by pc88mxer (Vicar) on Jun 24, 2008 at 19:32 UTC
    Why not see how File::Find performs: 
    use File::Find;

sub wanted {
  warn "adding $File::Find::name\n";
  push(@mainarray, $File::Find::name);
}

find(\&wanted, "D:");
writefile();
print "complete\n";
<STDIN>;

    [download]
[reply]
[d/l]
[select]
      I second this. It'll not only do a better job in less space, but it will avoid some of the pitfalls of doing it by hand. You will eventually run into trouble with symlinks and things if you do this by hand.

      There are times when it's appropriate to do so. Say you have a well behaved directory with hundreds of thousands of files in it and you need just one type... You might get a slight speed advantage with readdir/stat...

      Other than things like that, there's rarely a need to use readdir by hand.

      -Paul

[reply]
Re: File Indexing program
by moritz (Cardinal) on Jun 24, 2008 at 19:09 UTC
    the problem is the program completes abruptly after a certain depth of folder

    What does that mean? Is there an error? What's the return value for the program? (exit status).

    Also I see no reason why you have to push things onto an array and then later write it to disk - you can just write it to file immediately. Then you'll also see which progress it made.

    BTW the recommended way to walk through a directory tree is to use File::Find.

[reply]
Re: File Indexing program
by starbolin (Hermit) on Jun 25, 2008 at 19:54 UTC

    $i was being clobbered as psini pointed out but this was only one major problem. The second problem is that your filehandle 'DIR' is getting clobbered. I think that the program still is not working correctly. To demonstrate: 

    open FH, "<foobar" or die;
my @stats = stat FH;
print "First inode ", $stats[1], "\n";
{
   open FH, "<foo" or die;
   my @stats = stat FH;
    print "Second inode ", $stats[1], "\n";
}
@stats = stat FH;
print "First inode ", $stats[1], "\n"; # FH has been clobbered.

__END__

....
First inode 683751
Second inode 684162
First inode 684162

    [download]

    s//----->\t/;$~="JAPH";s//\r<$~~/;{s|~$~-|-~$~|||s |-$~~|$~~-|||s,<$~~,<~$~,,s,~$~>,$~~>,, $|=1,select$,,$,,$,,1e-1;print;redo}
[reply]
[d/l]

Back to Seekers of Perl Wisdom