It may seem counter intuitive, but I think you'll get a fair bang for your buck by opening an output pipe from the locate(1) program. It's designed to be efficient at answering these kinds of queries.

On the other hand, if you search for a given string "abc", it will output the name of the file even if "abc" is part of the pathname, not only the filename. But you could filter out the false hits by taking the basename of the returned file and checking it against your target string when you read it in.

# remember to untaint $str
open LOC, "/usr/bin/locate $str |" or die "Cannot open input pipe: $!\
+n";
while( <LOC> ) {
  chomp;
  my $file = basename $_;
  next unless index($filename, $str) > -1;
  # output this filename
}
close LOC;
[download]

I'm not convinced a DBM hash will speed things up: at the end of the day you're still doing a linear scan of 200K records. (Assuming that since you're doing an index partial substrings of names are allowed).

Antother idea would be to simplify the location file contents, to reduce its size. Just store the name of each file. If and when a person hits it in a search, stat the file at that point in time (the results will be fresher to boot). This will let you store more files per disk block, thus less disk blocks to store all of them.

Depending on a certain ratio of CPU to disk I/O performance that only testing can demonstrate, it may be faster to zip the file and read and decompress it on the fly, on the assumption that the extra CPU cost is offset by reducing the number of disk blocks read.

    my ($filename, $rest) = split /,/, $_, 2;
    if (index($filename, $to_find) > -1) {
      my ($cms, $path, $size, $day, $time) = split /,/, $rest;
[download]

  open(F, "+< $infile"); 
  while (sysread F, $_, 32768) { 
    $_ .= <F>;
    next unless /\Q$to_find\E/; # quickskip

    for(grep /^[^,]*?\Q$to_find\E/, split /\n/, $_) {
      ($filename, $cms, $path, $size, $day, $time) = split /,/;
      $href = "file:\\\\netd\\data".$path."\\$filename";
      $href =~ s/\s/%20/g;
      $table .= "<TR><TD><A HREF=\"$href\">$path\\$filename</A><TD>$si
+ze<TD>$day $tim
+e</TR>";
    }
  }
  close(F);
[download]

  open(F, "+< $infile"); 
  while (sysread F, $_, 32768) { 
    $_ .= <F>;
    next unless /\Q$to_find\E/; # quickskip

    while(/^([^\n,]*?\Q$to_find\E[^\n]*)/gm) {
      ($filename, $cms, $path, $size, $day, $time) = split /,/, $1;
      $href = "file:\\\\netd\\data".$path."\\$filename";
      $href =~ s/\s/%20/g;
      $table .= "<TR><TD><A HREF=\"$href\">$path\\$filename</A><TD>$si
+ze<TD>$day $tim
+e</TR>";
    }
  }
  close(F);
[download]

Makeshifts last the longest.

Thanks for your reply - I had thought about limiting the split, but I didn't believe it would enhance performance much. It does appear to help though. The simple reduced split you noted seems to improve performance by about 20%. However, the other snippets you provided don't work. I'm trying to figure out why. They return no matches for files I know exist.

Odd. Admitted, I didn't test them except for the regex in the last snippet when I posted them, but I did now and they seem to be working. *shrug* There's nothing obvious I can spot either.

Makeshifts last the longest.

Abigail

The bottleneck here seems to be processing speed, as opposed to memory or disk space. I suppose that could change if the index file grows to over 100M. It might increase efficiency somewhat to search a file containing only filenames, and then to retrieve path into from a DBM hash using the filename as a key, (with a little workaround for duplicate filenames in different directories). However, I think my initial idea of breaking the index file into separate, smaller indexes and choosing which to look through based on the search string may produce a greater increase in speed. If I didn't want to search for substrings in the filename this would be easy. The problem is that people don't always know the exact filename they are looking for, and it is useful to be able to display multiple matches (all files of a given extension, for example).

I am a lowly knuckle-dragging grunt and access to a database server is not an option. This organization regulates that sort of thing tightly.

Would anyone object to you installing DBI with DBD::SQLite? It installs a data base server, but it's a really tiny one, and a DB is conveniently stored as a single file.In short it means no administrative hassle and probably no need to ask anyone about it. SQLite implements the LIKE operator and wildcards in queries, so it should help you with your problem.

sysopen(F, $infile, 0); 
while (sysread F, $_, 32768) { 
  next unless /\Q$to_find\E/;
  while(/^([^\007,]*?\Q$to_find\E[^\007]*)/gm) {
      ($filename, $cms, $path, $size, $day, $time) = split /,/, $1;
[download]