Short answer: the pages are indexed in advance and these indices are stored in a database; the user's specified search terms are compared to the indices; the results are fairly speedy.

Long answer (or excuse for lack thereof): web searching is a hard and complicated problem. Here's a decent introductory article. Do some web searching on the subject. You'll find a lot of stuff. And if you had a good and novel approach to making it more effective or efficient, Google would probably be very happy to pay you handsomely.

Heh! Good segue into perl :)

Update: What? You guys don't think that was a good segue into perl from what is essentially a non-perl-related node?

PerlJam

Non-Perl answer, take a look at gnu id-utils. It munches through a tree and indexes words in a dbm file. (Very useful set of programs for some purposes).

One of the more exotic search technologies is employed by Google and is known as PigeonRank. It employs a distributed multi-agent system to rank search results. The agents are widely available but do need to be trained!!

inman

#!/usr/bin/perl
#web_search -- a program to simulate how sites index the 'Net.
#invocation: <web_search *.pl> to index all the *.pl files
#in the current directory.

use strict;

my %words;

### Index the files

undef $/; #file slurp mode

for my $name (@ARGV)
{
    open F,"<$name" or die "couldn't open $name\n";

    map { push( @{$words{$_}}, $name) } split(/\s+/, <F>);
    
    #print "$_ -> @{$words{$_}}\n" for keys %words; #pretty print
}

### Ask for search keywords

$/="\n"; #unslurp for user input

print "\nEnter you list of search words: ";

open I, "-";

while(<I>)
{
    for (split)
    {
        if (defined($words{$_}))
        {
            print "$_ found in ".join(", ",@{$words{$_}})."\n";
        }else
        {
            print "$_ not found\n";
        }
    }
}
[download]

New and improved version...
Read more... (2 kB)