Warning: I have little to say of a practical nature here. I'll just try to explain why what you think you want to do is probably not the way you should try to do it.

The URL of the referring page is not specifically a creature of the CGI protocol (it's defined by the HTTP protocol) the referrer is passed *to* the webserver by the client (the browser)¹, so an uncooperative client needn't provide that information. The environment variable REFERER, which is available to CGI programs, is where to look. In Perl, that's in the %ENV hash, i.e. $ENV{REFERER} holds the value of this variable.

The REFERER (yeah, that's how it's misspelled =) history isn't quite the same as the browser's history, either. Basically, the idea is that you have to follow a link, submit a form, or be redirected from a page to the current page for it to be the REFERER of the current page (so, e.g. if you type in a new URL, the page you were on when you typed that URL in will *not* be the referer of the new page; but it will be the previous one in your browser's history list, and it will be the one your browser attempts to display if you hit the "back" button). Otherwise, the value of REFERER is generally available to the server and (e.g.) can be included in the webserver's logs. You'll have to check with the admin of your webserver if you want that kind of information. There are log analysis tools that attempt to track the length of a client's "sessions" but they rely on a range of strategies, not just the referrers.

I don't believe that even with browsers that implement Javascript, the history is *by default* available to you. And if it is (and it may well be, it's been a while since I checked this sort of thing), that's criminally bad design.

Given all those caveats, if you *must* track user's progress ... and I'll assume you don't have nefarious purposes in mind -- is to do it from the server side. Set a cookie with a unique ID, and on each new request, add the URL of the request to the (ordered) list of things as the request is made. This will of course require that you have the ability to read the cookie on every request and modify your logs. This wouldn't be too hard to do in mod_perl, which allows you to define handlers for each phase of the Apache request cycle. That's pretty involved though.

a browser client could have a  toggle switch 
for browsing openly/anonymously, which would 
respectively enable/disable the sending of 
Referer and From information.
[download]

HTH.

perl -e 'print "How sweet does a rose smell? "; chomp ($n = <STDIN>); 
+$rose = "smells sweet to degree $n"; *other_name = *rose; print "$oth
+er_name\n"'
[download]

                - Ant
                - Some of my best work - Fish Dinner

Just to toss one more option into the pile....

I was quite pleased with the Analog logfile analyser. I was especially impressed with its ability to quickly churn through million line logfiles. Here are some sample stats linked to from the Analog main site.

-Blake

Well, the best way on getting site wide statistics is by using a weblog parser. A good one to try is awstats, which can provide some general stats on your website. It requires apache, but you should be using that anyways ;)

If you are looking for an alternative solution, I have also written a weblog parser that has some additional functionality in some areas. It has your standard browser/os/hits/graphs/etc, plus a few unique features like a hierarchy view. It also requires apache.

If by chance you don't have CGI available to you or you can't use the Apache modules, you could right a script that cookies the user and logs the page. You could call this script by inserting a 1x1 shim image onto each page. It's not the best solution but it works 99% of the time.

A lot of the sites we work on believe it or not do not have any CGI access for any users for security reasons. In these situations, we have to graft the functionality on in manners such as this.

-Lee

"To be civilized is to deny one's nature."

#!/usr/bin/perl
# v0.1 April 18 2001
# dennis@durrow.com
# Read apache log files
# and tell me unique visits per day
# and tell me the most popular page


open(LOG_FILE,"/var/log/httpd/access_log") || die "Can't open";

$totalsize = 0;
$count_not_found = 0;
$count_unique = 0;
$countrtget = 0;
$countrtpost = 0;
$countrtunknown = 0;

$counthttp1 = 0;
$counthttp11 = 0;
$counthttpunknown = 0;
$countsearchengine = 0;

while($line = <LOG_FILE>)
{
    chop($line);
    $count++;

    #break apart into the categories
    ($ip, $dash1, $dash2, $date, $tzoffset, $requesttype, $url, $http_
+type, $status, $size) = split(/ /,$line);

    print "$ip $url\n";

    if($url eq "/robots.txt")
    {
        $countsearchengine++;
    }
    if($count < 2)
    {
        $start_date = $date;
    }

    if($status eq "404")
    {
        $count_not_found++;
    }
    if($requesttype eq "\"GET")
    {
        $countrtget++;
    }
    elsif($requesttype eq "\"POST")
    {
        $countrtpost++;
    }
    else
    {
        $countrtunknown++;
    }
   
    if($http_type eq "HTTP/1.0\"")
    {
        $counthttp1++;
    }
    elsif($http_type eq "HTTP/1.1\"")
    {
        $counthttp11++;
    }
    else
    {
        $counthttpunknown++;
    }

    $totalsize += $size;
     
};



print "\nAccess log Results\n";
print "=====================\n";
print "Log Start: $start_date\n";
print "Log End: $date\n";
print "# of possible web crawlers: $countsearchengine\n";
print "# of unique hits: $count_unique\n";
print "# of get request types: $countrtget\n";
print "# of post request types: $countrtpost\n";
print "# of unknown request types: $countrtunknown\n";
print "# of 404 errors: $count_not_found\n";
print "# of Requests: $count\n";
print "# of bytes transferred: $totalsize\n";
print "# of 1.0 HTTP Request Types: $counthttp1\n";
print "# of 1.1 HTTP Request Types: $counthttp11\n";
print "# of unknown HTTP Request Types: $counthttpunknown\n";
[download]

Edit by tye to change PRE to CODE tags

Sorry about that formatting :) My bad... I promise to use CODE instead PRE from now on...

#!/usr/bin/perl
# v0.1 April 18 2001
# dennis@durrow.com
# Read apache log files
# and tell me unique visits per day
# and tell me the most popular page


open(LOG_FILE,"/var/log/httpd/access_log") || die "Can't open";

$totalsize = 0;
$count_not_found = 0;
$count_unique = 0;
$countrtget = 0;
$countrtpost = 0;
$countrtunknown = 0;

$counthttp1 = 0;
$counthttp11 = 0;
$counthttpunknown = 0;
$countsearchengine = 0;

while($line = <LOG_FILE>)
{
    chop($line);
    $count++;

    #break apart into the categories
    ($ip, $dash1, $dash2, $date, $tzoffset, $requesttype, $url, $http_
+type, $status, $size) = split(/ /,$line);

    print "$ip $url\n";

    if($url eq "/robots.txt")
    {
    $countsearchengine++;
    }
    if($count < 2)
    {
    $start_date = $date;
    }

    if($status eq "404")
    {
    $count_not_found++;
    }
    if($requesttype eq "\"GET")
    {
    $countrtget++;
    }
    elsif($requesttype eq "\"POST")
    {
    $countrtpost++;
    }
    else
    {
    $countrtunknown++;
    }
   
    if($http_type eq "HTTP/1.0\"")
    {
    $counthttp1++;
    }
    elsif($http_type eq "HTTP/1.1\"")
    {
    $counthttp11++;
    }
    else
    {
    $counthttpunknown++;
    }

    $totalsize += $size;
     
};



print "\nAccess log Results\n";
print "=====================\n";
print "Log Start: $start_date\n";
print "Log End: $date\n";
print "# of possible web crawlers: $countsearchengine\n";
print "# of unique hits: $count_unique\n";
print "# of get request types: $countrtget\n";
print "# of post request types: $countrtpost\n";
print "# of unknown request types: $countrtunknown\n";
print "# of 404 errors: $count_not_found\n";
print "# of Requests: $count\n";
print "# of bytes transferred: $totalsize\n";
print "# of 1.0 HTTP Request Types: $counthttp1\n";
print "# of 1.1 HTTP Request Types: $counthttp11\n";
print "# of unknown HTTP Request Types: $counthttpunknown\n";
[download]