Dear Monks,

OK, I am using a web scraper, to loop through several pages of about 20 or so different links to individual records. I have three phases, get the pages - separate the individual records, strip those records.

All was going rather nicely, until I hit these darn sites with javascript, and I'm thinking the problem (from what research I've done?) is the cookies - not really the javascript, so this is my scraper here:

#!/usr/bin/perl -w

use strict; 
use LWP::Simple;
use LWP::UserAgent;
use HTTP::Cookies;

my $counter = 0;
my $max_page = 49;  # Maximum page number (number of pages displayed p
+er search)
my $ua = LWP::UserAgent->new;
my $cookie_jar;

## This is a small program that uses a user agent (robot) module to ge
+t web pages
## of multiple record links, and store each web page in it's own seper
+ate file.

while ($counter <= $max_page) { ## loops through all pages of records

    #$ua->cookie_jar(HTTP::Cookies->new(file => "$ENV{HOME}/.cookies.t
+xt"));
    getstore('http://nye.nv.countygovernmentrecords.com/nyecounty/eagl
+eweb/docSearchResults.jsp?page=' . $counter . '&searchId=0', 'test.ht
+ml' . $counter) or die 'Unable to get page';
    $ua->cookie_jar({});
    $counter++;
    print "created test.html $counter\n";
} # end while
[download]

The two cookie jar lines are what I've been playing around with, not sure if they should be in or out of my loop, so one is commented out - also, I tried these two different methods, an empty jar, then the %ENV version. (I also have tried to use HTTP::Request because I was researching about using the two together( which that didn't seem to help ) and have tried every combination I can think of or find).

Now - what happens with these sites, is that I pass it the query string, with the search results (the page number looped - which has been working fine thus far), as you can see above, but it keeps giving me the log-in page over and over. I've been doing a lot research and reading, and I think this is a cookie problem. I have read the docs on all the modules I'm using, and I really need to find something that works with one of these modules, as I'm having issues with installation right now, another topic, but that is why I haven't tried WWW::Mechanize. I have tried a lot of different combinations, but I keep coming back to these two lines of code I'm trying above with the cookie jars, because based on all I've read, they should work?

Please...a little help....thanks

koolgirl