I didn't think I'd be able to answer this one, but then I saw the site you were trying to crawl.
NCBI has a nice API for retrieving data - you don't have to walk through the pages.

The real solution here is to use some modules specifically designed for getting data from NCBI. These modules use their API properly, instead of fudging through javascript. I suggest Bio::Perl for genomic data (I think it can do pubmed articles too -- ah yes, Bio::Biblio - they even have a sample script for PubMed queries included with the bioperl distribution), or NCBI's own Entrez Programming Utilities.

Personally, I use the Bio::Perl modules on a daily basis for a great deal of the work I do with NCBI data.

As for legal status, the data is freely available. They do specifically ask you use their API rather than spidering the pages. ;-)