Step one is probably to write an algorithm to do what you want. Something like this perhaps:
- Create your database table with columns for 'link', 'depth', 'read'
- read the first page and store the base URL
- for each link in the page, compare its base to the original base URL
- If they match, add to the DB with depth 2 and read 'no'
- For each entry in the table where read eq 'no', read the page, set read to 'yes', compare each link base to the original base URL
- If they match, add to the db with depth 3 and read 'no'
- repeat the last two steps, setting depth to 4 (i.e. a link found at depth 3)
- end
You could end when you don't find any entries in the db with depth <=3 and read eq 'no', that way it's easy to modify if you decide to read deeper.
--------------------------------------------------------------
"If there is such a phenomenon as absolute evil, it consists in treating another human being as a thing."
John Brunner, "The Shockwave Rider".
| [reply] |