Re: WWW::Robot memory management

I don't have experience with this module, but I'm curious as to why you need to look at the entire site in one shot. Is there any way you could break it up and search separate sections of the site, possibly combining the results after all of the searches have completed?

Alex / talexb / Toronto

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

Comment on Re: WWW::Robot memory management

Replies are listed 'Best First'.
Re^2: WWW::Robot memory management by imagestrips (Initiate) on Jan 17, 2006 at 18:28 UTC
Indeed, I have tried that approach too by using the save-state and restore-state hooks but it failed because - a)I think that I might have misunderstood what constitutes the state of the machine (list of urls visited and urls to be visited?) - b) the methods available do not provide a comprehensive interface to the robots state and c) I am dealing with a Domino R5 site that does not exactly lend itself to rational partitioning - it has very unfriendly and seemingly unstructured URLs (therefore not allowing me to devise an adequate segmentation method)	[reply]

Replies are listed 'Best First'.

Re^2: WWW::Robot memory management
by imagestrips (Initiate) on Jan 17, 2006 at 18:28 UTC

Indeed, I have tried that approach too by using the save-state and restore-state hooks but it failed because - a)I think that I might have misunderstood what constitutes the state of the machine (list of urls visited and urls to be visited?) - b) the methods available do not provide a comprehensive interface to the robots state and c) I am dealing with a Domino R5 site that does not exactly lend itself to rational partitioning - it has very unfriendly and seemingly unstructured URLs (therefore not allowing me to devise an adequate segmentation method)

[reply]