Re: Script Timeout Settings?

Reading your question I wonder why the script you are running, if it's just doing crawling of webpages etc, is a CGI script at all?

Would rewriting it as a standalone script running continously be an option?

Failing that you could cause the script to fork into the backgroud after outputing some information to the calling window.

Steve
---
steve.org.uk

Comment on Re: Script Timeout Settings?

Replies are listed 'Best First'.
Re^2: Script Timeout Settings? by ChrisJ-UK (Novice) on Jul 23, 2004 at 11:16 UTC
Steve, I'm not entirely sure myself why I have a CGI script. It just kind of started that way. I suppose I need to look at setting up a standalone script but remote control is quite new to me. From my limited understanding of this I gather I would need Telnet/SSH. I'll look into this and get back to you. Thanks for replying so quickly. Chris.	[reply]
Re^3: Script Timeout Settings? by tilly (Archbishop) on Jul 23, 2004 at 13:04 UTC
In addition to the suggestion to use LWP, you should pay attention to robots.txt since you're writing a robot. LWP::RobotUA makes this easy to do.	[reply]
Re^3: Script Timeout Settings? by Grygonos (Chaplain) on Jul 23, 2004 at 12:42 UTC
It sounds like you want to use LWP to grab these webpages ( assuming they are viewable via a browser) and run your analysis routines that way. Otherwise you would need to get a copy of the file you want to analyze, or read it off the server somehow, and analyze it that way.Net::FTP comes to mind, but I would say its poor practice to have your webpages setup in an FTP folder (just personal opinion there). Anyhow, sounds like a super fun project and I hope you find your answers. Grygonos	[reply]
Re^4: Script Timeout Settings? by ChrisJ-UK (Novice) on Jul 26, 2004 at 16:50 UTC
Thanks for the mention of LWP. I'm currently using the Lynx browser to grab the page before analysing it. Would I gain any advantage from LWP? I appreciate the need to be aware of robots.txt but the crawler only goes over one site and they have OK'ed the crawling of it. I wasn't aware of a specific module for this though so I'll look into it if I expand this project or begin a new one. Thanks again everyone, this really is a great place. Chris.	[reply]