Passing "actions" to server for processing to prevent timeout

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

That's the best title I could come up with at the moment.

I have a slow perl script (slow meaning it loads a few thousand public proxies) and a client was interested in hiring me to change my Perl script to a CGI script for them to use on their server.

The script itself can take up to 3-5 hours to process completely depending on how many proxies there are and how fast they respond. Either way, on the best day it's TOO SLOW to process on the server without it timing out.

My question is, how do Cpanel backups work? The script launches and tells you it'll email you when it's done. And maybe an hour later you get an email but the page you were on was done loading.

Is it possible to make a separate CGI script load off the web host instead of the browser? Would this mean it wouldn't be limited by the timeout time?

Would my first script take the information and exec the second script, then print "Done! Check your email for results!" and in a few hours the other script will finish and email the user?

Any help would be much appreciated.

Comment on Passing "actions" to server for processing to prevent timeout

Replies are listed 'Best First'.
Re: Passing "actions" to server for processing to prevent timeout by friedo (Prior) on Dec 23, 2006 at 19:50 UTC
merlyn wrote a good column about launching long-running processes from CGI here. Another method (which I use in a few places) is to write a daemon that monitors a database table for new jobs to execute. The CGI script places a new job in the table, and the daemon executes it and updates the table with its status. The CGI can then periodically reload itself and display the status. (Or you can do really fancy stuff with AJAX progress bars and the like.)	[reply]
Re: Passing "actions" to server for processing to prevent timeout by mvaline (Friar) on Dec 24, 2006 at 06:43 UTC
The simple approach you outlined is probably sufficient. Unless you need to report back live status to the calling html page, there's nothing wrong with exec'ing a second script that will notify the user by email. The second script doesn't need to have anything to do with CGI. Also, make sure you use exec here instead of system so your CGI won't be waiting for the second script to return.	[reply]