How to overcome a time limit on using a script

duffdoerak has asked for the wisdom of the Perl Monks concerning the following question:

Perl script is used to scrape match competition data from a website. Somehow the scraping is limited to 2 minutes on the external server. After 2 minutes the scraping is stopped, without error message and right in the middle of executing a command line. How can I overcome this problem?

Comment on How to overcome a time limit on using a script

Replies are listed 'Best First'.
Re: How to overcome a time limit on using a script by marto (Cardinal) on Feb 14, 2015 at 12:28 UTC
Have you considered asking the owner of the site in question? Perhaps they have an API you could use. Update: fixed typo.	[reply]
Re^2: How to overcome a time limit on using a script by duffdoerak (Initiate) on Feb 14, 2015 at 12:43 UTC
No, there is no API available. Our scraper is getting the data from different places on the website.	[reply]
Re: How to overcome a time limit on using a script by tangent (Parson) on Feb 14, 2015 at 13:10 UTC
the scraping is limited to 2 minutes on the external server This is probably done for a reason, and the reason is probably to stop people from scraping the site.	[reply]
A reply falls below the community's threshold of quality. You may see it by logging in.
Re: How to overcome a time limit on using a script by Jenda (Abbot) on Feb 15, 2015 at 02:30 UTC
Waitasecond, how do you run the scraping script? Is it perhaps something that's run as a "CGI script" i.e. by visiting a URL and then runs within a web server? Such scripts tend to have an execution time limit set by the web server. Most probably you should run the script in the background and only access the results by the web based script. Jenda Enoch was right! Enjoy the last years of Rome.	[reply]