Seeking Expert opinion on Schedule jobs for Perl scripts

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

I have a new website(ASP.net driven) which need to run around 500 Perl (Mechanize & Parser) scripts. Here what the scripts do is, it go to another site, take some data and arrange in XML format (so called scrapers which uses www::Mechanize and HTML::Treebuilder). Server may be windows server(Windows Server 2008 or above). I got some suggestions like, running these scripts under cron may be a tedious job, due to the large number of scripts. So, I need a script running system(or script controller or script automator or script running mechanism or whatever the techies says) which can be used to automate the complete work(Timely(scheduled) run of scipts, then after completion, taking XML, validating & uploading to DB etc.. etc..) and also it should have some good features like, Error reporting mechanism, etc...

Can any Monks kindly suggest me on this. Thanks in advance.

regards,

Sullivan

Comment on Seeking Expert opinion on Schedule jobs for Perl scripts

Replies are listed 'Best First'.
Re: Seeking Expert opinion on Schedule jobs for Perl scripts by choroba (Cardinal) on Jun 05, 2013 at 07:05 UTC
Crossposted at StackOverflow. It is considered polite to inform about crossposting so that people not attending both sites do not waste their efforts answering a question already answered at the other end of the Internets. لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ	[reply]
Re^2: Seeking Expert opinion on Schedule jobs for Perl scripts by Anonymous Monk on Jun 05, 2013 at 07:18 UTC
I am Sorry Mr. Choroba... First I posted here. I cross-posted in stackoverflow, because I need a solution asap, so I thought posting in two websites might get some more exposure for my question... Regards	[reply]
Re^3: Seeking Expert opinion on Schedule jobs for Perl scripts by dsheroh (Monsignor) on Jun 05, 2013 at 10:13 UTC
Just to clarify, the issue is not that you cross-posted. It is that you didn't state that you were cross-posting, so that people would know to check the other site before answering and see whether you've already gotten an answer there before making the effort to answer here.	[reply]
Re: Seeking Expert opinion on Schedule jobs for Perl scripts by derby (Abbot) on Jun 05, 2013 at 10:07 UTC
Can you expand on why running under cron is tedious? Do you mean entering 500 entries into cron is tedious? I'm not sure about Windows cron but this stuff is pretty simple and straight forward for *nix cron. There are commercial cron-like systems out there (autosys comes to mind) but I wouldn't go there unless your workflow is really complicated (need to kick off job Z if job A or B or C fails, etc). -derby	[reply]
Re: Seeking Expert opinion on Schedule jobs for Perl scripts by hbm (Hermit) on Jun 05, 2013 at 13:14 UTC
With the little detail you've given, it sounds like the 500 are similar enough to be boiler-plated. I'd aim for one cronjob that processes all 500, probably a handful in parallel, probably config-file driven (so you could simply add a new section for the 501st site.)	[reply]