best way to balance load to a Perl script

keiusui has asked for the wisdom of the Perl Monks concerning the following question:

Hello, I run a website on which one particular Perl script receives a lot of hits. It is called download.pl, and it enables my clients to download audio content for their radio shows.

During peak hours, download.pl can receive a hundred hits every minute. download.pl then starts running slowly, as does the rest of the web server.

I am wondering what the best way to balance the load to download.pl would be.

One idea I came up with would be to create ten separate download Perl scripts called download1.pl, download2.pl, download3.pl and so forth. Then, download.pl would act as a switch and call one of the ten Perl scripts by using the system command. Would this help?

Or, maybe it's simply time to get a second web server altogether and balance the load between two servers rather than between ten separate Perl scripts?

I would be curious to get your feedback here. Thanks so much for your help.

Comment on best way to balance load to a Perl script

Replies are listed 'Best First'.
Re: best way to balance load to a Perl script by Your Mother (Archbishop) on Jul 25, 2009 at 21:34 UTC
It sounds like you're not in a persistent environment. If that's right, give this a look--FCGI--and check to see if your host is running one of its engines/drivers on your webserver. It would help; maybe a lot. If it's really a problem with too many connections, you should investigate streaming/sending the content to them differently. This is a big problem space in which I'm not adept and the best/easiest solution is likely not going to be Perl-oriented; you might get a good direction/approach from another monk here.	[reply]
Re: best way to balance load to a Perl script by thunders (Priest) on Jul 25, 2009 at 23:59 UTC
You will get a fair bit of mileage by converting download.pl to a mod_perl handler. In that case you will at least lose the startup time of the perl interpreter. Copying/renaming the script is very unlikely to do anything. As far as your server is concerned calling multiple processes called "/usr/bin/perl /path/to/download.pl" is no more expensive than "/usr/bin/perl /path/to/download2.pl" What is perl actually doing in this case? Is it actually serving the audio? Or is it just drawing a page that has links to the audio files? In the former case, it may be that perl is not the best tool for the job. Using multiple load balanced servers will definitely improve performance, but it's critical to identify where the bottleneck is first. Running the script from the command line with the core module Devel::DProf or the CPAN module Devel::NYTProf will help in identifying what's slow.	[reply]
Re: best way to balance load to a Perl script by CountZero (Bishop) on Jul 26, 2009 at 19:14 UTC
download.pl then starts running slowly, as does the rest of the web server. There could be scores of reasons why this happens. Something which seems very likely is that when lots of people start downloading at the same time, you are running out of bandwith and that slows everything down. CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James	[reply]