Perl on a cluster

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi Im running a perl script from a node on a linux cluster. This script needed some modules (HTML::Parser, HTML::TableParser) which I downloaded using CPAN. This worked fine when I worked from the main node of the cluster. However it gives an error when I run the program from any other node. I tried configuring CPAN on the node I was working on but it didnt work as it couldnt install Bundle::CPAN. What am I missing out here? Any insights would be really welcome-and if this is too naive a question, do excuse me :-)

Comment on Perl on a cluster

Replies are listed 'Best First'.
Re: Perl on a cluster by zejames (Hermit) on Nov 23, 2004 at 08:52 UTC
Hi, Don't know what kind of linux cluster you're using, but for sure it won't work that easily : each script is running on one node, and expects to be able to reach everything it needs from there. So you have several solutions : Either you install localy each module Or you manage to share (through nfs for example) data you do not want to duplicate. Beware : each node of the cluster have to be exactly the same. I'd advise you the former solution, and it seems to be your choice. So, could be give us more information about why you couldn't install Bundle::CPAN ? -- zejames	[reply]
Re: Perl on a cluster by fglock (Vicar) on Nov 23, 2004 at 11:07 UTC
Not directly related to your question, but this makes life easier: if you have an Apache webserver, you can setup a CPAN cache that all nodes can use: `<IfModule mod_proxy.c> ProxyRequests Off ProxyPass /cpan/cpan.pair.com http://cpan.pair.com ProxyPassReverse /cpan/cpan.pair.com http://cpan.pair.com <Directory proxy:*> Order deny,allow Allow from all </Directory> CacheRoot "/var/cache/apache" CacheSize 1000000 CacheGcInterval 4 CacheMaxExpire 10 CacheLastModifiedFactor 0.1 CacheDefaultExpire 1 </IfModule>` [download] Plase change `cpan.pair.com` to the mirror nearest you.	[reply] [d/l] [select]
Re: Perl on a cluster by hmerrill (Friar) on Nov 23, 2004 at 12:40 UTC
Yeah, you really need to give more information about how your cluster is set up - hard to help you without more details. As zejames already said, you basically have 2 options - install your modules and script(s) on each node of the cluster individually, or install perl itself, the modules, and your script(s) on a partition that is shared (nfs?) among all the cluster members. The kicker about installing all on a shared partition IIRC is that only one cluster member can have the shared partition mounted at one time. Give more details about your setup and maybe we can help more.	[reply]
Re^2: Perl on a cluster by Fletch (Bishop) on Nov 23, 2004 at 13:48 UTC
No, you can have several machines NFS mounting the same shared directory. In fact that makes administering things much easier as you can update on one box and everything will pick up the new version.	[reply]