Troubleshoot a path problem when multithreading on cluster

wayne_deng2000 has asked for the wisdom of the Perl Monks concerning the following question:

I tried to run several tasks (written in perl) on a 8-core node (divided into 8 tasks), but encounter an error for finding a perl module. I tested single task on one thread, no such problem occurred. The weird thing is that the module path is already in ENV path when checking with perl -V. The perl file that gives error begins like this:

#!/usr/bin/perl
use strict; 
use warnings; 
use XML::Parser; ...
[download]

Error message is like this:

Can't locate XML/Parser.pm in @INC (@INC contains: ...  /usr/lib64/per
+l5 /usr/share/perl5 .) at ./blast_parser.pl line 4.
BEGIN failed--compilation aborted at ./blast_parser.pl line 4.
[blastall] FATAL ERROR: File write error
[download]

The interesting thing is that XML::Parser.pm is in the path of /usr/lib64/perl5 that is listed in above @INC and in %ENV (I checked). I tried to include this path in perl file (use lib ), but got same error message. Couldn't figure out where is wrong and how to solve. The perl version: v5.10.1 (*) built for x86_64-linux-thread-multi. Linux version: 2.6.32-358.2.1.el6.x86_64 Thanks for any suggestions. Wayne

Comment on Troubleshoot a path problem when multithreading on cluster Select or Download Code

Replies are listed 'Best First'.
Re: Troubleshoot a path problem when multithreading on cluster by Loops (Curate) on Nov 08, 2014 at 21:58 UTC
Hello and welcome to the monastery. "File write error" looks very suspicious if all its doing is trying to find a module. What exactly did you do in your "single task on single thread" test that worked? How was the code different than the code that is failing?	[reply]
Re^2: Troubleshoot a path problem when multithreading on cluster by wayne_deng2000 (Initiate) on Nov 10, 2014 at 02:38 UTC
I guess "File writing error" is due to messy data not parsed properly by parser module. Two runs I tested use almost same perl code (except a bit difference for multi-task requiring a submission script (batch jobs in PBS manager). Forgot to mention that XML::Parser also requires another Expat library (.so) to work properly. Just suspect that PBS cluster manager ignored the path in ENV or something related to dynamic library? tried to set ld_Library_Path in script, so far the error still persists. Any input is appreciated.	[reply]