The fact that your proc jumps up that much is a decent sign. It more than likely means your process is not contending with any other processes for time on the proc. What I would be interested in seeing is the output from a vmstat 5 10. One prior to initating the process and another while the processes are running.

Without really looking into it too much, I want to point a finger at the I/O subsystem. I recently had similar problems with system utilities on a particular host. Come to find out the RAID controller was flaky. I couldn't see much of an issue, but it was the only thing I hadnt tested. So the disks got moved off of the hardware based controller, and ported over to software RAID. Viola the issue went away.

Its not much without a more fine tuned analysis of your particular system, but its another place to look. Seeing as how the same code runs flawlessly on a different host, it is on initial assumption more than likely hardware based. Another alternative route looking at a software level would be to use the 'strace' program to watch what is actually happening during runtime on that host. Its not the easiest tool to use at a 'junior' level, but will give great insights into whats going on.

I am planning on looking at the raid controller here soon. I have to bring the box down to do this so I will have to wait till early tomorrow morning. ugh... or maybe even this weekend... double ugh... *sigh*

wendel@develop ~
$ vmstat 5 10
   procs                      memory      swap          io     system 
+     cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs 
+us sy id
 0  0  0 201948  51692 202216 452744    0    0     2     0    2     1 
+ 1  1  1
 0  0  0 201948  51692 202228 452744    0    0     0    10  149   196 
+ 1  0 98
 0  0  0 201948  51692 202228 452744    0    0     0     0  132   182 
+ 1  0 99
 0  0  0 201948  51692 202228 452744    0    0     0     0  110   136 
+ 0  1 98
 0  0  0 201948  51692 202232 452744    0    0     0    29  108   113 
+ 1  2 98
 0  0  0 201948  51692 202232 452744    0    0     0    42  111   112 
+ 0  0 99
 0  0  0 201948  51692 202232 452744    0    0     0     0  109   135 
+ 0  0 100
 0  0  0 201948  51692 202232 452744    0    0     0     0  107   130 
+ 1  0 99
 0  0  0 201948  51692 202232 452744    0    0     0     0  110   136 
+ 1  1 99
 0  0  0 201948  51692 202232 452744    0    0     0     0  116   156 
+ 1  0 99
wendel@develop ~
$ vmstat 5 10
   procs                      memory      swap          io     system 
+     cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs 
+us sy id
 1  0  0 201936  92416 202284 452744    0    0     2     0    2     1 
+ 1  1  1
 1  0  0 201936  83428 202296 452744    0    0     0    41  107   128 
+19  1 80
 0  0  0 201936  73672 202304 452744    0    0     0    50  106   121 
+22  1 77
 1  0  0 201936  78388 202312 452744    1    0     1    38  105   121 
+17  2 81
 1  0  0 201936  81252 202324 452744    0    0     0    42  109   133 
+19  1 80
 1  0  0 201936  84052 202328 452748    2    0     3    42  158   144 
+22  2 76
 1  0  0 201936  74168 202336 452748    1    0     1    73  141   161 
+21  3 76
 0  0  0 201936  63516 202348 452748    2    0     2    80  117   155 
+24  1 75
 0  0  0 201936  63516 202356 452748    0    0     0    42  111   138 
+ 0  1 99
 0  0  0 201936  63516 202356 452748    0    0     0     0  130   185 
+ 0  0 99
[download]

It seems as if the only big difference would be the number of blocks read in from disk. hrm... Raid controller... How did you move away from using the Raid controller? Did you get a different box? Disable it?

Yeah. Everything looks good there, though I don't see the behaviour you noted with the proc going up to 100% utilized.

Take a peek at the controller and see what happens. Im gonna put my money on it right now. (hopefully I dont lose ;) )

use perl;

The one CPU being at 100% is likely your problem. Here's why:

You have a quad cpu system, and a threaded version of perl, but based on your original post, your script isn't making use of the threads. That means that it is going to run serially, thus only making use of one processor. Now, in theory, each 'exec'ed script would fire off on a different processor, but depending on how its being called it might not be the case.

Each user will get partial use of a different CPU. Meaning if 1 user connects he uses CPU1, another connects and she gets use of CPU2. That helps quite a bit. However, User1 can't use cpu1 and cpu2 at the same time. That hurts.

thanks again... yah... I am wondering if the development team forgot to take some real-world systems related issues into account when designing their code...

The issue that stings the most here is that the server we are executing this code on is sitting idle. It's a development server that we use to test various things on and is not being hit by a large number of users.

My boss tends to think it is some sort of disk IO problem. I am leaning towards agreeing with him but think I had better cover all my bases.

Wish me luck, I will need it...