Check $! for the reason.

Forgot to mention: $! said Bad file descriptor.

In the meantime, we had the same effect on a different process, and we found out that the general behaviour was that the process was not able to open a pipe for reading anymore. For example, open(H,"xxx|") failed in the same way. Since the application executes the external commands with qx(...)), and this implicitly requires reading from a pipe, every execution of an external command was affected (while execution of commands using system was not affected). Hence the cause was the sudden inability of opening pipes, and the effect happened twice so far (and on different hosts) within a couple of days (while we had no such problems during the weeks before). I suspected that, for some reason, the process might have too many open handles, but comparing this process with others on the same host (using ProcessExplorer) did not show anything unusual.

-- 
Ronald Fischer <ynnor@mm.st>

On linux, I'd use strace to see what's happening under the hood. I don't know if there's a similar tool for Windows.

I'm not sure whether this is relevant but I had a problem recently when adding some housekeeping code to a script. The script continuously monitored something, writing stuff to a log file. I decided to fork a child process to do housekeeping on old logs and I set $SIG{ CHLD } = q{IGNORE} in the parent before forking as I did not want the parent to interrupt its monitoring while waiting for the child to finish.

Inside the child I was using system to invoke compression on selected log files and deleting others, logging what it was doing to a separate file. I could see from the log that system was successfully invoking bzip2 and the file was being replaced on disk by a compressed version but, strangely, the script was detecting a non-zero exit status and was logging a failure. Further investigation showed that $? contained -1 but that didn't seem to fit with the fact that the child (bzip2) did successfully launch. It turned out that the problem was caused by the $SIG{ CHLD } = q{IGNORE} which caused system to return a -1 exit code. $! was being set and, IIRC, the error was "No child process" or something like but I can't be sure of that. I think the -1 exit code was being returned because, since children were ignored, exit codes were of no use. The solution in my script was to do local $SIG{ CHLD } = q{DEFAULT} in each scope where I was going to do a system call.

None of this explains why you should suddenly get strange behaviour on just one server or why only qx{ ... } is affected. I should explain that I tried substituting qx{ ... } for system( ... ) without success before I got to the root cause of my problem. Although it doesn't seem that your problem is closely connected to the one I had, I thought it had enough parallels to make it worth mentioning here in case it gave you any clues. The platform was RHEL 5.4 with perl 5.8.8 interpreter.

I hope this is helpful.