What is happening is, once in a while these processes are throwing "Not enough space" error in $! and continuing running. I dont see this error unless I print $!.

So as a rule, you don't check the value of $!? Thats not good.

My question is, why is this happening?

Because you're running out of space (due to some kind of system admin imposed limit maybe {ulimit})? Who knows, maybe even

$! = 12;
die "Uh oh: $!";
__END__
Uh oh: Not enough space at - line 2.
[download]

Sounds like you'd benefit from a code review.