if ($!)
{
    print "Error occured: $!";
        # reset the error so that we can catch next genuine error cond
+ition
        $! = 0;
}
[download]

Edited by davido to add code and readmore tags.

I think the point has already been made. You aren't supposed to look in $! unless an error has just occured. For instance

print $fh "foo\n" 
  or die "Error writing: $!";
[download]

Since we dont know where your $! testing code occurs, theres any number of possibilities as to how $! can be set. For all we know some part of your code catches an error at some point and leaves $! set. Consider the following program:

perl -le "print BLAH 'foo' or warn $!; print 'foo'; print $!";
[download]

The error message persists long after its relevent.

As and aside: please use <code> tags around any code you post. At the very least it will avoid errors like the one at the bottom of your post. ;-)

HTH

---
demerphq

thanks demerphq.
With your point in mind, its very clear that some part of my code is throwing "Not enough space" error message. Now how is that possible as there is enough memory availbel. No matter which part of the program sets it, its MY program only which is setting it. So even if I check for $! value at the end of the program, and it has an error value that signifies that the program generated that error while running ( though at that point the error may be invalid or in other words the error may have been generated at the line 1 itself and I am catching it at end of the program). So the question why/how this error "Not enough space" is possible in a system having so much memory and resources. For example:

@arr[some very high no which causes a memory exception] = 88;
# at this point $! is set to "Not enough memory"
#I do some work here
print "Hello world";
print "Hello world";
print "Hello world";
# here I check $! and $! is "Not enough memory" which means that the p
+rogram threw "Not enough memory" at some point in time during its exe
+cution before this point. And it just happens that no other system ca
+ll has generated any other error from the time "Not enough memory" wa
+s thrown. If the pevious array allocation didnt throw any error then 
+$! here will be still unset or 0( thats what I expect, please correct
+ me if I am wrong).
if ($!)
{
# error
 exit(0);
}
print "$arr[0]";
[download]


Or is it that the value of $! can be "Not enough space" even without my program causing a memory exception? Meaning $! can be a dangling one? appreciate your thoughts. Jayanta

...
my $x = db_login( ... ) || die( "Could not open ..." );
my $xx = db_login( ... ) || die( "Could not open ..." );
my $xxx = db_login( ... ) || die( "Could not open ..." );
my $xxxx = db_login( ... ) || die( "Could not open ..." );
my %j = <data from DB>
my %jj = <data from DB>;
my %jjjj = <data from DB>;
my $pid1 = &executeChild( ....... );
my $pid2 = &executeChild( ....... );
my $pid3 = &executeChild( ....... );
my $pid4 = &executeChild( ....... );
my $pid5 = &executeChild( ....... );
my $pid6 = &executeChild( ....... );
...
[download]

Definitely time to refactor. Or even better yet, start from scratch if time and budget allow for it.

Check return codes from everything that can fail. In particular, see of fork is returning undef. I bet that's your problem, though it's just my intuition.

Yeah, that could be one point where its failing. I have noticed that few times as well. But sometime all forks succeed ( I see 7 processes parent + 6 child in top, which validates that all 6 forks were successful) and then error crops up later.

Now let me give you what I have found out after doing as you guys suggested. This time I ran the program 4 times and all 4 times fork failed with ENOMEM. But as I say system staus says there is enough RAM + swap available.
Total Memory-RAM : 8G, 1.6G free
Swap : 10G, 2.5G free when the error occurs.
Now parent process takes about 1.6G memory and all child processes also take up same 1.6G memory. Fork failed after 4 child processes were spawned. So logically there is no reason to get an ENOMEM in fork - right? Your thoughs please.
Here's what I have found : If I keep allocating memory from within a process

perl -le 'while () {$a.="x"x(1<<26);}'
[download]

, after the allocated memory hits 4G for the process it terminates ( sometimes with malloc failure - Out of memory, sometimes with Segmentation fault)
So my question is is there any process level limitations (I guess my perl is 32 bit) on how much memory can be allocated ? Also can you guys please point out if there are other things I need to check like ulimit/swap space/ram/no of processes per user/no of open file descriptors/etc/etc to track this error? Also please leave pointers on how to check also.
Appereciate your help.