I think the point has already been made. You aren't supposed to look in $! unless an error has just occured. For instance

print $fh "foo\n" 
  or die "Error writing: $!";
[download]

Since we dont know where your $! testing code occurs, theres any number of possibilities as to how $! can be set. For all we know some part of your code catches an error at some point and leaves $! set. Consider the following program:

perl -le "print BLAH 'foo' or warn $!; print 'foo'; print $!";
[download]

The error message persists long after its relevent.

As and aside: please use <code> tags around any code you post. At the very least it will avoid errors like the one at the bottom of your post. ;-)

HTH

thanks demerphq.
With your point in mind, its very clear that some part of my code is throwing "Not enough space" error message. Now how is that possible as there is enough memory availbel. No matter which part of the program sets it, its MY program only which is setting it. So even if I check for $! value at the end of the program, and it has an error value that signifies that the program generated that error while running ( though at that point the error may be invalid or in other words the error may have been generated at the line 1 itself and I am catching it at end of the program). So the question why/how this error "Not enough space" is possible in a system having so much memory and resources. For example:

@arr[some very high no which causes a memory exception] = 88;
# at this point $! is set to "Not enough memory"
#I do some work here
print "Hello world";
print "Hello world";
print "Hello world";
# here I check $! and $! is "Not enough memory" which means that the p
+rogram threw "Not enough memory" at some point in time during its exe
+cution before this point. And it just happens that no other system ca
+ll has generated any other error from the time "Not enough memory" wa
+s thrown. If the pevious array allocation didnt throw any error then 
+$! here will be still unset or 0( thats what I expect, please correct
+ me if I am wrong).
if ($!)
{
# error
 exit(0);
}
print "$arr[0]";
[download]


Or is it that the value of $! can be "Not enough space" even without my program causing a memory exception? Meaning $! can be a dangling one? appreciate your thoughts. Jayanta

You can't do a bunch of things and then check $! and have it be meaningful at all. Until you know exactly which statement caused $! to be set there is no point in looking at it at all. DBI, or any one of the modules you have in use may be setting $! inadverdantly and its value by the time you check it is actually meaningless. Put a or die "Error:$!"; everywhere it makes sense and see what happens, but drop the catchall test on $! at the end of your code, its worse than useless.

---
demerphq

its very clear that some part of my code is throwing "Not enough space" error message.

Strictly speaking, you can't be sure, since errno is only defined to have a meaning immediately after an error. Not sometime later, or before an error occured.

You might be right that your program does indeed cause an ENOMEM error, but since you don't check $! at the right place, you're never going to be sure if and why it does.

You can fix this easily - system calls have a return code. Use it.

"What should it profit a man, if he should win a flame war, yet lose his cool?"

Or is it that the value of $! can be "Not enough space" even without my program causing a memory exception?

Yes. Check to see if the print right before the # here I check $! returned true or false. If it returned false, that print got an error ("Not enough space."). If it returned true, $! is meaningless. Which is to say an error may or may not have occured earlier, but you have no way of knowing. This is stated in the documentation for $! in perlvar.

...
my $x = db_login( ... ) || die( "Could not open ..." );
my $xx = db_login( ... ) || die( "Could not open ..." );
my $xxx = db_login( ... ) || die( "Could not open ..." );
my $xxxx = db_login( ... ) || die( "Could not open ..." );
my %j = <data from DB>
my %jj = <data from DB>;
my %jjjj = <data from DB>;
my $pid1 = &executeChild( ....... );
my $pid2 = &executeChild( ....... );
my $pid3 = &executeChild( ....... );
my $pid4 = &executeChild( ....... );
my $pid5 = &executeChild( ....... );
my $pid6 = &executeChild( ....... );
...
[download]

Definitely time to refactor. Or even better yet, start from scratch if time and budget allow for it.

Yeah, that could be one point where its failing. I have noticed that few times as well. But sometime all forks succeed ( I see 7 processes parent + 6 child in top, which validates that all 6 forks were successful) and then error crops up later.

Now let me give you what I have found out after doing as you guys suggested. This time I ran the program 4 times and all 4 times fork failed with ENOMEM. But as I say system staus says there is enough RAM + swap available.
Total Memory-RAM : 8G, 1.6G free
Swap : 10G, 2.5G free when the error occurs.
Now parent process takes about 1.6G memory and all child processes also take up same 1.6G memory. Fork failed after 4 child processes were spawned. So logically there is no reason to get an ENOMEM in fork - right? Your thoughs please.
Here's what I have found : If I keep allocating memory from within a process

perl -le 'while () {$a.="x"x(1<<26);}'
[download]

, after the allocated memory hits 4G for the process it terminates ( sometimes with malloc failure - Out of memory, sometimes with Segmentation fault)
So my question is is there any process level limitations (I guess my perl is 32 bit) on how much memory can be allocated ? Also can you guys please point out if there are other things I need to check like ulimit/swap space/ram/no of processes per user/no of open file descriptors/etc/etc to track this error? Also please leave pointers on how to check also.
Appereciate your help.

Heh :-) I think I found your problem:

see this link on 4Gb limits /w perl5.6.1/Solaris:

If you do want to be able to allocate more than 4GB memory inside perl, then you should use the Solaris malloc, since the perl malloc breaks when dealing with more than 2GB of memory. You can do this with

sh Configure -Uusemymalloc

Note that this will break binary compatibility with any version that was not compiled with -Uusemymalloc.

You can test if your perl interpreter was compiled with usemymalloc this flag by doing

> perl -V|grep malloc
[download]

It should show "usemymalloc=n" if your perl supports more than 4 Gb. If it doesn't, you need to recompile perl or get another binary. I suggest using the most recent stable perl for this (5.8.6 at the moment).

I'm wondering if you can't get the algorithm to work without taking up more than 4Gb, though. :-)

"What should it profit a man, if he should win a flame war, yet lose his cool?"