Yeah, that could be one point where its failing. I have noticed that few times as well. But sometime all forks succeed ( I see 7 processes parent + 6 child in top, which validates that all 6 forks were successful) and then error crops up later.

Now let me give you what I have found out after doing as you guys suggested. This time I ran the program 4 times and all 4 times fork failed with ENOMEM. But as I say system staus says there is enough RAM + swap available.
Total Memory-RAM : 8G, 1.6G free
Swap : 10G, 2.5G free when the error occurs.
Now parent process takes about 1.6G memory and all child processes also take up same 1.6G memory. Fork failed after 4 child processes were spawned. So logically there is no reason to get an ENOMEM in fork - right? Your thoughs please.
Here's what I have found : If I keep allocating memory from within a process 

perl -le 'while () {$a.="x"x(1<<26);}'

[download]
, after the allocated memory hits 4G for the process it terminates ( sometimes with malloc failure - Out of memory, sometimes with Segmentation fault)
So my question is is there any process level limitations (I guess my perl is 32 bit) on how much memory can be allocated ? Also can you guys please point out if there are other things I need to check like ulimit/swap space/ram/no of processes per user/no of open file descriptors/etc/etc to track this error? Also please leave pointers on how to check also.
Appereciate your help.

In reply to Re^4: Error: Not enough space by jhazra
in thread Error: Not enough space by jhazra

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.