But when there is a lot of virtual memory (250 GB) used by storing it in a hash, the system call doesn't work

This may have to do with the operating system you are using (you said it's Centos 7), and the way it is tuned. You see, the only way to run a program on most Unix-like systems is to fork() a copy of the existing process, then exec() inside the child process to replace the currently running image with a different one. (There is another way involving the vfork() system call, which freezes the parent and makes it undefined behaviour to do almost anything in the child process before the exec(), but almost no-one uses it except some implementations of posix_spawn() standard library function.) Yes, copying entire 250G of the address space of a process just to throw it away on the next system call is wasteful, so when fork() happens, Linux kernel makes the child process refer to the same physical memory that the parent uses, only making a copy when one of the processes tries to change the contents ("copy on write").

This optimisation makes it possible to fork() processes occupying more than 50% of the memory, at the same time introducing a way to break the promise of the allocated process memory: now if both parent and child try to use all of their rightfully allocated (or inherited) address space, the system will run out of physical memory and will have to start swapping. Some people disable this behaviour because they prefer some memory allocation requests (including allocating memory for a fork() of a large process) to fail instead of letting them be paged out or killed by OOM-killer. What is the value of overcommit settings on the machine you are running this code on?

There is a kludge you can use to work around this behaviour: at the beginning of the program, fork() a child process that never does anything besides reading command lines to launch over a pipe from the parent and feeding them to system. This way, the parent stays small enough to have a good chance fork() succeeding, even after the grand-parent grows huge.

In reply to Re: System call doesn't work when there is a large amount of data in a hash by aitap
in thread System call doesn't work when there is a large amount of data in a hash by Nicolasd

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.