Re^4: System call doesn't work when there is a large amount of data in a hash

Thanks again for your time
Yes my desktop also has 0 for overcommit_memory.
I do not have root permissions on the Centos and none of the future users will have it either.
I will need to run on even larger datasets of human patients, so can't waste any additional memory

I will have look at POSIX::RT::Spawn

It is weird that there is no easy solution for this, is this also with python or other languages?
Because the mother process doesn't need to interact with sister process. I can write the information I need for 'blastn' to a file, then generate a new file that the mother process opens to read the result.<
So only the timing when the sister process starts is important, there is no need for a direct interaction.

Comment on Re^4: System call doesn't work when there is a large amount of data in a hash

Replies are listed 'Best First'.
Re^5: System call doesn't work when there is a large amount of data in a hash by Anonymous Monk on Apr 29, 2020 at 16:56 UTC
It is weird that there is no easy solution for this, is this also with python or other languages? Yes, any huge process trying to call `fork()` on this system will have the same problem. Well, I don't know that for sure, but we can check. Try to run this C program on the CentOS system: `/* * Pass this program the number of gigabytes it should * allocate before forking as the only command line argument * (fractions are accepted). For example: ./a.out 250 * It will print any errors it encounters. / #include <stdlib.h> #include <stdio.h> #include <unistd.h> int main(int argc, char * argv) { void ptr; if (argc != 2) return -1; ptr = malloc(1024 1024 * 1024 * atof(argv[1])); if (!ptr) { perror("malloc"); return -2; } if (fork() < 0) perror("fork"); free(ptr); return 0; }` [download] (To compile and run the program, try `gcc program.c -o program && ./program 250` or even `make program && ./program 250`.) Unless the language uses `vfork()` or `posix_spawn()` in its `system()` implementation, the call will fail. Perhaps the administrators of the CentOS machine could shed some light on the problem? They might know what to do better than me if you tell them that your processes have trouble forking when they allocate more than 50% RAM.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^5: System call doesn't work when there is a large amount of data in a hash
by Anonymous Monk on Apr 29, 2020 at 16:56 UTC

It is weird that there is no easy solution for this, is this also with python or other languages?

fork()

/*
 * Pass this program the number of gigabytes it should
 * allocate before forking as the only command line argument
 * (fractions are accepted). For example: ./a.out 250
 * It will print any errors it encounters.
 */
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

int main(int argc, char ** argv) {
    void *ptr;

    if (argc != 2) return -1;
    ptr = malloc(1024 * 1024 * 1024 * atof(argv[1]));
    if (!ptr) {
        perror("malloc");
        return -2;
    }

    if (fork() < 0) perror("fork");

    free(ptr);
    return 0;
}
[download]

(To compile and run the program, try gcc program.c -o program && ./program 250 or even make program && ./program 250.)

vfork()

posix_spawn()

system()

[reply]
[d/l]
[select]