Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Re^3: System call doesn't work when there is a large amount of data in a hash

by Anonymous Monk
on Apr 29, 2020 at 15:49 UTC ( [id://11116222]=note: print w/replies, xml ) Need Help??


in reply to Re^2: System call doesn't work when there is a large amount of data in a hash
in thread System call doesn't work when there is a large amount of data in a hash

overcommit is set to 0 I think, I checked it like this:less /proc/sys/vm/overcommit_memory
Huh, I thought it was 2. I bet your desktop also has 0 there, but for some reason it works there. I am not sure what other settings could influence this behaviour. You could set it to 1 if you have root access and it may even help, but at the cost of potentially summoning OOM-Killer later.
So I should look in to vfork() or in the last suggestion you gave?
There is POSIX::RT::Spawn that might use vfork() under the hood. Try it first. Creating your own child spawn helper is harder, but you could copy the code from Bidirectional Communication with Yourself and start from there. Both options are specific to *nix-like systems and should be avoided if $^O eq 'MSWin32' at least.

Replies are listed 'Best First'.
Re^4: System call doesn't work when there is a large amount of data in a hash
by Nicolasd (Acolyte) on Apr 29, 2020 at 16:19 UTC
    Thanks again for your time
    Yes my desktop also has 0 for overcommit_memory.
    I do not have root permissions on the Centos and none of the future users will have it either.
    I will need to run on even larger datasets of human patients, so can't waste any additional memory

    I will have look at POSIX::RT::Spawn

    It is weird that there is no easy solution for this, is this also with python or other languages?
    Because the mother process doesn't need to interact with sister process. I can write the information I need for 'blastn' to a file, then generate a new file that the mother process opens to read the result.<
    So only the timing when the sister process starts is important, there is no need for a direct interaction.

      It is weird that there is no easy solution for this, is this also with python or other languages?
      Yes, any huge process trying to call fork() on this system will have the same problem. Well, I don't know that for sure, but we can check. Try to run this C program on the CentOS system:
      /* * Pass this program the number of gigabytes it should * allocate before forking as the only command line argument * (fractions are accepted). For example: ./a.out 250 * It will print any errors it encounters. */ #include <stdlib.h> #include <stdio.h> #include <unistd.h> int main(int argc, char ** argv) { void *ptr; if (argc != 2) return -1; ptr = malloc(1024 * 1024 * 1024 * atof(argv[1])); if (!ptr) { perror("malloc"); return -2; } if (fork() < 0) perror("fork"); free(ptr); return 0; }

      (To compile and run the program, try gcc program.c -o program && ./program 250 or even make program && ./program 250.)

      Unless the language uses vfork() or posix_spawn() in its system() implementation, the call will fail. Perhaps the administrators of the CentOS machine could shed some light on the problem? They might know what to do better than me if you tell them that your processes have trouble forking when they allocate more than 50% RAM.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11116222]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others musing on the Monastery: (6)
As of 2024-04-19 09:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found