But when there is a lot of virtual memory (250 GB) used by storing it in a hash, the system call doesn't work

This may have to do with the operating system you are using (you said it's Centos 7), and the way it is tuned. You see, the only way to run a program on most Unix-like systems is to fork() a copy of the existing process, then exec() inside the child process to replace the currently running image with a different one. (There is another way involving the vfork() system call, which freezes the parent and makes it undefined behaviour to do almost anything in the child process before the exec(), but almost no-one uses it except some implementations of posix_spawn() standard library function.) Yes, copying entire 250G of the address space of a process just to throw it away on the next system call is wasteful, so when fork() happens, Linux kernel makes the child process refer to the same physical memory that the parent uses, only making a copy when one of the processes tries to change the contents ("copy on write").

This optimisation makes it possible to fork() processes occupying more than 50% of the memory, at the same time introducing a way to break the promise of the allocated process memory: now if both parent and child try to use all of their rightfully allocated (or inherited) address space, the system will run out of physical memory and will have to start swapping. Some people disable this behaviour because they prefer some memory allocation requests (including allocating memory for a fork() of a large process) to fail instead of letting them be paged out or killed by OOM-killer. What is the value of overcommit settings on the machine you are running this code on?

There is a kludge you can use to work around this behaviour: at the beginning of the program, fork() a child process that never does anything besides reading command lines to launch over a pipe from the parent and feeding them to system. This way, the parent stays small enough to have a good chance fork() succeeding, even after the grand-parent grows huge.