Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re^3: Problem in RAM usage while threading the program

by marioroy (Parson)
on Dec 22, 2019 at 03:49 UTC ( #11110501=note: print w/replies, xml ) Need Help??


in reply to Re^2: Problem in RAM usage while threading the program
in thread Problem in RAM usage while threading the program

Thanks for the laugh karlgoethebier :) From time to time, I search for MCE and threads on PerlMonks. This looked interesting and thought a great use case for Kyoto Cabinet. See also, Tokyo Dystopia.

It is mind boggling to me, seeing many processes running near 100% CPU utilization no matter if involving IO. The OP stated running on a machine having 500 GiB of memory. The beauty of it all is that the 100 GiB NR.fasta file is read into file system cache by the operating system (this is true on Linux - the entire file if available memory). Subsequent scans are then read from cache. Workers do not make an extra copy due to reading from a memory-mapped I/O handle. Meaning that one can run with many workers. The limiting factor is the memory subsystem - how fast.

Well, I enjoyed writing the demonstrations including one with chunking.

Lots of love, Mario

  • Comment on Re^3: Problem in RAM usage while threading the program

Replies are listed 'Best First'.
Re^4: Problem in RAM usage while threading the program
by karlgoethebier (Abbot) on Dec 22, 2019 at 13:07 UTC

    Thanks Mario for the cool demonstrations. And for the links. I didn’t know this Kyōto/Tokyo stuff yet. As alway, best regards. Karl

    «The Crux of the Biscuit is the Apostrophe»

    perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11110501]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (4)
As of 2020-09-22 05:23 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    If at first I don’t succeed, I …










    Results (128 votes). Check out past polls.

    Notices?