Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re^2: Problem in RAM usage while threading the program

by karlgoethebier (Abbot)
on Dec 20, 2019 at 19:28 UTC ( #11110453=note: print w/replies, xml ) Need Help??


in reply to Re: Problem in RAM usage while threading the program
in thread Problem in RAM usage while threading the program

I assume you know what a decoy is 🤪. Best regards, Karl

«The Crux of the Biscuit is the Apostrophe»

perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

Replies are listed 'Best First'.
Re^3: Problem in RAM usage while threading the program
by marioroy (Parson) on Dec 22, 2019 at 03:49 UTC

    Thanks for the laugh karlgoethebier :) From time to time, I search for MCE and threads on PerlMonks. This looked interesting and thought a great use case for Kyoto Cabinet. See also, Tokyo Dystopia.

    It is mind boggling to me, seeing many processes running near 100% CPU utilization no matter if involving IO. The OP stated running on a machine having 500 GiB of memory. The beauty of it all is that the 100 GiB NR.fasta file is read into file system cache by the operating system (this is true on Linux - the entire file if available memory). Subsequent scans are then read from cache. Workers do not make an extra copy due to reading from a memory-mapped I/O handle. Meaning that one can run with many workers. The limiting factor is the memory subsystem - how fast.

    Well, I enjoyed writing the demonstrations including one with chunking.

    Lots of love, Mario

      Thanks Mario for the cool demonstrations. And for the links. I didn’t know this Kyōto/Tokyo stuff yet. As alway, best regards. Karl

      «The Crux of the Biscuit is the Apostrophe»

      perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11110453]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (4)
As of 2020-10-24 18:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My favourite web site is:












    Results (246 votes). Check out past polls.

    Notices?