Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re^2: Small Hash a Gateway to Large Hash?

by lsherwood (Beadle)
on Feb 18, 2014 at 21:35 UTC ( [id://1075388]=note: print w/replies, xml ) Need Help??


in reply to Re: Small Hash a Gateway to Large Hash?
in thread Small Hash a Gateway to Large Hash?

A lab version of the hash has nearly 150,000 keys; a full production version may have three million or so.

Keys are simply short strings of integers. Values also are short strings (50 or so bytes) containing timezone and some geoloc information.

  • Comment on Re^2: Small Hash a Gateway to Large Hash?

Replies are listed 'Best First'.
Re^3: Small Hash a Gateway to Large Hash?
by BrowserUk (Patriarch) on Feb 18, 2014 at 21:55 UTC
    Keys are simply short strings of integers.

    Hm. Does that mean each key is a single short integer, or each key contains several integers?

    Here's why I ask. The following build two hashes with 1 million key/value pairs.

    1. In this one the keys are strings containing 3 x 7 digit integers; and the values are 10 x 7 digits integers delimited by spaces:
      C:\test>p1 Perl> #5.6MB ;; Perl> $h{ "$_ " x 3} = "$_ " x 10 for 1e6 .. 2e6;; Perl> # 299MB ;;

      1 million key/value pairs 300MB

    2. Same thing, but packing the integers to 4-byte binary values:
      C:\test>p1 Perl> # 5.6MB ;; Perl> $h{ pack 'V3', ($_) x 3} = pack 'V10', ($_) x 10 for 1e6 .. 2e6; +; Perl> # 237MB ;;

      Voila! You've stored the same information and saved 1/5th of the space.

    If the numbers can be stored as shorts or bytes, you can save even more.

    I'm also having trouble reconciling your OP mention of a 3GB hash with the description of key/value sizes and total numbers above?

    This creates a 3 million key/value pair hash with 21-byte keys and 80-byte values and the total memory requirement is <1GB:

    Perl> $h{ "$_ " x 3} = "$_ " x 10 for 1e6 .. 4e6;; Perl> # 912MB ;;

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1075388]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having an uproarious good time at the Monastery: (5)
As of 2024-04-25 14:40 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found