Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re^2: Hash Search is VERY slow

by Tux (Canon)
on Sep 30, 2021 at 07:59 UTC ( [id://11137132]=note: print w/replies, xml ) Need Help??


in reply to Re: Hash Search is VERY slow
in thread Hash Search is VERY slow

If the number of keys is the cause of the memory cunsumption, make the keys smaller and more efficient. It might matter if you are low on memory. Best reason to pack the IP's is the it is much easier to sort

use List::Util qw( shuffle ); use Devel::Size qw( total_size ); my ($x, %ip0, %ip1, %ip2, %ip3, %ip4, %ip5) = 31; foreach my $i0 ((shuffle 1..254)[0..$x]) { foreach my $i1 ((shuffle 1..254)[0..$x]) { foreach my $i2 ((shuffle 1..254)[0..$x]) { foreach my $i3 ((shuffle 1..254)[0..$x]) { $ip0{"$i0.$i1.$i2.$i3"}++; $ip1{"$i0.$i1"}{"$i2.$i3"}++; $ip2{$i0}{$i1}{$i2}{$i3}++; $ip3{pack "CCCC", $i0, $i1, $i2, $i3}++; $ip4{pack "CC", $i0, $i1}{pack "CC", $i2, $i3}++; $ip5{pack "C", $i0}{pack "C", $i1}{pack "C", $i2}{pack "C", $i3}++ +; }}}} printf "{192.168.101.208} : %9d\n", total_size (\%ip0); printf "{192.168}{101.208} : %9d\n", total_size (\%ip1); printf "{192}{168}{101}{208} : %9d\n", total_size (\%ip2); printf "{xC0xA8x65xD0} : %9d\n", total_size (\%ip3); printf "{xC0xA8}{x65xD0} : %9d\n", total_size (\%ip4); printf "{xC0}{xA8}{x65}{xD0} : %9d\n", total_size (\%ip5); ==> {192.168.101.208} : 116483356 {192.168}{101.208} : 69879492 {192}{168}{101}{208} : 71176834 {xC0xA8x65xD0} : 106954864 {xC0xA8}{x65xD0} : 69611776 {xC0}{xA8}{x65}{xD0} : 71176690

On my machine using perl-5.28.0, $ip4{pack "CC", $i0, $i1}{pack "CC", $i2, $i3} proves to be the most memory-friendly approach


Enjoy, Have FUN! H.Merijn

Replies are listed 'Best First'.
Re^3: Hash Search is VERY slow
by LanX (Saint) on Sep 30, 2021 at 11:19 UTC

    > $ip4{pack "CC", $i0, $i1}{pack "CC", $i2, $i3} proves to be the most memory-friendly approach

    This might be true for long keys, but the original strings of the keys are only stored once.

    {192.168}{101.208} : 69879492 ... {xC0xA8}{x65xD0} : 69611776

    that's a gain of ~0.3% °

    I'd call this micro optimization, or did I miss something?

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

    °) 0.383% to be more precise, so rather ~0.4%

      Your suggestion of using {"192.168"}{"101.208"} is indeed almost as small as my pack version, so it shows close to no gain in memory. Both are however 40% smaller than the original {"192.168.101.208"}.

      I just included all of those to show the differences in size.

      I first thought I'd use Socket::inet_aton, but that of course does not support the {A.B}{C.D} split.

      The win in using pack over plain is not the .3% gain in memory size, but the ease of sorting.


      Enjoy, Have FUN! H.Merijn
        > The win in using pack over plain is not the .3% gain in memory size, but the ease of sorting.

        I'm confused ... when do I need to sort hash keys?

        And why is it "easier" to sort the packed version of "023.156" ? Do you mean faster?

        Cheers Rolf
        (addicted to the Perl Programming Language :)
        Wikisyntax for the Monastery

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11137132]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others admiring the Monastery: (5)
As of 2024-03-28 19:34 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found