Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re: Fastest way to lookup a point in a set

by kcott (Archbishop)
on Aug 05, 2017 at 07:48 UTC ( [id://1196790]=note: print w/replies, xml ) Need Help??


in reply to Fastest way to lookup a point in a set

G'day eyepopslikeamosquito,

I tried using a matrix: $cells{x_coord}{y_coord}.

For comparison, here's the benchmark using your original code:

Benchmark: timing 200000 iterations of Big, Pak, Str... Big: 10 wallclock secs (10.36 usr + 0.02 sys = 10.38 CPU) @ 19 +267.82/s (n=200000) Pak: 8 wallclock secs ( 9.81 usr + 0.03 sys = 9.84 CPU) @ 20 +325.20/s (n=200000) Str: 9 wallclock secs ( 8.30 usr + 0.02 sys = 8.32 CPU) @ 24 +038.46/s (n=200000)

So, while it looks like your machine is about twice as fast as mine, the results are roughly equivalent.

I then changed the last part of your code to:

sub mat_hash { my %cells; $cells{$_->[0]}{$_->[1]} = undef for @points; my $ncells = 0; $ncells += keys %{$cells{$_}} for keys %cells; $ncells == $npoints or die; exists $cells{$_->[0]}{$_->[1]} or die for @points; exists $cells{'notfound'} and die; exists $cells{'notfound2'} and die; exists $cells{'notfound3'} and die; return \%cells; } sub mat_look { my $cells = shift; exists $cells->{$_->[0]}{$_->[1]} or die for @points; exists $cells->{'notfound'} and die; exists $cells->{'notfound2'} and die; exists $cells->{'notfound3'} and die; } my $str_ref = str_hash(); my $big_ref = big_hash(); my $pak_ref = pak_hash(); my $mat_ref = mat_hash(); timethese 200000, { Str => sub { str_look($str_ref) }, Big => sub { big_look($big_ref) }, Pak => sub { pak_look($pak_ref) }, Mat => sub { mat_look($mat_ref) }, };

I'm pretty sure mat_hash and mat_look capture all the same operations as your *_hash and *_look functions; however, it wouldn't hurt for a second look at that — preferably with a non-popping eye :-)

Other than the shebang line

#!/usr/bin/env perl

the remainder of the code is unchanged.

I ran the benchmark three times:

Benchmark: timing 200000 iterations of Big, Mat, Pak, Str... Big: 10 wallclock secs (10.06 usr + 0.02 sys = 10.08 CPU) @ 19 +841.27/s (n=200000) Mat: 8 wallclock secs ( 7.51 usr + 0.01 sys = 7.52 CPU) @ 26 +595.74/s (n=200000) Pak: 10 wallclock secs ( 9.63 usr + 0.01 sys = 9.64 CPU) @ 20 +746.89/s (n=200000) Str: 8 wallclock secs ( 7.78 usr + 0.02 sys = 7.80 CPU) @ 25 +641.03/s (n=200000) Benchmark: timing 200000 iterations of Big, Mat, Pak, Str... Big: 10 wallclock secs (10.01 usr + 0.02 sys = 10.03 CPU) @ 19 +940.18/s (n=200000) Mat: 8 wallclock secs ( 7.52 usr + 0.01 sys = 7.53 CPU) @ 26 +560.42/s (n=200000) Pak: 10 wallclock secs ( 9.72 usr + 0.01 sys = 9.73 CPU) @ 20 +554.98/s (n=200000) Str: 8 wallclock secs ( 8.00 usr + 0.02 sys = 8.02 CPU) @ 24 +937.66/s (n=200000) Benchmark: timing 200000 iterations of Big, Mat, Pak, Str... Big: 10 wallclock secs (10.05 usr + 0.01 sys = 10.06 CPU) @ 19 +880.72/s (n=200000) Mat: 8 wallclock secs ( 7.47 usr + -0.01 sys = 7.46 CPU) @ 26 +809.65/s (n=200000) Pak: 9 wallclock secs ( 9.66 usr + 0.02 sys = 9.68 CPU) @ 20 +661.16/s (n=200000) Str: 9 wallclock secs ( 8.09 usr + 0.01 sys = 8.10 CPU) @ 24 +691.36/s (n=200000)

So, it looks like Mat is slightly faster than Str (the previous fastest). Your post stressed speed but, in case it matters, the increased number of keys will equate to greater memory usage.

I'm using:

$ perl -v | head -2 | tail -1 This is perl 5, version 26, subversion 0 (v5.26.0) built for darwin-th +read-multi-2level

— Ken

Replies are listed 'Best First'.
Re^2: Fastest way to lookup a point in a set
by roboticus (Chancellor) on Aug 05, 2017 at 16:36 UTC

    Ken:

    I'd've saved myself a little time had I read the entire thread before going off and trying a double-hash. Mine was essentially the same as yours, but I was concerned about auto-vivification making the hash grow without bounds during use. When I tested with my anti-autovivification bit, I noticed a speed boost on the double-hash vs. the others when doing lookups for items that don't exist.

    I modified all the lookup routines accept a check count so I could specify the number of points to check, and built a points2 array with a bunch of points that aren't in the original. So my lookup routine looks like:

    sub dbl_look { my $cells = shift; my $chk = shift; my $cnt = 0; for my $p (@points2[0 .. $chk]) { (exists $cells->{$p->[0]} and exists $cells->{$p->[0]}{$p->[1] +}) and ++$cnt; } exists $cells->{'notfound'} and die; exists $cells->{'notfound2'} and die; exists $cells->{'notfound3'} and die; }

    I then called each of the routines with the original data, the original data plus the same amount of non-existing points, and then the original data plus twice the number of non-existing points:

    timethese 200000, { StjA => sub { stj_look($stj_ref, $npoints }, StjB => sub { stj_look($stj_ref, 2*$npoints }, StjC => sub { stj_look($stj_ref, 3*$npoints }, ... };

    The results were:

    $ perl pm_1196786.pl 131, keys 131 str: 131 131, keys 131 stj: 131 big: 131 pak: 131 dbl: 131 Benchmark: timing 200000 iterations of BigA, BigB, BigC, DblA, DblB, D +blC, PakA, PakB, PakC, StjA, StjB, StjC, StrA, StrB, StrC... BigA: 7 wallclock secs ( 6.51 usr + 0.00 sys = 6.51 CPU) @ 30 +698.39/s (n=200000) BigB: 13 wallclock secs (12.95 usr + 0.00 sys = 12.95 CPU) @ 15 +439.25/s (n=200000) BigC: 19 wallclock secs (18.97 usr + 0.00 sys = 18.97 CPU) @ 10 +543.52/s (n=200000) DblA: 6 wallclock secs ( 6.50 usr + 0.00 sys = 6.50 CPU) @ 30 +773.97/s (n=200000) DblB: 10 wallclock secs ( 9.23 usr + 0.00 sys = 9.23 CPU) @ 21 +659.09/s (n=200000) DblC: 12 wallclock secs (11.97 usr + 0.00 sys = 11.97 CPU) @ 16 +709.83/s (n=200000) PakA: 6 wallclock secs ( 6.31 usr + 0.00 sys = 6.31 CPU) @ 31 +685.68/s (n=200000) PakB: 13 wallclock secs (12.45 usr + 0.00 sys = 12.45 CPU) @ 16 +060.39/s (n=200000) PakC: 18 wallclock secs (18.34 usr + 0.00 sys = 18.34 CPU) @ 10 +902.75/s (n=200000) StjA: 7 wallclock secs ( 6.91 usr + 0.00 sys = 6.91 CPU) @ 28 +960.32/s (n=200000) StjB: 14 wallclock secs (13.56 usr + 0.00 sys = 13.56 CPU) @ 14 +746.00/s (n=200000) StjC: 20 wallclock secs (19.94 usr + 0.00 sys = 19.94 CPU) @ 10 +031.10/s (n=200000) StrA: 6 wallclock secs ( 6.36 usr + 0.00 sys = 6.36 CPU) @ 31 +446.54/s (n=200000) StrB: 13 wallclock secs (12.67 usr + 0.00 sys = 12.67 CPU) @ 15 +782.83/s (n=200000) StrC: 19 wallclock secs (18.69 usr + 0.00 sys = 18.69 CPU) @ 10 +702.63/s (n=200000)

    Dbl fares pretty well against the others when a significant amount of lookups fail.

    Full source...

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

      That's a good point. Depending on usage and distribution, short-circuiting on "exists $cells[$x]" could also improve speed. In fact, knowing more about the distribution, organising the matrix as "$cells[$y][$x]", instead of "$cells[$x][$y]", could be another improvement.

      — Ken

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1196790]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others taking refuge in the Monastery: (3)
As of 2024-04-20 01:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found