Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: Eliminate exact duplicates from array of hashes

by NetWallah (Canon)
on Oct 09, 2019 at 19:22 UTC ( [id://11107259]=note: print w/replies, xml ) Need Help??


in reply to Eliminate exact duplicates from array of hashes

Using core modules only, and assuming a flat hashref (i.e. not nested),
The following should be faster than other methods that do external serializing:
my %unique = map { my @k=sort(keys %$_); join("",@k, @$_{@k}) => $_ } @test_data; print Dumper [values %unique];

                "From there to here, from here to there, funny things are everywhere." -- Dr. Seuss

Replies are listed 'Best First'.
Re^2: Eliminate exact duplicates from array of hashes
by LanX (Saint) on Oct 09, 2019 at 19:32 UTC
    really?
    use strict; use warnings; use Data::Dumper; my @test_data = ( { a=>1, b=>2 }, { ab => 12} ); my %unique = map { my @k=sort(keys %$_); join("",@k, @$_{@k}) => $_ } @test_data; print Dumper [values %unique];

    $VAR1 = [ { 'ab' => 12 } ];

    updates

    • replacing "" in join($; ,@k, @$_{@k}) solves this issue *
    • please note that other solutions kept the order
    *) in most cases

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery FootballPerl is like chess, only without the dice

      OK - more robust version:
      use strict; use warnings; use Digest::MD5; use Data::Dumper; my @test_data = ( { Tag1 => "1", Tag2 => "a" }, { Tag1 => "1", Tag2 => "a" }, { Tag1 => "1", Tag2 => "b" }, { Tag1 => "1", Tag2 => "c" }, { Tag1 => "1", Tag2 => "a" }, { Tag1 => "2", Tag2 => "a" }, { Tag1 => "2", Tag2 => "d" }, { Tag1 => "2", Tag2 => "a" }, { Tag1 => "3"}, { Tag1 => "sun", Tag2 => "a" }, { Tag1 => "sun", Tag2 => "a" }, ); my @test2=( { a=>1, b=>2 }, { ab => 12} ); # for performance, MD5 is fastest. my %unique = map {my @k=sort(keys %$_); my $ctx=Digest::MD5->new() or die "Cannot make MD5 o +bj"; $ctx->add($_ . $;) for @k; $ctx->add("=>"); #separator for values $ctx->add($_ . "+") for @$_{@k}; $ctx->digest() => $_ } @test_data; print Dumper [values %unique]; %unique = map {my @k=sort(keys %$_); my $ctx=Digest::MD5->new() or die "Cannot make MD5 ob +j"; $ctx->add($_ . $;) for @k; $ctx->add("=>"); #separator for values $ctx->add($_ . "+") for @$_{@k}; $ctx->digest() => $_ } @test2; print Dumper [values %unique];

                      "From there to here, from here to there, funny things are everywhere." -- Dr. Seuss

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11107259]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others romping around the Monastery: (9)
As of 2024-03-28 18:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found