removing duplicates entries from an array

JFarr has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks, I've been stumped by a little problem. I have a subroutine RDup ( remove duplicates )

sub RDup
{
    my @arrays = sort @_;
    my @temp = ();
    my %Seen = ();
    
    foreach my $elem(@arrays)
    {
        next if $Seen{$elem}++;
        $elem = join(" ", split " ", $elem);
        push @temp, $elem;
    }
    
    return @temp;
}
[download]

This sub takes in an array, obvisously. If I creat an array @t1 = ("z", "a", "r", "a", "a"); It sorts and removes the duplicate entries file. But I have some code that is searching for dll names and if found, pushes it on an array.

foreach $d(@dllExeLines)
{        
    if( $d =~ m/$serviceName/ )
    {
    foreach $f( split /,/, $d )
    {
       if( $f =~ /$dll/igcgxm )
       {
          push @unique, $dll;                   }
    }
    }
}

RDup(@unique);
[download]

When I pass in @unique, the duplicates are not removed. What am I doing wrong, and not understanding here in Perl. There must be a difference in what is placed on an array. I've tried everyting I can think of, but nothing works. Thanks.

Comment on removing duplicates entries from an array Select or Download Code

Replies are listed 'Best First'.
Re: removing duplicates entries from an array by VSarkiss (Monsignor) on Jan 06, 2006 at 17:05 UTC
The immediate problem is that your `RDup` routine is returning a new array that you're not saving. IOW do something like this: `@unique = RDup(@unique);` [download] There are a lot of ways you can improve removing duplicates, though, and there are several suggestions in the Perl FAQ list. Try `perldoc -q duplicate` on your machine for some suggestions. Do not rebuke them with harsh words ... but rather lead them gently - with URLs - so that they may learn wisdom.	[reply] [d/l] [select]
Re: removing duplicates entries from an array by ikegami (Patriarch) on Jan 06, 2006 at 17:09 UTC
What's with all the options on that regexp? `igcgxm`?? And assuming $serviceName and $dll are strings, not regexp, you don't even quote your variables? Your sorting before removing the duplicates, when it would be cheaper to sort after they have been removed (since you algorithm don't take into account the fact that the array is sorted). `join(" ", split " ", $elem)` is very weird too. When you want unique, use a hash! `sub RDup { my %unique; $unique{lc($_)} = $_ foreach @_; return sort values %unique; } my @unique; foreach $d (@dllExeLines) { if ($d =~ /\Q$serviceName/) { foreach $f (split(/,/, $d)) { if ($f =~ /\Q$dll/i) { push @unique, $dll; } } } } @unique = RDup(@unique);` [download]	[reply] [d/l] [select]
Re^2: removing duplicates entries from an array by kulls (Hermit) on Jan 06, 2006 at 17:18 UTC
We can directly replace `push @unique, $dll;` with `$unique{$dll}=1` . why not ? - kulls	[reply] [d/l] [select]
Re^3: removing duplicates entries from an array by ikegami (Patriarch) on Jan 06, 2006 at 17:27 UTC
Yes, `RDup` could be eliminated. I didn't remove it for the sake of reusability. `my %unique; foreach $d (@dllExeLines) { if ($d =~ /\Q$serviceName/) { foreach $f (split(/,/, $d)) { if ($f =~ /\Q$dll/i) { $unique{lc($dll)} = $dll; } } } } my @unique = sort values %unique;` [download]	[reply] [d/l] [select]
Re: removing duplicates entries from an array by l3v3l (Monk) on Jan 06, 2006 at 17:20 UTC
... and there is also the ever popular : `%seen = (); @unique = sort(grep { ! $seen{$_} ++ } @list);` [download]	[reply] [d/l]
Re: removing duplicates entries from an array by Perl Mouse (Chaplain) on Jan 06, 2006 at 17:07 UTC
The uniques are being returned from RDup. But you ignore its return value. `Perl --((8:>*`	[reply]
Re: removing duplicates entries from an array by ambrus (Abbot) on Jan 07, 2006 at 12:57 UTC
Is it possible that you are throwing away the result of the RDup call? In specific, shoulnd't `RDup(@unique);` [download] be `@unique = RDup(@unique);` [download] instead?	[reply] [d/l] [select]
Re: removing duplicates entries from an array by smokemachine (Hermit) on Jan 06, 2006 at 18:40 UTC
could be this? `perl -e '@a=qw/z a r a z a a x/; $regex="\|";foreach(@a){$regex.=$_."\| +" unless /^"$regex"$/ && $regex}@a=split /\\|/, $regex; print @a'` [download]	[reply] [d/l]

Back to Seekers of Perl Wisdom