Filtering one array using another array

teabag has asked for the wisdom of the Perl Monks concerning the following question:

Fellow Monks, lend me your ears...

I'm trying to write an urlgrabber and I thought it would be nice to include a sort of blacklisting system with it. Now the code for the grabbing/webscraping works perfectly. But I'm having problems with the blacklisting system.

I have 2 arrays. One (@sites) contains the grabbed urls, the other one (@blacklist) contains the blacklisted keywords. Now the code I came up with does "kinda" spot the blacklisted urls, but is clumsy, slow, inefficient and has to be filtered.

Could anyone point out an easier (and faster) way?

#!/usr/bin/perl
# example blacklisting

@sites = (
    "http://www.rtfm.com", "http://www.alottatax.com",
    "http://www.kingdom.com/cgi-bin/script.pl"
);

@blacklist = ( "cgi", "blabla", "testme" );

foreach $site (@sites) {
    &blacklist();
}

sub blacklist {
    foreach $blacklist (@blacklist) {
        if ( $site =~ m/$blacklist/gi ) {
            print "$site blacklisted - $blacklist\n";
        }
        else {
            print "$site ok\n";
        }
    }
}
[download]

Teabag
I'm sure there's more than one way, but /me just needs one anyway - Teabag

Comment on Filtering one array using another array Select or Download Code

Replies are listed 'Best First'.
Re: Filtering one array using another array by Abigail-II (Bishop) on Sep 04, 2003 at 13:27 UTC
You mean that a site is blacklisted, if one of the elements of @blacklist matches against the url? In that case, I'd make a single regex out of @blacklist. Something like (untested): `my $re = join"\|" => map {"(?:$_)"} @blacklist; my @blacklisted = grep {/$re/} @sites;` [download] Abigail	[reply] [d/l]
Re: Re: Filtering one array using another array by teabag (Pilgrim) on Sep 04, 2003 at 13:34 UTC
Excellent Abigail-II! That works perfectly for me, thanks! Teabag `Sure there's more than one way, but one just needs one anyway - Teabag`	[reply] [d/l]
Re: Filtering one array using another array by broquaint (Abbot) on Sep 04, 2003 at 13:33 UTC
How's about `use strict; my @sites = ( "http://www.rtfm.com", "http://www.alottatax.com", "http://www.kingdom.com/cgi-bin/script.pl" ); my $blacklist = join '\|', map quotemeta, qw/ cgi blabla testme /; for(@sites) { printf "%s %s\n", $_, $_ =~ /($blacklist)/ ? "blacklisted - $1" : 'ok'; } __output__ http://www.rtfm.com ok http://www.alottatax.com ok http://www.kingdom.com/cgi-bin/script.pl blacklisted - cgi` [download] See. `perlre` and `quotemeta` for more info. Also a module that might be of interest - `Regex::Presuf`. HTH `_________ broquaint`	[reply] [d/l]
Re: Re: Filtering one array using another array by teabag (Pilgrim) on Sep 04, 2003 at 13:40 UTC
Perfect broquaint! I understand the very simple regexes, but get lost when they get complicated. I'll check out that module you mentioned right away. Sounds very handy. Thanks Teabag `Sure there's more than one way, but one just needs one anyway - Teabag`	[reply] [d/l]
Re: Filtering one array using another array by gjb (Vicar) on Sep 04, 2003 at 13:24 UTC
Put the blacklisted sites in a hash rather than a list, that way you can do a fast lookup. ~~A bit cleaner would be to put them in a Set::Scalar, but this is a matter of taste.~~ Since you need approximate matching, you could have a look at Tie::Hash::Approx. If you'd like to get a list of URLs not in the blacklist, you could use a `grep`. Hope this helps, -gjb- Update: Oops, I should thoroughly read a post before replying, I didn't notice that a regex match was required. Thanks Abigail-II.	[reply] [d/l]
Re: Filtering one array using another array by Abigail-II (Bishop) on Sep 04, 2003 at 13:31 UTC
Put the blacklisted sites in a hash rather than a list, that way you can do a fast lookup. Eh, hashes are great for exact matches, but exact matching isn't what the OP is doing. He's matching against regexes, and hashes aren't going to help him there. Abigail	[reply]
Re: Filtering one array using another array by Kimi_1973 (Initiate) on Sep 04, 2003 at 13:40 UTC
Try the following: `#!/usr/bin/perl # example blacklisting @sites = ( "http://www.rtfm.com", "http://www.alottatax.com", "http://www.kingdom.com/cgi-bin/script.pl" ); @blacklist = ( "cgi", "blabla", "testme" ); foreach $site (@sites) { &blacklist(); } sub blacklist { foreach $blacklist (@blacklist) { if ( $site =~ m/$blacklist/gi ) { print "$site blacklisted - $blacklist\n"; return; #Change1 } else { print "$site ok\n"; return; #Change2 } } }` [download]	[reply] [d/l]
Re: Re: Filtering one array using another array by teabag (Pilgrim) on Sep 04, 2003 at 13:53 UTC
Yup, that works in this example. But it seems to be working only for the first word in @blacklist? Teabag `Sure there's more than one way, but one just needs one anyway - Teabag`	[reply] [d/l]
Re: Re: Filtering one array using another array by Kimi_1973 (Initiate) on Sep 04, 2003 at 13:50 UTC
Plese Ingore the Changes1,2 .	[reply]


We don't bite newbies here... much
	PerlMonks