How do I compute the difference of two arrays? How do I compute the intersect ion of two arrays?

Use a hash. Here's code to do both and more. It assumes that each element is unique in a given array:

@union = @intersection = @difference = ();
%count = ();
foreach $element (@array1, @array2) { $count{$element}++ }

foreach $element (keys %count) {
  push @union, $element;
  push @{ $count{$element} > 1 ? \@intersection : \@difference
  }, $element;
}
[download]

Note that this is the *symmetric difference*, that is, all elements in either A or in B but not in both. Think of it as an xor operation.

The example computes the symmetric difference, but most likely you will only be interested in the pages that are new on the web and missing in your local copy, so you will want to modify the check as follows so it only gives you the locally missing items:

use strict;

my (@local) = get_local_ids();
my (@remote) = get_remote_ids();

my %have_local = ();
foreach $element (@local) { $have_local{$element}++ };

foreach $id (@remote) {
  next if $have_local{$id};
  retrieve($id);
  $have_local{$id}++;
}
[download]

Sort the two lists and compare with comm:

$ cat a
one
two
three
four
five
six
seven
$ cat b
two
four
five
six
seven
$ sort a > a.sorted
$ sort b > b.sorted
$ comm -23 a.sorted b.sorted
one
three
$
[download]

Update 2009 sep 2.

See Re^2: Joining two files on common field for a list of other nodes where unix textutils is suggested to merge files.

Use zsh! :)

$ comm -23 <(sort a) <(sort b)
[download]

That works in bash too, and I'd use that too; but it's not as portable so I didn't write it here.

Compare Re: comparing arrays and Re: help with lists.