This is addressed in perlfaq4 at How do I compute the difference of two arrays. The code there could be adapted to your specific need pretty easily.

Here's the code from perlfaq4:

 my (@union, @intersection, @difference);
    my %count = ();
    foreach my $element (@array1, @array2) { $count{$element}++ }
    foreach my $element (keys %count) {
        push @union, $element;
        push @{ $count{$element} > 1 ? \@intersection : \@difference }
+, $element;
    }
[download]

Here's one way to adapt it.

sub difference {
    my $array_ref1 = shift;
    my $array_ref2 = shift;
    my @difference;

    foreach my $element (@$array_ref1, @$array_ref2) { $count{$element
+}++ };
    foreach my $element (keys %count) {
        push @difference, $element if $count{$element} == 1;
    }
    return \@difference;
}
[download]

(please do test it -- this is untested code.)