What you're trying to do, essentially, is calculate set union and/or set difference (depending on how you look at it) using arrays.

This is covered in Perl FAQ section 4. There are variations in the Perl Cookbook, and people have posted questions and code snippets here in the monastery. You can also snoop around the Categorized Questions and Answers. One of those should provide answers.