In addition to my previous post on caveats about the smart match operator, it just occurred to me that I left part of your question unanswered. You might consider the List::Compare module, which does probably exactly what you need.
Comment on Re: any replacements for smartmatch on comparing two arrays