As I look at it again, you're more interested in left complement and right complement than in symmetrical difference, because you're going to need to build a list based on what's only in jibber, a list based on what's only in jabber, and then the intersection (a list based on what's in both jibber and in jabber).

List::Compare is going to get you most of the way there, if you look at the two input lists as:

@{$in_hash{jibber}} # Left list
@{$in_hash{jabber}} # Right list
[download]