Your priority values are hashes. So how do you want to compare {'IS-ABC' => '3'} with {'IS-XYZ' => '3'}? Do you compare IS-ABC with IS-XYZ alphabetically, and what if they are equal?

From your desired output here's something that I think will work:

sub sortBySite {
        my ($site_a) = keys %{$tmp{$a}{'priority'}};
        my ($site_b) = keys %{$tmp{$b}{'priority'}};
        return $site_a cmp $site_b;
}
[download]

Also, does your data really need to be stored as a HoH? Would this simpler structure work for you?

  'hostX1-1-l.example.com' => {
    priority => 'IS-ABC',
    priority_level => 3,
    os => 'linux',
    pager => 'man-duty',
  }
[download]