The basic closure solutions have already been given, so I will use a bad glob trick. Prior to 5.6 this is OS specific.

sub neighbours {
  my $str = join ":", map {"{".($_-1).",$_,".($_+1)."}"}@_;
  my $not_wanted = join ":", @_;
  grep {$not_wanted ne $_} glob($str);
}

print map "$_\n", neighbours(2,3,4);
[download]