From the way you have populated %ref, it looks like a simple hash whose key is $port and value is a (host)name.

In that case, the following code:

 {foreach $port (@{$ref{$key}{'port'}})
[download]

is trying to treat %ref as a HOH. IMHO, the outer loop should look like:

for my $port (keys %ref){
  my $host = $ref{$port};
  # Process the $host/$port combo
  ...
[download]