Maybe I'm just misreading, but why not store it in a flatter sort of data structure in the first place? It'd probably be much easier to work with. Something like:

my %server = (
  name => "SomeServer",
  pid  => 8328,
  fd   => "Some FD",
  lsof => [ <some array here> ],
);
[download]

You then could just have an array of these server hashes. Not quite what you were asking for, I know, but better to avoid a problem entirely than to solve it, I think.

Note: Edited to fix typo, after being notified by AnomalousMonk (thanks!).