Maybe I'm just misreading, but why not store it in a flatter sort of data structure in the first place? It'd probably be much easier to work with. Something like:
my %server = (
name => "SomeServer",
pid => 8328,
fd => "Some FD",
lsof => [ <some array here> ],
);
You then could just have an array of these server hashes. Not quite what you were asking for, I know, but better to avoid a problem entirely than to solve it, I think.
Note: Edited to fix typo, after being notified by
AnomalousMonk (thanks!).