You might be having some difficulty involving the way you have it as a Package....but...in general, you share the fileno of the socket's filehande. See: Simple threaded chat server for a usage example.
I'm sure there is a backtracking trail you could go thru, via some socket utility, like socklist, but I think your best bet is to just store the info in a hash, like