Is there a way I can pass an array or a hash across the socket without formatting it to ASCII first then having to reassemble on the other end?

Not that I'm aware of. Passing any sort of data over a socket requires a serialization process.

If you are looking for modules which will do a lot of the work for you, have a look at Storable - it will serialize just about anything.

Update: Also have a look at IPC::Socket::SIPC.