You might want to look at PDL, it's fast bulky matrix like math for perl. Keeps it's data in C format so you wouldn't have to pack/unpack. It has routines for reading/writing several common file formats.

I don't remember exact syntax but there's a whitespace delimited reader, and a method to get the C pointer.