I've never done serial communication in Perl, but back when I had to deal with console connections, everyone seemed to get stung sooner or later by flow control, which could explain the symptoms you're having -- particularly as you mentioned 'raw' data, and you don't look to be checking for XON / XOFF.

If you're using hardware flow control, nevermind.