Do you use the XS version of Win32::SerialPort?

From the doc:

Buffers The size of the Win32 buffers are selectable with buffers. But each read method currently uses a fixed internal buffer of 4096 bytes. This can be changed in the Win32API::CommPort source and read with internal_buffer. The XS version will support dynamic buffer sizing. Large operations are automatically converted to multiple smaller ones by the tied FileHandle methods.

Another idea :
Maybe do you need to purge the buffer sometimes?
$PortObj->lookclear; # empty buffers

See Methods used with Tied FileHandles ,

hth,
PooLpi