I haven't done a lot of read/writing to ports so I don't know if my answer will be the best. But instead of keeping the handle open all the time, could you do a periodic poll?

while(1) {
  open( COM, "/dev/ttyUSB0") or die "Can't read serial port : $!\n";
  while(<COM>) {
    # stuff
  }
  close(COM);
  sleep(1);
}
[download]

However, I'm not sure what about Device::SerialPort is overkill to you. Seems like a pretty simple implementation to me. Here's an example of it being used (which also uses a sleep): http://www.windmeadow.com/node/38