I can't help you specifically with your problem, but I use Debian on a regular basis and I have much more luck installing the Debian packaged versions of CPAN modules when possible. In this case, you can just:

apt-get install libdevice-serialport-perl

And it will take care of dependencies and whatnot for you. And in general, they work (I have never had a failure from a debian package, but I guess it's possible that you might).