My RasPi died at the beginning of the year, so I can't check directly, but I remember having the same sort of trouble with CPAN. I had to change some parameters in the config.txt file on the boot partition (gpu_mem was one of them), and boot to the command line (so essentially turn off everything that wasn't absolutely necessary). Also had a 2GB swap on the SD card.

I did manage to install a working Catalyst::Devel doing that, although it took a while to build!

So, in short, it can be done, but you're better off just using cpanm.