Maybe perl is simply not appropriate on low performance platforms except for one-liners and I have to rewrite the stuff in C.
But I thought earlier you said "Once running, the performance is good"? It sounds like you're just running this script once on boot, like a daemon, or does it get started & stopped more often? If just once on boot, I'm not sure what the problem is, why do you really need to optimize the boot time? I understand the speed might be a bit annoying during development but hey, that's the price for running on that hardware ;-) Maybe you can develop on a desktop machine?
The Rasperry Pi is in general "slow" by modern standards so if you want to use it you'll have to live with a bit of relative slowness no matter what you do with it - try running startx on it to get a feeling for the speed! FWIW, I've written a bit of Perl for the Pi myself, admittedly with less modules than you're using, and those scripts ran just fine.