I have following to advice to you, 100% based on my own practices:

• use LiveCD linuxes: you boot from CD with your USB pen inserted, play w/o installing anything with Perl. When you go, you'll data storied to USB. I use knoppix, but this could be anything else. Some linuxes boot from USB device and are located entirely there.
• Use WinCE with perl on it. Sometimes I use it within 5-min breaks at lunch, or in any unexpected places.

But for usage Perl from your own USB stick - I would advice using ActivePerl, because it has many modules bundled, so you do not need to recompile them. ActivePerl could be perfectly moved from one directory to another easily, without editing any files.