Thanks to everyone for their comments. A few days ago I wasn't aware of a single portable version of Perl and now I have more options than I know what to do with!

The immediate problem I had was solved with TinyPerl as I needed to do some simple pinging and executable launching. That was very messy in Windows batch script and nice and simple in Perl.

Next I need to figure out which of these variants will get pride of place on the USB key. Given that I have plenty of free space I wonder if I could fit all of them on there... :)