It looks like LWP::Protocol::http.pm will let you use a variable called @EXTRA_SOCK_OPTS to do what you want. I think the code would be:

use LWP::Protocol::http;
@LWP::Protocol::http::EXTRA_SOCK_OPTS = (LocalAddr => "$vhost:$port");
[download]

then do the normal LWP stuff. Not tested in any way.