If you can swing it a better place to put the path to your templates would be in a PerlSetVar directive and have something like:

<VirtualHost ...:80>
## . . .
PerlSetVar TEMPLATE_PATH /www/host.../templates
## . . . 
</VirtualHost>

## then in your handler use . . .
my $template_path = $r->dir_config( 'TEMPLATE_PATH' );
[download]

If you're stuck reading from the environment, you might have a FixupHandler which uses this method to set the environment variable you're using now before the content phase gets run (and maybe a cleanup handler to unset it just to be paranoid . . . :).

Update: And just to clarify this doesn't address possibly having multiple versions of code for different virtual servers. To handle that you'd need to do something similar to what Apache::Registry does by pulling things into separate packages.