I had a similar problem with a shared library in a non-standard location, and found that DynaLoader has its own search path to which I could add the location containing my shared library. You might be able to do something like this (which I don't claim is a complete solution to your question but might be a good starting point):

require DynaLoader;

# Add our lib directory to the library search path
push @DynaLoader::dl_library_path, $libdir;
[download]

You should be able to calculate $libdir using your config module as you wanted to.