I could indeed use an absolute path in the use lib.... but that would make it less portable... it would only work for that location

If you want your package to be relocatable, use FindBin. Untaint $FindBin::Bin by applying a capturing match in a BEGIN block, and use the result as argument to use lib; - e.g.

#!/usr/bin/perl -T
# file taint.pl
use FindBin;
BEGIN { $FindBin::Bin =~ /^([\w\/\.]+)$/ and $FindBin::Bin = $1 }
use lib "$FindBin::Bin/tmp";
use Qwargl qw(blorgh);
blorgh();
[download]

# file tmp/Qwargl.pm
package Qwargl;
use 5.10.0;
require Exporter;
@ISA = qw(Exporter);
our @EXPORT_OK = qw(blorgh);
sub blorgh {
    say "strzdiwuddz!";
}
"yecch";
[download]

Adapt the pattern in the BEGIN block to fit your needs.