A great counterexample is POSIX:

Q:\>perl -MPOSIX -wle "print for grep {$_ =~ /$$/} `tasklist`"
perl.exe                    5508 Console                 0      3.936 
+K


Q:\>perl -MPOSIX=strftime -wle "print for grep {$_ =~ /$$/} `tasklist`
+"
perl.exe                    4904 Console                 0      3.672 
+K
[download]

POSIX.pm exports all its functions and constants by default, which costs 300K memory, while importing just the few needed functions reduces the amount of memory spent.

tl; dr - just like you said, no general answer here.