Yes for me I get attempts to 127.0.0.1 and ::1 as well
Setting tcp_timeout and udp_timeout changed nothing for me. Having a look in _udp_send the timeout used for can_read is made from the 'retrans' and 'retry' settings. Based on the default values (5 and 4) with the DNS timeout of 2.5s doubling, this results in a timeouts of: 2.5,2.5,5,5,10,10,20,20 for both servers (total 150 seconds).
This reduced the issue to 2.5 seconds:
Net::DNS::Resolver->tcp_timeout(1);
Net::DNS::Resolver->udp_timeout(1);
But then I noticed that the local host name servers are in the defaults as well! So this 'fixes' it by setting the default nameservers to nothing:
Net::DNS::Resolver->nameserver4([]);
Net::DNS::Resolver->nameserver6([]);