I'm following up to myself because I think someone has discovered the cause.

Unless you specify which port Oracle is listening on it will try port 1526 first, which would explain a long delay in waiting for a response before trying the more standard port of 1521.

my $dbh = DBI->connect("dbi:Oracle:host=$host;sid=$sid;port=$port");
[download]

Source: http://use.perl.org/~jdavidb/journal/20440