You've had some good advice, particularly doing the join on the database rather than in Perl. I'd like to comment on the following observation:

it seems to crunch about 5000 rows at once then rests for 5 minutes. I'm not sure what it's doing.

This is probably due to buffering; output is accumulating in a buffer at a relatively steady rate until the buffer fills, then you get a whole lot of results as the buffer is flushed. As you're doing the join manually in Perl, this is most likely to be buffering of Perl's output, so I'm guessing you must be writing to a file (Perl 'typically' has line buffering when writing to a terminal and block buffering otherwise).

HTH

Roger