The problem is that it only happens about 10 minutes into the run.

The main loop of the program is something like this:

while (not_done_yet) {
  if (queue not full) {
    add new element to queue
  }

  @new_queue = ();
  foreach $element (@queue) {
    if (state==1) { blah blah; state=2 }
    ...
    if (timed_out) { next; }
    push $element, @new_queue;
  }
  @queue = @new_queue;
}
[download]

The issue does not show up if I decrease the queue size from the normal 50 to 5, or the timeout from the default 600 seconds to 30. It shows up after the queue has completely filled up (takes a few minutes), and the first few elements have timed out.

I'll see what I can do to decrease it, but I'm not hopeful.