That does seem to work, however, I feel the benefit of using phantomJS to speedup testing is getting overshadowed by these pauses.

That is funny :) The reason to use phantomJS is so you get free browser with javascript support ... you know the difficult stuff ... there is nothing you can do if the javascript does network communication or other stuff -- it will take as long as it will take, at least you didn't have to reverse engineer the javascript