The while at the top exits immediately because you opened the file for appending (>>) instead of for reading (<).

I think I see what you intend to do: you want to use the file as a queue, where you append new URLs to the end, and read unprocessed ones from the start (middle).

A file is the wrong type of a datastructure for that.

You could use an array, where you push new URLs in at one end, and shift them out at the other.

Of course, in a real webcrawler, that array is going to get rather large.