The file size shouldn't affect performance very much. The real answer will depend on what file system you use and how scattered over the disk your file list may be. A very long file list may trigger swap if you are short on memory.

I don't think that 100-200k unlink's will take more than four hours so long as you use unlink instead of calling system rm. Runtime will be faster for journaled file systems, but I can't swear to the speed of the background operations.

In short, we don't know enough about your code or your system to give better than general advice.