I think that you will find that the problem lies with the way Windows scans directories and shares.

Have you tried using FTP instead? I think your directory access will be much quicker with FTP. Worth trying a benchmark.