--k.
In reply to Re^2: Advice on Efficient Large-scale Web Crawling by Kanji in thread Advice on Efficient Large-scale Web Crawling by Anonymous Monk