Thanks all!

I decided that I would try multiple times with different settings before declaring a link bad. Some servers don't do HEAD requests, others need a useragent and some want a referrer.

This makes the whole thing slower, but that doesn't matter when it runs automatically at night.

Thanks again!

--Pileofrogs