in reply to Re^2: Speeding up/parallelizing hundreds of HEAD requests
in thread Speeding up/parallelizing hundreds of HEAD requests
You mentioned caching the requests instead of actually doing them every time. This is a good way to do that. It's more efficient than a bunch of database queries, and makes it very easy to control timeouts on the cache, in case you only want to check every 2 hours or so.