Yes! Definitely! Ideally I would want as fat a bandwidth as possible. Ideally with lots and lots of memory. I'm negotiating with my University but that is unlikely to get anywhere fast. It looks like I'm going to have to fork out for a dedicated server myself. Seeing that I am concentrating on .com domains at the moment geographical location and connection factors of the box could be important. I was thinking something like http://www.m5hosting.com/ValueNet.php These guys also seem to be the only ones I can find with decent OpenBSD dedicated servers. With 1,500 GB/month transfer and a maximum burstable speed of 100Mbps with no additional charge it looks like as good a deal as I'm likely to find. According to http://www.m5hosting.com/network.php they seem to be pretty well connected as well. Especially for US based traffic. I've also used these guys before and their service was pretty good. They fix problems fast and don't charge for the service. However, as it stands I'm connected to the internet through a USB modem via a mobile phones operator that offers variable bandwidths (depending on where you are) and is limited to only 100 hours per month. In any case, the amount of time I crawl the net in is always going to ultimately limited by bandwidth and how best I use it. Asynchronous DNS and HTTP seem to be fundamental issues. I'm almost at the point where I'm thinking. Why didn't I just code this in C from the very beginning? I once made (years ago and I no longer have the code) an asynchronous DNS resolver in C and, yes, it did take much longer to implement but as far as I can remember it was as fast as the 100Mbps burstable speed could take. In fact, it ran so well that we had to redirect it to a better DNS server (a cluster of load balanced DNS servers) so that they could keep up with the requests. I've even thought that maybe it is a good idea to write my own recursive resolver to see if there are ways this process can be optimised. Maybe this is overkill but I want this to work in hours (maybe in days) but certainly not in years.

In reply to Re^12: Async DNS with LWP by jc
in thread Async DNS with LWP by jc

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.