I submitted an article to the Perl Journal (or was it the Perl Review...?) a while back about using Mojolicious to do this. Never did hear anything more. Hm. Well.

It boils down to it being really, really easy to make Mojo respond any way you want to a URL, so you can have your spider "visit" the Mojo URL, get a page with a bunch of links in it, and then test all the different kinds of things that could happen (timeout, 404, 500, you name it) by sending appropriately-crafted URLs to the Mojo server - which all happen to be on the first page you crawl. You need one "that's all folks" URL to make the Mojolicious server go away, but that's easy enough to do.

In reply to Re: Testing a web crawler by pemungkah
in thread Testing a web crawler by dlarochelle

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.