If you just want to crawl the links on a site and dump the pages to disk, wget is the way to go. (regardless of if thelinks are dynamic or static)

The form issue is going to get you into trouble. I doubt you'll find any general purpose tools for doing what you want, because there are so many variables: should it enter 1 set of values in every form, what are those values? is there javascript that mutates the form input prior to submission?

The good news is, using LWP, HTML::Tree and HTML::Element It's REALLY easy to:

  1. Download a page
  2. scrape all of the links from that page and remember them
  3. check for any forms on that page
  4. get a list of all the form elements in that form
  5. prompt the user how to fill our hte form and/or look up in some datastructure what to do with forms/elements that have those names.

In reply to Re: Dumping dynamic CGIs by hossman
in thread Dumping dynamic CGIs by Michalis

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.