My question is how do I get LWP useragent to act like a browser and find the default page in a directory?

It has nothing to do with your browser, and everything to do with your web server. I tested your example on a site I had control of (running apache). Here's what happened: 
[jon@valium jon]$ telnet divisionbyzero.com 80
Trying 168.103.109.84...
Connected to divisionbyzero.com.
Escape character is '^]'.
GET /decss HTTP/1.0

HTTP/1.1 301 Moved Permanently
Date: Tue, 19 Feb 2002 02:47:50 GMT
Server: Apache/1.3.22 (Unix)  (Red-Hat/Linux) mod_ssl/2.8.5 OpenSSL/0.
+9.6b mod_perl/1.24_01
Location: http://www.divisionbyzero.com/decss/
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>301 Moved Permanently</TITLE>
</HEAD><BODY>
<H1>Moved Permanently</H1>
The document has moved <A HREF="http://www.divisionbyzero.com/decss/">
+here</A>.<P>
<HR>
<ADDRESS>Apache/1.3.22 Server at www.divisionbyzero.com Port 80</ADDRE
+SS>
</BODY></HTML>
Connection closed by foreign host.

[download]
The web server sent me a 301 since /decss wasn't an actual file, but rather, a directory. My web browser followed that redirect automatically, which is what browsers are supposed to do when the http method used is GET or HEAD. I suspect your troubles are caused because you are using the POST method, which is explicitly forbidden to redirect you without notifying the user.

BlueLines

Disclaimer: This post may contain inaccurate information, be habit forming, cause atomic warfare between peaceful countries, speed up male pattern baldness, interfere with your cable reception, exile you from certain third world countries, ruin your marriage, and generally spoil your day. No batteries included, no strings attached, your mileage may vary.

In reply to Re: Checking "incomplete" URLs by BlueLines
in thread Checking "incomplete" URLs by nop

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.