Hello all-knowing Perl Monks,

This is my first offering but not my first visit past these gates.
I apologize in advance if this question is too large (too much text)!

I am trying to automate an LWP lookup at a doctor-search web site by importing a list of doctor’s names from a text file and printing the results to an html file (as well as displaying them in the browser window). I have rewritten this chunk as many times and ways as my skill and knowledge provides – and now I turn to you!

This doctor-search site will either return
  * doctorname “not found”
  * the doctor’s info
  * an options-list of found doctors by doctor-license number
When the site returns an options list, I pull out the option-values and re-search for each option.

What is/is NOT happening:
When my imported list has only one name, the script works great.
When the list has multiple names (as intended) I get “not found” results (thus a response from their site) from all the searches except the last name from my list, which then gives me the intended doctor’s info.

What I’ve tried:
I tried slowing down the script thinking maybe I was overwriting existing UserAgent requests… but decided this wasn’t the case since I CAN receive all the responses for the options-list to one doctor’s name (if it’s the last on the list), which entails multiple requests.

I broke down each code-chunk and tested it as a unique entity. They all work.


This is the core code chunk:
use LWP::UserAgent;
use HTTP::Request::Common qw(POST);

  $ua = LWP::UserAgent->new;
  $ua->agent("Mozilla/4.0 (compatible; MSIE 5.0; Windows 95)");

  &ParseForm; ### this sub-routine not included here ###

  $input = $key{'input'};
  $output = $key{'output'};

  if( -r $input ){
    open(IN, "<$input") || print "$!";
    @list = <IN>;
    close(IN);
  }else{
    print "<h3>Could not read from INPUT file</h3>";
    exit;
  }

  &ParseList;

  exit;


sub ParseList {

  foreach $line (@list){
    my($first, $last) = split(/\|/, $line);

    my $content = AimAgent($first,$last);

    if( $content =~ /<option/gi ){
      $_ = $content;

      while( /<option.*?value=\"(.*?)\">/gi ){
        my $profile = AimAgent($first,$last,$1);
        PublishProfile($first,$last,$profile); ### not included ###
      }
    }else{
      PublishProfile($first,$last,$content); ### not included ###
    }

  }
}



sub AimAgent {

  my($first, $last, $license) = @_;

  my $request = HTTP::Request->new(POST=>"http://cgi.docboard.org/cgi-
+shl/nhayer.exe");
  $request->content_type("application/x-www-form-urlencoded");

  if( $license ){
    $request->content("form_id=medname&state=na&medlname=$last&medfnam
+e=$first&mednumb=$license");
  }else{
    $request->content("form_id=medname&state=na&medlname=$last&medfnam
+e=$first");
  }

  my $response = $ua->request($request);

  if( $response->is_success ){
    return $response->content;
  }else{
    return $response->status_line;
  }
}

[download]

In reply to Batch process LWP search by langsor

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.