my code is erroring out when i have it read its urls from a list

RayRay459 has asked for the wisdom of the Perl Monks concerning the following question:

Monks...
My script worked when i hard coded the url address into the script. Now that i want the url read in and then when it is done move on to the next one...it is erroring out. ..I am printing the response code on the screen along with the url and when it is done with the first url it moves onto the second. I get a 200 (which is what i want) and then it hangs for a min and then i get 400s...they don't stop until i hit Ctrl + C. i added a sleep in my script to slow things down a bit, and that helps me to stop it before it gets really out of hand...but it seems that the first url data isn't getting flushed after it runs through my while loop. Can anyone help? Below i will post my code, my input list and the results.txt
Thnx.
Ray

#!D:/perl/bin -w

    use LWP::UserAgent; 
    use HTTP::Request; 
    use HTML::TableExtract; 
     
      
    $OutFile = "results.txt"; 
      
    
    open(OUT,">>$OutFile") || die "Can't create $OutFile: $!"; 
    open(IN,"ipList1.txt") || die "Can't open ipList1.txt: $!";
    
    OUTER: while(<IN>) {
    $url = $_;
    print $url . "\n";
    $ua = new LWP::UserAgent; 
    $request = new HTTP::Request('GET',$url); 
    $request->authorization_basic('login', 'password'); 
    $ua->timeout(10); 
    $response     = $ua->request($request); 
    $responsecode = $response->code(); 
    print $responsecode . "\n";
    INNER: if ($responsecode != 200) { 
        sleep 1;
        redo OUTER; 
    } else { 
        # login successful, let's get the html code into a variable 
        @ARRAY_OF_LINES = (split "\n", $ua->request($request)->as_stri
+ng); 

        foreach $line (@ARRAY_OF_LINES) { 
             $html_code .= $line . "\n"; 
         
                     
        }       
    }   
 
$te = new HTML::TableExtract( depth => 0, count => 1 ); 
$te->parse($html_code); 

foreach $ts ($te->table_states) {
    #print "Table (", join(',', $ts->coords), "):\n";
    foreach $row ($ts->rows) {
        $td = join(" ", @$row);
        $td =~ s/\n/ /g;
        if ( $td =~ /^\s*(\d)\s+(\w+)\s+(\d)\s+(\w+)/) {
            $outlet1 = $1;
            $host1   = $2;
            $outlet2 = $3;
            $host2   = $4;
            print OUT "Outlet: $outlet1, Host: $host1\n";
            print OUT "Outlet: $outlet2, Host: $host2\n\n";
            sleep 1;
        }
    }
}
}
[download]

Here is a sample of my input list:


http://192.168.10.20/pdumaina
http://192.168.10.21/pdumaina
[download]

And here is my results.txt:


Outlet: 1, Host: KanaApp1
Outlet: 5, Host: Kyle

Outlet: 2, Host: KanaApp2
Outlet: 6, Host: Cres

Outlet: 3, Host: Kenny
Outlet: 7, Host: Ebb2

Outlet: 4, Host: Eric
Outlet: 8, Host: Flood2

Outlet: 1, Host: KanaApp1
Outlet: 5, Host: Kyle

Outlet: 2, Host: KanaApp2
Outlet: 6, Host: Cres

Outlet: 3, Host: Kenny
Outlet: 7, Host: Ebb2

Outlet: 4, Host: Eric
Outlet: 8, Host: Flood2

Outlet: 1, Host: KanaApp1
Outlet: 5, Host: Kyle

Outlet: 2, Host: KanaApp2
Outlet: 6, Host: Cres

Outlet: 3, Host: Kenny
Outlet: 7, Host: Ebb2

Outlet: 4, Host: Eric
Outlet: 8, Host: Flood2

Outlet: 1, Host: KanaApp1
Outlet: 5, Host: Kyle

Outlet: 2, Host: KanaApp2
Outlet: 6, Host: Cres

Outlet: 3, Host: Kenny
Outlet: 7, Host: Ebb2

Outlet: 4, Host: Eric
Outlet: 8, Host: Flood2

Outlet: 1, Host: KanaApp1
Outlet: 5, Host: Kyle

Outlet: 2, Host: KanaApp2
Outlet: 6, Host: Cres

Outlet: 3, Host: Kenny
Outlet: 7, Host: Ebb2

Outlet: 4, Host: Eric
Outlet: 8, Host: Flood2

Outlet: 1, Host: KanaApp1
Outlet: 5, Host: Kyle

Outlet: 2, Host: KanaApp2
Outlet: 6, Host: Cres

Outlet: 3, Host: Kenny
Outlet: 7, Host: Ebb2

Outlet: 4, Host: Eric
Outlet: 8, Host: Flood2
[download]

As you can see ...it never gets any info from the second url. I just keeps going over the first URL please help.

Comment on my code is erroring out when i have it read its urls from a list Select or Download Code

Replies are listed 'Best First'.
Re: my code is erroring out when i have it read its urls from a list by blakem (Monsignor) on Sep 12, 2001 at 01:47 UTC
For one thing, the structure of your loops is overly complex. You have: `OUTER: while( ... ) { ... INNER: if ( ... ) { ... redo OUTER; } else { ... } }` [download] 'IF' is a branching construct, not a loop, so you don't need the labels to do this. Restructure it to: `while( ... ) { ... if ( ... ) { next; # <-- next is all you need } else { ... } }` [download] This might actually fix your problem as well.... -Blake	[reply] [d/l] [select]
(tye)Re: my code is erroring out when i have it read its urls from a list by tye (Sage) on Sep 12, 2001 at 02:59 UTC
One problem is that you don't initialize $html_code and so, even if you fetched the data from the second site, it would (at least) be preceeded by a second copy of the data from the first site before the data from the second site was output. Depending on details of how you parse the HTML, it might even prevent data from the second site being displayed at all. - tye (but my friends call me "Tye")	[reply]
Re: my code is erroring out when i have it read its urls from a list by buckaduck (Chaplain) on Sep 11, 2001 at 23:04 UTC
I'm not sure I understand, but have you tried chomping the URLs after you read them? `OUTER: while(<IN>) { chomp; $url = $_; ...` [download] It's possible that newline characters after the URL are causing some kind of problem... buckaduck	[reply] [d/l]