oalvi has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

My first posting here - I have been trying to script something robust which could monitor websites. I've mostly used perls for pattern matching in logfiles, and for reporting purposes.

I was suggested to use LWP, and I'm still learning the ropes in this function, but I've been halted in my tracks by some " 500 connect time out" errors.

I've even set the proxy server, but with no success.

Following is the most recent version I've been using.

---------------
#!C:\perl\bin use strict; use warnings; use LWP::UserAgent; my $wsr = LWP::UserAgent -> new; $wsr -> timeout( 20 ); $wsr->proxy(['http', 'ftp','https'], 'http://192.168.5.39:8080/'); my $url = shift or die "URL expected\n"; my $useragent = LWP::UserAgent->new; my $request = HTTP::Request->new( HEAD => $url ); my $response = $useragent->request($request); print $response->status_line, "\n"; print $response->is_success, "\n";
-------------

Had some limited success but getting the timeout errors for most websites.

C:\SCRIPTING>urltestv4.pl http://www.hp.com

500 Can't connect to www.hp.com:80 (connect: timeout)

C:\SCRIPTING>urltestv4.pl http://www.hp.com:8080

500 Can't connect to www.hp.com:8080 (connect: timeout)

There doesn't seem any delay in accessing the internet from this particular server.

The following, a local site with no security restrictions, came back successfully.

C:\SCRIPTING>urltestv4.pl http://elm-as-01

200 OK

The thing is that the external websites (behind proxy) are givign time outs and not authenitcation or forbidden messages.

C:\SCRIPTING>urltestv4.pl http://www.google.com

500 Can't connect to www.google.com:80 (connect: timeout)

Lookin forward to your support

Thanks and regards,

-Alvi

Replies are listed 'Best First'.
Re: LWP for URL monitoring - 500 connect timeout errors
by ig (Vicar) on Jul 20, 2009 at 02:00 UTC

    You set $wsr as an instance of LWP::UserAgent and set the proxy server on this instance but you don't use it. You then set $useragent as another instance of LWP::UserAgent, don't set the proxy server and use this latter instance to submit your request.

    Does it work if you set the proxy server on the $useragent instance?

      Hi ig,

      Thanks a million, that was it; now I can continue in peace. I'd done all kinds of troubleshooting, and reading up.

      By the way, how is it that we can limit a proxy not to be used for local addresses, and even if we do that, how will local addresses be identified by the proxy?.

      .

      Really, appreciate the excellent support and patience to look into the code itself :).

      .

      Thanks and Regards.

        You can use the no_proxy method of LWP::UserAgent to specify hosts or domains that should not be passed to the proxy server. It takes a list of strings that are matched verbatim (i.e. the strings are not regular expressions against the host name in the URL).

        The match is anchored to the end of the host name which, if the host name is a fully qualified DNS name, allows one no_proxy string to match any hosts within that domain.

        I don't see any provision for easily excluding URLs by network (e.g. 192.168.0.0/24). You could list all the IP addresses in the network.

        And, now that your problem is solved, I suggest you investigate some of the excellent applications available for monitoring websites and other network resources.

        Freshmeat has an article (picked lazily and somewhat dated) that lists some of the features you might think about, with links to a few of the available applications. Otherwise, search for Network Management System to find much more.

        Even if you are doing this as a learning exercise, I suggest you have a look at what existing applications do. They will give you many ideas for features to implement.

Re: LWP for URL monitoring - 500 connect timeout errors
by zwon (Abbot) on Jul 19, 2009 at 19:14 UTC
    The thing is that the external websites (behind proxy) are givign time outs and not authenitcation or forbidden messages.

    It looks like your proxy can't connect to external servers. Perhaps firewall problem.

      But, I am accessing all websites from my browser (IE 7) with the same proxy configured.

      as I have the proxy configured, I'm not aware that the command line, or code would behave differently as compared to a browser, or is that the case?

      Additonally, the thing is its NOT giving a forbidden or authentication codes, its giving only a timeout. And the default period is 180 seconds - and still it times out.

      Authentication and forbidden errors seem to come up immediately.

Re: LWP for URL monitoring - 500 connect timeout errors
by ikegami (Patriarch) on Jul 19, 2009 at 21:05 UTC
    What does the following give?
    perl -MIO::Socket::INET -le"IO::Socket::INET->new(PeerAddr => '192.168 +.5.39:8080') or die $^E; print 'ok'"
Re: LWP for URL monitoring - 500 connect timeout errors
by Anonymous Monk on Jul 20, 2009 at 13:21 UTC
    Hi, Well, I've progressed some as you can see from this thread - but I've run into another issue, slightly different. How to deal with redirections. For example I've got the following website to monitor http://www.eserve.com.sa Which redirects to http://www1.elm.com.sa/Portal/Ar/Topics/Services/ However, for monitoring purposes I would like to monitor the first website as the rederiction can be changed. Now, following was my script:
    # !C:\Perl\bin use strict; use warnings; use LWP::UserAgent; my $url = shift or die "URL expected\n"; my $useragent = LWP::UserAgent->new; $useragent->proxy(['http', 'ftp','https'], 'http://192.168.5.39:8080/' +); $useragent -> timeout( 20 ); my $request = HTTP::Request->new( HEAD => $url ); my $response = $useragent->request($request); print $response->status_line, "\n"; print $response->is_success, "\n"; print $useragent->proxy('http'),"\n";
    And following is the response to ach of the abiove websites: C:\SCRIPTING>urltestv5.pl http://www.eserve.com.sa 400 Bad Request C:\SCRIPTING>urltestv5.pl http://www1.elm.com.sa/Portal/Ar/Topics/Services/ 200 OK Is there any way around this? Or if I can capture the redirection from the script itself and then do HEAD to that? Any ideas? Appreciate your support and assistance
      That is weird, can you try
      # for window set http_proxy=http://192.168.5.39:8080/ # for linux export http_proxy=http://192.168.5.39:8080/ lwp-request -USe -m get https://www.eserve.com.sa/ lwp-request -USe -m head https://www.eserve.com.sa/
      and show the output
        So, is it an anomaly,
        C:\Perl\bin>lwp-request -Use -m get http://www.eserve.com.sa/ GET https://www.eserve.com.sa/ User-Agent: lwp-request/5.810 500 Connect failed: connect: Unknown error; Unknown error Content-Type: text/plain Client-Date: Tue, 21 Jul 2009 08:39:54 GMT Client-Warning: Internal response 500 Connect failed: connect: Unknown error; Unknown error C:\Perl\bin>lwp-request -Use -m head http://www.eserve.com.sa/ HEAD https://www.eserve.com.sa/ User-Agent: lwp-request/5.810 500 Connect failed: connect: Unknown error; Unknown error Content-Type: text/plain Client-Date: Tue, 21 Jul 2009 08:40:09 GMT Client-Warning: Internal response

        Don't know how to proceed from here. Or iwll I have to simply hardcode all the redirections?