http://qs1969.pair.com?node_id=342561

anev has asked for the wisdom of the Perl Monks concerning the following question:

I'm a perl newbie and have a question for you perlmonks... I have a particular URL, and I need to find out if the content for that URL is available on the hosting server. I would use the Ping module, but I read in the Perl Cookbook that "It's better to think of a ping as testing whether a machine is reachable, rather than whether the machine is doing its job." So it will only tell me if the server is there, but won't let me know if I can access the content for that URL (i.e. is the site up and running? or is it down?). Is there something like Ping that I could use to find out this information? Thanks in advance! Anev

Replies are listed 'Best First'.
Re: How to check if a website is up?
by nmcfarl (Pilgrim) on Apr 05, 2004 at 04:14 UTC
    This simplest solution is LWP::Simple
    use LWP::Simple; my $url="http://cpan.org"; if (! head($url)) { die "The Server is DOWN!!!!" }
    an alternative is using a get:
    if (! get($url)) {
    but that would fetch the content of the page. A head call uses the least resources, just telling you weather the machine is responding to that url.

      There is a whole continuum of remote server "up" detection - from mere connectivity, through to accurate and timely responses. A machine may be in several states and conditions, and depending on quite what you are looking for, some of which can be:

      • single-user
      • multi-user
      • port 80 accepting
      • http://domain responding
      • http://domain/file responding
      • http://domain/file responding quickly enough
      It may also be that the site content is unpredictable, so a specific URL may not be something you want to check.

      Then there is a question of whether one of the unacceptable conditions might change if you retry. For example, instantaneous loads may indicate a very poor response time, but with a perfectly acceptable average.

      Hi I use the head as you mentioned, but it always says the server being down with the successful URL.
Re: How to check if a website is up?
by biosysadmin (Deacon) on Apr 05, 2004 at 03:55 UTC
    You want the LWP::UserAgent module. Here's some example code from the manpage:
    require LWP::UserAgent; my $ua = LWP::UserAgent->new; $ua->timeout(10); my $response = $ua->get('http://search.cpan.org/'); if ($response->is_success) { print $response->content; # or whatever } else { die $response->status_line; }
    This will try connect to the URL 'http://search.cpan.org' and $response->is_success will be true if the HTTP GET request was successful.

    As an FYI a ping is a special sort of request sent to a machine in order to determine if it is accepting network connections. It's very often used in port scanning, and many people will turn off ping response in order to make their machines appear to be down to the casual ping scanner. Lack of a ping response doesn't mean a machine is down, and existence of a ping response doesn't mean that the machine is accepting HTTP requests.

    Hope that this helps. :)

Re: How to check if a website is up?
by Abigail-II (Bishop) on Apr 05, 2004 at 09:10 UTC
    Ping is no help to you. Ping is a networking debugging tool. To check whether a service is running, you will have to actually use the service. Request the page (using your favourite module). And go one step further than what the other replies are saying. Don't just look for a 200 status response. Also look at what you are actually getting is what you think you should be getting. A misconfigured Apache server, or one running on a machine having DNS problems, might serve a default page instead of the (virtual) hosts pages.

    Abigail

Re: How to check if a website is up?
by chimni (Pilgrim) on Apr 05, 2004 at 08:27 UTC
    Hi,
    You could check out LWP::UserAgent and LWP::Simple.
    If you want to extend into monitoring and health checks look at WWW::Mechanize.
    use WWW::Mechanize; my $mech = WWW::Mechanize->new(); $mech->get( $url );

    Just get the page and check for a 200 status (the documentation on these modules is self explanatory)
    If you dont want to go with a perl module and want a simple command line solution check WGET.
     wget -q --proxy=off --cache=off $URL -O /tmp/urlget.html
    HTH
    Regards,
    Rajdeep.
    Abigail is right,content checking for a string you are expecting is a good idea
Re: How to check if a website is up?
by inman (Curate) on Apr 05, 2004 at 09:33 UTC
    You need to conduct a series of positive tests. As already mentioned, LWP (and it's derivatives) are what you want to use to write your tests. You will need to write a series of tests if you want to determine whether the whole of a website is up and running (rather than just a simple page).

    I have previously used Test::Simple to build a set of tests that excercise various parts of a website. Example tests include:

    • Get the homepage - as the user would see it. This tests a load balancer if present.
    • Get pages from servers directly (without a load balancer)
    • Login and retrieve the appropriate session cookies.
    • Get pages that are only returned for logged in users.
    • Submit a search to the search engine and verify that an acceptable number of results were returned.

    Typically you try and test individual pieces of functionality so that you can pin-point the failure. This is especially important where a website relies on multiple hardware and software components performing together.

Re: How to check if a website is up?
by Vautrin (Hermit) on Apr 05, 2004 at 04:49 UTC
    Since pings can be used to DOS a server, many sites disable pinging. Thus, if you get an unreachable host by ping, you may still be able to view its web page. Someone mentioned LWP::Simple, which is probably your best bet for web content. For anything else (i.e. mail), use IO::Socket to open up a port on the remote machine. What you'd want to look for would depend on what protocol you were monitoring (i.e. for SMTP you might look for an HELO string)

    Want to support the EFF and FSF by buying cool stuff? Click here.
Re: How to check if a website is up?
by Koosemose (Pilgrim) on Apr 05, 2004 at 03:47 UTC

    Well probably the simplest thing (I can think of) would be to just request the page's headers. As I'm not very familiar with any of the HTTP related modules, I'll just drop that Idea and leave implementation to you, or perhaps another friendly monk.

    Just Another Perl Alchemist
      thank you all for the extremely helpful pointers! :) Anev
Re: How to check if a website is up?
by blue_cowdawg (Monsignor) on Apr 05, 2004 at 16:11 UTC

        So it will only tell me if the server is there, but won't let me know if I can access the content for that URL

    Besides all of the suggestions where LWP::Simple are implemented you can also take a sniglet out of one of my scripts. This won't check the content itself but will check to see if the server in question is serving pages at all:

    use Socket; : : # skip a few lines of a very large script : : sub portCheck { my ($host,$port)=@_; my ($iaddr,$paddr,$proto,$line); + $iaddr=inet_aton($host) or die "Could not aton!"; $paddr=sockaddr_in($port,$iaddr); $proto= getprotobyname('TCP'); socket(SOCK, PF_INET, SOCK_STREAM, $proto) || die "socket: $!"; my $res=connect(SOCK, $paddr); + return 1 if $res; return 0; + + }
    You would invoke it as in the following example:
    : : : portCheck("www.mysite.com",80) or die "Server not serving!"; : :

    If you want to check the content itself to make sure it hasn't been hacked or otherwise horribly modified then you might want to use LWP::Simple and friends runnng it one time when you know that A) the site is up and B) the page itelf is intact, generate an MD5 sum or some other fingerprint of your choosing and store that someplace for later comparison.

    The sniglet above comes out of a very old script that I deployed a long time ago and it runs against several sites that I have interest in (clubs I belong to, my own web site, former customers of mine, etc.) and pages me if the site is unreachable so I can make the proper notifications.

    I plan on overhauling it though given I have learned many better ways of doing things that the script does now which is why it hasn't been posted here in its entirety.

Re: How to check if a website is up?
by jdavidboyd (Friar) on Apr 05, 2004 at 16:41 UTC
Re: How to check if a website is up?
by talexb (Chancellor) on Apr 05, 2004 at 18:59 UTC

    As many have told you already, a ping isn't really the right tool, as some network interfaces have ping disabed (for security reasons and also to prevent DOS attackes); also, a ping won't tell you if the web server is running. If I can stretch a metaphor here, pinging is like calling someone's house and getting the (plugged into the wall) answering machine. What you really want to do is make sure the web server is returning valid content, or inside my metaphor, reach one of the folks in the house and speak to them. They are not quite the same thing.

    From the *nix command line, you can use LWP::Simple and do a one-line script, or you can even use GET to get one (or more) of the pages from the site.

    Finally, I can highly recommend Nagios as a monitoring tool if you want to jump up a level to an industrial strength method.

    Alex / talexb / Toronto

    Life is short: get busy!

Re: How to check if a website is up?
by visuss (Initiate) on Oct 28, 2007 at 07:50 UTC
    Wonderful suggestions ! Though a old thread someone might be still needing help with such a thing as I do. I tired using Net::DNS module to lookup the domain name and it fails in most of the cases. Probably need much more configuration on the Name Server side. The LWP Head module seems to do the same task pretty simple and quick. As I only wanted to check if the domain name exists. What other things should i consider ? and is there really any difference testing a page http://www.whizblaze.com and http://whizblaze.com / Socket does not seem to be perfect all the times as there are still people who has configured http request to a different port.
Re: How to check if a website is up?
by Anonymous Monk on May 23, 2009 at 21:31 UTC
    use http://www.monitis.com
Re: How to check if a website is up?
by Anonymous Monk on Mar 21, 2009 at 18:22 UTC
    go to this website isitup.org