prathap keerthipati has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I need to download the content of the websites which are having dynamic ipaddresses. Is there a possibility to do it through perl scripts?

  • Comment on Connection to urls which are having dynamic ipadddresses through perl scripts

Replies are listed 'Best First'.
Re: Connection to urls which are having dynamic ipadddresses through perl scripts
by huck (Prior) on Jun 29, 2017 at 09:43 UTC

    Sure, how do you get the ip address now?

    Just write a script segment to automate that, start by running that segment, and if you get a "cant connect error", rerun that segment and retry with that new address

      Thanks for the reply

      I have used nslookup command to get the ipaddresses. Its is giving some 5 to 6 ipadresses. When i tried to open the website with these ipaddresses in chrome i am getting an error like "Bad request". What does it mean? Are these not the original ones?

        There are a lot of reasons you can get a 400-bad request error, most are due to malformed requests. But one in particular can happen if you are trying to access via an ip address number. Many places use what are called virtual hosts, that is the same computer services many webistes. Since they all share the same ip address number the way it can tell which one you are asking for is by a Host: header. But if you try to go to a website via a ip address number in a browser it will not include the host header or include the host: header with the ip address number. And so that webserver cannot tell which site you are trying to access and returns the 400 error.

        For instance a friend of mine has a business website at http://jimstubblefieldfurniture.com. This is a shared webhost at the former yahoo. If you look up its ip address you find it is 67.195.61.46. but that is not the only website at that ip address. if you try to go to http://67.195.61.46 you get the "400 - bad request" error because it does not know what website you are trying to reach because it does not include a host header or instead includes a host:67.195.61.46 header

        If you were using LWP to reach a site like this you could as LWP to include a proper Host: header with the dns name of the site you are going to, but i dont know of a way to ask a browser to do so.

        Most of the reasons nslookup returns multiple ip numbers is for what is called load balancing. The numbers dont always get returned in the same order and your browser will use the first it finds. This results in the load being spread out over many servers. This is not really "dynamic ipaddresses" as the numbers dont change, just the order. In a case like that you still want to use the full name to access the website in a browser anyway.

Re: Connection to urls which are having dynamic ipadddresses through perl scripts
by talexb (Chancellor) on Jun 29, 2017 at 13:24 UTC

    Yes, it's possible. What have you tried?

    Also how dynamic are these IP addresses? Usually when you go to a website, you specify a URL, and the browser just takes you there. Under the hood, your local DNS service resolves the name into an IP address .. the IP address of a website doesn't usually change that much, so I'm wondering if you have a really stinky DNS service locally that you're even aware of this problem. You can always fall back on Google's DNS server at 8.8.8.8 (really) if necessary.

    Alex / talexb / Toronto

    Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.

A reply falls below the community's threshold of quality. You may see it by logging in.