in reply to Fast efficient webpage section monitoring

Leaving the legality to your own assessment/conscience; I would have thought doing a head and checking the headers for expiry information would reduce your overhead; though I'm not sure how that plays with in-page changes.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
In the absence of evidence, opinion is indistinguishable from prejudice.
  • Comment on Re: Fast efficient webpage section monitoring

Replies are listed 'Best First'.
Re^2: Fast efficient webpage section monitoring
by Marcool (Acolyte) on Apr 02, 2016 at 18:37 UTC
    Are you referring to the if-modified-since option in curl? Unless I'm mistaken, the site doesn't seem to be issuing this part of the header (the "Last-Modified" line is missing from curl -I http://...)
[reply]
      the "Last-Modified" line is missing from curl

      That's one I was thinking about; there are other that might be useful if the server provides them:

      1. Age:The age the object has been in a proxy cache in seconds. Eg. Age: 12
      2. Content-MD5:A Base64-encoded binary MD5 sum of the content of the response. Eg. Content-MD5: Q2hlY2sgSW50ZWdyaXR5IQ==
      3. Content-Length The length of the response body in octets (8-bit bytes). Eg. Content-Length: 348
      4. ETag:An identifier for a specific version of a resource, often a message digest. Eg. ETag: "737060cd8c284d8af7ad3082f209582d"

      Basically, look at what headers are returned and check them from consecutive requests and look for anything that doesn't change per-request; but does change when the content changes.

      Just a thought.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom