[OT] HTTP downloads and caching

syphilis has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: [OT] HTTP downloads and caching by kcott (Archbishop) on Oct 07, 2015 at 04:59 UTC
G'day Rob, I tried this using `wget` on Mac OS X 10.10.3 as follows: ken@ganymede: ~/tmp $ wget http://www.sisyphusion.tk/ppm/Math-Float128.ppd --2015-10-07 15:10:03-- http://www.sisyphusion.tk/ppm/Math-Float128.p +pd Resolving www.sisyphusion.tk (www.sisyphusion.tk)... 184.154.90.58, 18 +4.154.90.58 Connecting to www.sisyphusion.tk (www.sisyphusion.tk)\|184.154.90.58\|:8 +0... connected. HTTP request sent, awaiting response... 200 OK Length: 4302 (4.2K) [application/vnd.cups-ppd] Saving to: `Math-Float128.ppd' 100%[================================================================= +=============>] 4,302 --.-K/s in 0s 2015-10-07 15:10:06 (216 MB/s) - `Math-Float128.ppd' saved [4302/4302] ken@ganymede: ~/tmp $ ls -al Math-* -rw-r--r-- 1 ken staff 4302 5 Oct 15:23 Math-Float128.ppd ken@ganymede: ~/tmp $ grep 07 Math-Float128.ppd ken@ganymede: ~/tmp $ grep 08 Math-Float128.ppd <SOFTPKG NAME="Math-Float128" VERSION="0,08,0,0"> <PROVIDE NAME="Math::Float128" VERSION="0.08" /> ken@ganymede: ~/tmp $ [download] So, it would appear that I'm getting the "new" version. I don't know that I can help much with the "why is this happening?" question. The following is just a shot in the dark. I encountered a situation in the past (actually about 15 years ago) where a service provider had multiple servers and changes had been applied across their servers — except they missed one (little used) server when applying the changes. The result being that everything worked as expected most of the time but, on the rare occasions when the unchanged server was accessed, problems ensued. This was so long ago that I really don't remember more details than that; however, that may be an avenue worth investigating. Your comment "The company that hosts the website says it can't possibly be their fault ..." brought this to mind as I seem to recall that was pretty much the initial feedback I got at the time. Even the senior "technical guys" refused to entertain the idea that the problem was at their end; however, this was for a substantial commercial project which allowed me to escalate the issue to their senior management and get all the servers checked (which resulted in the problem being found and fixed). That's not the sort of leverage I'd normally command; unless there's "big bucks" involved, you may be stuck with the "not our fault" answer. — Ken	[reply] [d/l] [select]
Re^2: [OT] HTTP downloads and caching by syphilis (Archbishop) on Oct 07, 2015 at 12:23 UTC
Yeah, my initial thoughts were along the lines that the hosting company had somehow stuffed up a server upgrade. If I can find a case where someone other than me can hit the same issue then I'll consider entertaining the same thought again. But, as it currently stands, it apparently affects only me and it happens to me every time I download that file over http. OTOH, if it is affecting only me, then that doesn't really matter - that would be good news. There are some other oddities that I didn't mention. When I updated www.sisyphusion.tk/ppm/Math-Float128.ppd, I also updated www.sisyphusion.tk/ppm/package.xml - but there's no problem grabbing the updated version of package.xml. It's only the updated version of Math-Float128.ppd that I can't grab. Now that's really odd, isn't it. Two files in the same directory - no problem getting the current version of one of them, but I can only get the previous version of the other ! Actually, there's at least one other file (in a different directory) that was also updated, and is affected in the same way. Anyway, thanks for checking - it helps to know that it's not just the hosting company that can wget the current version of the file. And thanks for going to the additional trouble of thinking about it, and of articulating those thoughts. I would strongly suspect that my problem was indicative of some caching on my local machine if not for the fact that it's happening on all three of my local machines - two (ie the two Linux ones) of which don't get fired up all that often. That all three machines are affected in the same way leads me to think that it's my ISP that's throwing the spanner into the works. (My next step will probably be to see what my ISP thinks of that hypothesis :-) Cheers, Rob	[reply]
Re^3: [OT] HTTP downloads and caching by hippo (Archbishop) on Oct 07, 2015 at 12:54 UTC
That all three machines are affected in the same way leads me to think that it's my ISP that's throwing the spanner into the works. (My next step will probably be to see what my ISP thinks of that hypothesis :-) That sounds like the obvious conclusion to me, too. You could always check whether it is a caching issue (either at the ISP or elsewhere) by appending a query string to the URL. Or indeed by using an https URL if one is available for that resource. Good luck with your investigations.	[reply]
Re^4: [OT] HTTP downloads and caching by syphilis (Archbishop) on Oct 08, 2015 at 03:29 UTC
Re^5: [OT] HTTP downloads and caching by Your Mother (Archbishop) on Mar 25, 2017 at 15:04 UTC
Re^5: [OT] HTTP downloads and caching by hippo (Archbishop) on Oct 08, 2015 at 13:49 UTC
Some notes below your chosen depth have not been shown here
Re: [OT] HTTP downloads and caching by syphilis (Archbishop) on Mar 25, 2017 at 10:28 UTC
This upstream malware attack has recurred again today, starting nearly 24 hours ago - same site, different files. This time I'm seeing that the "query string" solution is still working, but it doesn't clear the cache. My problem is that: `ppm install http://www.sisyphusion.tk/ppm/Cairo.ppd` [download] installs an older version of the Cairo ppm package than is currently on the server. It accesses an outdated (non-existent) Cairo.ppd, and installs outdated (non-existent) binaries. Out of curiosity I tried: `ppm install http://www.sisyphusion.tk/ppm/Cairo.ppd?no=cache` [download] but the ppm utlity croaks on that. Besides, it still would have installed the cached binaries. I haven't tried `wget --no-cache` yet as that cleared the cache last time. Instead, I've decided to leave the cache uncleared in case it helps my ISP (who I've contacted again) remove the malware. My ISP did not respond last time ... let's see what happens this time. This update is largely for my own records - but I'm also submitting it in case someone has something to add. Cheers, Rob	[reply] [d/l] [select]
Re^2: [OT] HTTP downloads and caching by syphilis (Archbishop) on Jun 09, 2017 at 11:26 UTC
I haven't tried wget --no-cache yet as that cleared the cache last time. Thankfully `wget --no-cache` still does the trick. And I've come across an improvement in the form of `wget --spider --no-cache` which clears the cache without having to actually download the file. (This is particularly useful if the cached file is large ... as can be the case with some of the tar.gz files that PPM needs to download.) I still haven't been ale to work out just where the offending cache is located - though I have established that it's upstream of my router. It still remains to work out how I can patch Strawberry's ppm utility such that running this wget command in advance is not needed. Cheers, Rob	[reply] [d/l] [select]
Re^3: [OT] HTTP downloads and caching by Anonymous Monk on Jun 10, 2017 at 06:19 UTC
My off the cuff guess is that your ISP is running a so called transparent proxy. (Cox?) Also, it sounds like this server is under your control. If so, you should be able to configure your web server to use HTTP to prevent the caching. I've linked to the w3 doc for 1.1, look for the section 14.9 Cache-Control which includes a comment on Pragma: no-cache for HTTP 1.0 There are ways to probe the caches and figure out what is going on but I believe that cache-control should do the trick. The next step is less easy. https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html	[reply]
Re^4: [OT] HTTP downloads and caching by syphilis (Archbishop) on Jun 11, 2017 at 02:26 UTC
Re^4: [OT] HTTP downloads and caching by Anonymous Monk on Jun 10, 2017 at 06:22 UTC
Re: [OT] HTTP downloads and caching ... groan ... not again ... please reap by syphilis (Archbishop) on Mar 25, 2017 at 10:28 UTC
This malware attack has recurred again today, starting nearly 24 hours ago - same site, different files. This time I'm seeing that the "query string" solution is still working, but it doesn't clear the cache. My problem is that: `ppm install http://www.sisyphusion.tk/ppm/Cairo.ppd` [download] installs an older version of the Cairo ppm package than is currently on the server. It accesses an outdated (non-existent) Cairo.ppd, and installs outdated (non-existent) binaries. Out of curiosity I tried: `ppm install http://www.sisyphusion.tk/ppm/Cairo.ppd?no=cache` [download] but the ppm utlity croaks on that. Besides, it still would have installed the cached binaries. I haven't tried `wget --no-cache` yet as that cleared the cache last time. Instead, I've decided to leave the cache uncleared in case it helps my ISP (who I've contacted again) remove the malware. My ISP did not respond last time ... let's see what happens this time. This update is largely for my own records - but I'm also submitting it in case someone has something to add. Cheers, Rob	[reply] [d/l] [select]