Re^2: Improve My Code...

Replies are listed 'Best First'.
Re^3: Improve My Code... by merlyn (Sage) on Aug 02, 2009 at 05:30 UTC
It says "the user choose a date". If I was writing that, I'd have the expectation that humans would be entering a date, not a mechanized attack of all dates. True, there's nothing that forbids what he wants to do, but it's also a matter of courtesy. Don't go hammering someone else's machine without permission, and I wouldn't consider that paragraph permission, just description. -- Randal L. Schwartz, Perl hacker The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.	[reply]
Re^3: Improve My Code... by Anonymous Monk on Aug 02, 2009 at 01:30 UTC
Maybe they'll think about repackaging the data. Its static, organized by date, and a CGI interface isn't really the best way to deliver it.	[reply]
Re^4: Improve My Code... by cheech (Beadle) on Aug 09, 2009 at 21:33 UTC
Can you suggest a more practical alternative?	[reply]
Re^5: Improve My Code... by Anonymous Monk on Aug 10, 2009 at 00:28 UTC
Sure 2009 \|-- 01 \| \|-- 01.xml.gz \| \|-- 02.xml.gz \| \|-- 03.xml.gz \| ... `-- 02 \|-- 01.xml.gz \|-- 02.xml.gz \|-- 03.xml.gz ... 2009.tar.gz # contains whole year 2009.tar.gz.asc # gpg signature to verify data integrity [download] The data (html) that WXDaily.exe outputs is static and fixed size (10,443 bytes). It compresses well. $ tree 1896/01 1896/01 \|-- 18960101.txt.gz \|-- 18960102.txt.gz \|-- 18960103.txt.gz \|-- 18960104.txt.gz \|-- 18960105.txt.gz \|-- 18960106.txt.gz \|-- 18960107.txt.gz \|-- 18960108.txt.gz \|-- 18960109.txt.gz \|-- 18960110.txt.gz \|-- 18960111.txt.gz \|-- 18960112.txt.gz \|-- 18960113.txt.gz \|-- 18960114.txt.gz \|-- 18960115.txt.gz \|-- 18960116.txt.gz \|-- 18960117.txt.gz \|-- 18960118.txt.gz \|-- 18960119.txt.gz \|-- 18960120.txt.gz \|-- 18960121.txt.gz \|-- 18960122.txt.gz \|-- 18960123.txt.gz \|-- 18960124.txt.gz \|-- 18960125.txt.gz \|-- 18960126.txt.gz \|-- 18960127.txt.gz \|-- 18960128.txt.gz \|-- 18960129.txt.gz \|-- 18960130.txt.gz `-- 18960131.txt.gz $ du -h 1896 124K 1896/01 124K 1896/Copy (10) of 01 124K 1896/Copy (11) of 01 124K 1896/Copy (2) of 01 124K 1896/Copy (3) of 01 124K 1896/Copy (4) of 01 124K 1896/Copy (5) of 01 124K 1896/Copy (6) of 01 124K 1896/Copy (7) of 01 124K 1896/Copy (8) of 01 124K 1896/Copy (9) of 01 124K 1896/Copy of 01 1.5M 1896 $ du -hs 1896.tar.gz 372K 1896.tar.gz [download] 1.5M to host a whole year (that is on-disk size) and 372K to download whole year. WXDaily output is structured enough to easily convert into xml, which is easily transformed into any format you required. With this arrangement the webserver is doing what it is optimized for; WXDaily.exe doesn't become a bottleneck. If 1 webserver can't keep up with demand, a mirror can be established effortlessly.	[reply] [d/l] [select]