Re: Parsing data from a report not meant for machine readability.

On another tangent, perhaps you could consider an automated tool that presents a marked-up version of the advisory, for a human to peruse and approve or disprove before automatic updating.

Essentially, upon reception of an advisory, a pre-tool attempts to parse as much as possible, and then marks up the original report in varying colours etc to show exactly what the pre-tool understood. If the human approves this, then the data extracted from the pre-tool goes straight into the post-tool (which does the graphing).

It's a compromise between entirely-human oriented data fetching, and entirely-automated processing. At least in the short term it would give you confidence in the processing quality of your new tool.

Comment on Re: Parsing data from a report not meant for machine readability.

Replies are listed 'Best First'.
Re^2: Parsing data from a report not meant for machine readability. by Anonymous Monk on Jun 09, 2007 at 16:34 UTC
I am not a Perl programmer, but rather PHP. I just finished the same task. If you've already figured this out, congrats! If not, here are some hints. First thing to keep in mind is what data are you hoping to extract? The watches and warnings issued for storms are not standard in this or any of the other standard package. There is a new form the NHC publishes when watches/warnings are issued. I do not know the header but they can be found on the NHC's website. The other thing is with the wind fields and forecasts. All wind fields will not be present (if a storm has 40kt winds, they will not issue wind fields for 50kts and 64 kts respectively). Same thing with the forecasted wind fields. Not all forecasts will have all the usual wind fields anyway. i.e. even if a storm is expected to have 100kt winds in 3 days, the 3 day forecast will only list 50kt wind radii. Make sense? Regarding the forecasts, there will not always be all five time periods for obvious reasons: if the storm makes landfall or is expected to dissipate. You should check your forecast line to make sure it contains valid data. If not, just break out of that portion of your script. Other items are optional: the eye diamter for example. For a complete advisory, I used Katrina (2005), advisory 13. That forecast/advisory lists all the fields with valid data. Also, be careful about corrected advisories. You may get an advisory just issued five minutes ago and then five minutes from you getting hte info, the NHC will correct something. Could be winds, lat, lon, anything vital. If your running a chron job, run it continuously till at least 30 minutes after the scheduled advisory release (0300Z, 0900Z, 1500Z and 2100Z (Z is same as UTC)). Other than that, most everything is pretty simple. When I get my site up, I do plan on posting this info as an XML file for other programs/sites to use. Let me know if you're interested. I don't/won't charge for it but an acknowledgement would be nice!	[reply]
Re^3: Parsing data from a report not meant for machine readability. by Anonymous Monk on Jun 09, 2007 at 16:35 UTC
I guess I could have left my email (forgot I don't post in these forums!!!). You can write me at tim.trice@gmail.com. The website I'm building is personalhurricanecenter.com.	[reply]