Aldebaran has asked for the wisdom of the Perl Monks concerning the following question:

I've got a lot of irons in the fire with perl now, but I have one in particular that has been tickling in the back of my head for this pandemic. I've been trying to make an accounting of fomites, and there isn't a lot out there for rigorous data, but I started here. To excerpt:

Both viruses exhibited exponential decay in virus titer across all experimental conditions, as 41 indicated by linear decrease in the log 10 TCID 50 /mL over time (Figure 1B). HCoV-19 and SARS-CoV-1 42 exhibited similar half-lives in aerosols, with median estimates around 1.1-1.2 hours, and 95% credible 43 intervals of 0.64, 2.64 hours for HCoV-19 and 0.78 , 2.43 hours for SARS-CoV-1 (Figure 1C, Table 44 S1).

What's more, I live with a party who is immunocompromised, so it's super important that I don't bring this back to her as I use the door hardware, stair railings, and plumbing fixtures. We routinely wipe things down with products having ammonium chloride (Lysol and WinCo brands). One can't wipe down a textile, so I attempted to come up with a "program" for disinfecting and rotating things like masks and gloves. My actionable ideas I've captured with photos and comments on my website.

What I'm counting on is disinfecting using the sun, heat, and moving air, but how would I build a realistic model of how long I need to leave a pair of gloves on my dashboard? My thoughts return to bliako's post from about a year ago here: Re: bracketology was Re^2: making a markovian "mad lib".

If that was a "markovian process" then we could describe it simply by a "transition matrix" which is a convenient way to convey the information of what the probability of "next" state is given "current" state in the form of a 2d array often called a stochastic matrix:
rain sunny dry-cloudy rain 0.6 0.2 0.2 sunny 0.3 0.5 0.2 dry-cloudy 0.4 0.3 0.3
In the above array, row represents current state and column the next state, e.g. the probability of rain when now is raining is 0.6, the prob of dry-cloudy when now is sunny is 0.2 etc. The important property of the above matrix is that the probabilities of all the possible events from a current state must sum to 1: all rows sum to 1. Why? If we are now in current state of "rain" we have 3 possible outcomes. And so their probabilities must sum to one because one of them will happen with absolute certainty (as far as our model goes).

He continues:

The Graph or matrix can also be constructed by hand from imagination. Feed that information to your simulator in order to run the random process. That's probably how a computer game would calculate the weather in Mars.

Okay, then... Q1) How do I perform this simulation with perl with the above simple model?

And then, once we're able to do that with an elementary example, how do we make a stochastic matrix with the breadth of outcomes from NOAA:

$ ./2.icons.pl { "\@context" => [], "icons" => { bkn => { description => "Mostly cloudy" }, blizzard => { description => "Blizzard" }, cold => { description => "Cold" }, dust => { description => "Dust" }, few => { description => "A few clouds" }, fog => { description => "Fog/mist" }, fzra => { description => "Freezing rain" }, haze => { description => "Haze" }, hot => { description => "Hot" }, hurricane => { description => "Hurricane conditions" }, ovc => { description => "Overcast" }, rain => { description => "Rain" }, rain_fzra => { description => "Rain/freezing rain" }, rain_showers => { description => "Rain showers (high cloud cove +r)" }, rain_showers_hi => { description => "Rain showers (low cloud cover +)" }, rain_sleet => { description => "Rain/sleet" }, rain_snow => { description => "Rain/snow" }, sct => { description => "Partly cloudy" }, skc => { description => "Fair/clear" }, sleet => { description => "Sleet" }, smoke => { description => "Smoke" }, snow => { description => "Snow" }, snow_fzra => { description => "Freezing rain/snow" }, snow_sleet => { description => "Rain/sleet" }, tornado => { description => "Tornado" }, tropical_storm => { description => "Tropical storm conditions" }, tsra => { description => "Thunderstorm (high cloud cove +r)" }, tsra_hi => { description => "Thunderstorm (low cloud cover +)" }, tsra_sct => { description => "Thunderstorm (medium cloud co +ver)" }, wind_bkn => { description => "Mostly cloudy and windy" }, wind_few => { description => "A few clouds and windy" }, wind_ovc => { description => "Overcast and windy" }, wind_sct => { description => "Partly cloudy and windy" }, wind_skc => { description => "Fair/clear and windy" }, }, } ======================== Stochastic Matrix with side: hurricane rain_showers_hi few fzra rain_snow bkn snow_fzra snow snow_sleet wind_ovc blizzard cold smoke tropical_storm skc haze wind_few sct rain sleet ovc tsra tornado tsra_sct hot fog wind_bkn rain_fzra dust wind_sct tsra_hi rain_showers rain_sleet wind_skc ======================== URL = https://api.weather.gov/icons/land/day/tsra_sct,20/tsra_sct,40?s +ize=medium tsra_sct => 'Thunderstorm (medium cloud cover)' URL = https://api.weather.gov/icons/land/day/rain_showers,30/tsra_hi,3 +0?size=medium tsra_hi => 'Thunderstorm (low cloud cover)' URL = https://api.weather.gov/icons/land/night/rain_showers,30/rain_sh +owers?size=medium rain_showers => 'Rain showers (high cloud cover)' URL = https://api.weather.gov/icons/land/day/bkn?size=medium bkn => 'Mostly cloudy' $ cat 2.icons.pl #!/usr/bin/perl -w use strict; use LWP::UserAgent; use Data::Dump; use JSON::Parse 'parse_json'; use 5.016; my $ua = LWP::UserAgent->new( 'send_te' => '0' ); my $r = HTTP::Request->new( 'GET' => 'https://api.weather.gov/icons', [ 'Cache-Control' => 'max-age=0', 'Connection' => 'keep-alive', 'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*; +q=0.8', 'Accept-Encoding' => 'gzip, x-gzip, deflate, x-bzip2, bzip2', 'Accept-Language' => 'en-US,en;q=0.5', 'Host' => 'api.weather.gov:443', 'User-Agent' => 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firef +ox/74.0', 'Upgrade-Insecure-Requests' => '1', ], ); my $out; my $res = $ua->request( $r, ); if ( $res->is_success ) { my $json = $res->decoded_content; $out = parse_json $json; } else { print "Error: " . $res->status_line . "\n"; } dd $out; my %xlated_abbrev; #simple abbreviation table => description foreach my $key ( keys %{ $out->{icons} } ) #gen simple xlate table { $xlated_abbrev{$key} = $out->{icons}{$key}{description}; } my @keys2 = keys %xlated_abbrev; say "========================"; say "Stochastic Matrix with side: "; for (@keys2){ say "$_"; }; say "========================"; my @urls = ( 'https://api.weather.gov/icons/land/day/tsra_sct,20/tsra_sct,40?size +=medium', 'https://api.weather.gov/icons/land/day/rain_showers,30/tsra_hi,30?s +ize=medium', 'https://api.weather.gov/icons/land/night/rain_showers,30/rain_showe +rs?size=medium', 'https://api.weather.gov/icons/land/day/bkn?size=medium' ); foreach my $url (@urls) { my $last_path = ( split( '/', $url ) )[-1]; my ($abbrev_to_xlate) = $last_path =~ /^(\w+)/; print "URL = $url\n"; print " $abbrev_to_xlate => \'$xlated_abbrev{$abbrev_to_xlate}\'\n +\n"; } __END__ Created from curl command line curl 'https://api.weather.gov/icons' -H 'User-Agent: Mozilla/5.0 (X11; + Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0' -H 'Acce +pt: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp, +*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Conn +ection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Cache-Contr +ol: max-age=0' $

Thanks all for comments and know that not all Americans are as insouciant, impatient, and selfish in this crisis as our loud right-wing minority.

Replies are listed 'Best First'.
Re: using a stochastic matrix to simulate weather conditions
by bliako (Abbot) on Apr 23, 2020 at 20:44 UTC

    Aldebaran, here is something to get you started on collecting the numbers,

    #!/usr/bin/env perl use strict; use warnings; use DateTime::Format::Strptime; #use DateTime; use REST::Client; use Data::Roundtrip qw/:all/; my $lat = 31; my $long = 80; # this is our fetcher, similar to LWP::UserAgent # but better suited for this kind of web service: REST my $rest = REST::Client->new() or die "failed to construct client"; # see examples in # https://www.weather.gov/documentation/services-web-api # set the host $rest->setHost('https://api.weather.gov'); # and this is our query with lat,long specified above my $query = "/gridpoints/TOP/$lat,$long/forecast"; # make the request and check the response code, 200 is good my $response = $rest->GET($query) or die "failed to GET($query)"; if( $rest->responseCode() != 200 ){ die "failed to GET(".$rest->getHos +t()."/$query) with ".$rest->responseCode() } # we get back JSON my $jsonstr = $response->responseContent(); # convert JSON string to a perl variable my $pv = json2perl($jsonstr); if( ! defined $pv ){ die "something wrong with this alleged json : '$j +sonstr'" } # go to the interesting part my $forecasts = $pv->{'properties'}->{'periods'}; # or print it all and examine it # print perl2dump($pv); # we have some dates in the data in ISO8601 format # this is a parser to convert that date to a DateTime # object which we can query about things (like seconds-unix-epoch) my $dateparser = DateTime::Format::Strptime->new( # parses 2020-04-26T06:00:00-05:00, # %F then literal T then %T then timezone offset pattern => '%FT%T%z' ) or die "failed to DateTime::Format::Strptime->new()"; # we store each prediction in this hash, keyed on start-end dates # see below $key my %parsed; for my $aforecast (@$forecasts){ print "go: ".$aforecast->{'name'}."\n"; # extract various things from 1 prediction record # examine the record like : print perl2dump($aforecast); # like "This Afternoon" my $period_str = $aforecast->{'name'}; # start and end times of the prediction, convert them to objec +ts my $start_time_str = $aforecast->{'startTime'}; my $start_time_dateobj = $dateparser->parse_datetime($start_time_s +tr) or die "parsing date '$start_time_str'."; my $end_time_str = $aforecast->{'endTime'}; my $end_time_dateobj = $dateparser->parse_datetime($end_time_str) +or die "parsing date '$start_time_str'."; # sunny? my $forecast_str = $aforecast->{'shortForecast'}; # temperature as a number, see later for the units my $temp = $aforecast->{'temperature'}; # store this record/prediction in our %parsed hash # keyed on this: my $key = $start_time_str." to ".$end_time_str; $parsed{$key} = { 'date-human-str' => $period_str, # edit: added this 'date-from' => $start_time_str, 'date-to' => $end_time_str, 'date-span-hours' => ($end_time_dateobj->epoch()-$start_time_d +ateobj->epoch())/3600, 'date-from-epoch' => $start_time_dateobj->epoch(), 'date-to-epoch' => $end_time_dateobj->epoch(), 'forecast-string' => $forecast_str, # we append temp unit to the key, e.g. 'F' 'forecast-temp-'.$aforecast->{'temperatureUnit'} => $temp, } } print perl2dump(\%parsed);

    Edit: added comments to above

    You still need to figure out how to convert the received data to machine-readable format, re: "sunny", and, most importantly, figure out how to store received data for easy access over time (SQLite database?, csv files?, json files? perl variable files which you can later eval? - in order of my preference) - what search keys, what identifies a unique record? etc.

    Then you need to figure out how to create the sequence of "sunny"->"cloudy"->... given the time periods of the forecast etc.

    But it looks ;ole I missed something because you should be training your model with actual data and not forecasts. But that's trivial to correct hopefully, just change your urls.

    those who can do and those who can't blahblah, so Aldebaran++

    bw, bliako (who is not a medical doctor and has no opinion about the efficacy of said dissinfectation methods).

      here is something to get you started on collecting the numbers,

      thx, bliako, for this very useful script. I amaze at how quickly you can throw it together. You have just enough comments to teach me how this works as I work it. From the original, I had to dink around with the gridpoints by looking at the json from these urls in this form:

      # https://api.weather.gov/points/{latitude},{longitude} # https://api.weather.gov/points/45.4836356,-122.4170501

      , from which we glean the RHS of the url to be fetched:

      my $query = "gridpoints/PQR/120,99/forecast";

      I customized the program to output orderly to a log as well, in time sequence and added times in Pacific Time.

      #!/usr/bin/env perl use strict; use warnings; use DateTime::Format::Strptime; use REST::Client; use Data::Roundtrip qw/:all/; use 5.016; use Log::Log4perl; my $log_conf3 = "/home/hogan/Documents/hogan/logs/conf_files/3.conf"; my $log_conf4 = "/home/hogan/Documents/hogan/logs/conf_files/4.conf"; #Log::Log4perl::init($log_conf3); #debug Log::Log4perl::init($log_conf4); #info my $logger = Log::Log4perl->get_logger(); $logger->info($0); # this is our fetcher, similar to LWP::UserAgent # but better suited for this kind of web service: REST my $rest = REST::Client->new() or die "failed to construct client"; # see examples in # https://www.weather.gov/documentation/services-web-api # set the host $rest->setHost('https://api.weather.gov'); # https://api.weather.gov/points/{latitude},{longitude} # https://api.weather.gov/points/45.4836356,-122.4170501 # and this is our query with lat,long specified above #my $query = "gridpoints/TOP/$lat,$long/forecast"; my $query = "gridpoints/PQR/120,99/forecast"; # make the request and check the response code, 200 is good my $response = $rest->GET($query) or die "failed to GET($query)"; if ( $rest->responseCode() != 200 ) { die "failed to GET(" . $rest->getHost() . "/$query) with " . $rest->responseCode(); } # we get back JSON my $jsonstr = $response->responseContent(); # convert JSON string to a perl variable my $pv = json2perl($jsonstr); if ( !defined $pv ) { die "something wrong with this alleged json : '$jsonstr'"; } # go to the interesting part my $forecasts = $pv->{'properties'}->{'periods'}; # or print it all and examine it # print perl2dump($pv); # we have some dates in the data in ISO8601 format # this is a parser to convert that date to a DateTime # object which we can query about things (like seconds-unix-epoch) my $dateparser = DateTime::Format::Strptime->new( # parses 2020-04-26T06:00:00-05:00, # %F then literal T then %T then timezone offset pattern => '%FT%T%z' ) or die "failed to DateTime::Format::Strptime->new()"; # we store each prediction in this hash, keyed on start-end dates # see below $key ### hash no longer at this scope ### make an array container my @ordered; my $refordered = \@ordered; my $index = 0; for my $aforecast (@$forecasts) { print "go: " . $aforecast->{'name'} . "\n"; # extract various things from 1 prediction record # examine the record like : print perl2dump($aforecast); # like "This Afternoon" #print perl2dump($aforecast); my $period_str = $aforecast->{'name'}; # start and end times of the prediction, convert them to objects my $start_time_str = $aforecast->{'startTime'}; say "start is $start_time_str"; my $start_time_dateobj = $dateparser->parse_datetime($start_time_str +) or die "parsing date '$start_time_str'."; $start_time_dateobj->set_time_zone('America/Los_Angeles'); my $st = $start_time_dateobj->strftime('%Y-%m-%d-%H:%M +:%S %Z'); my $end_time_str = $aforecast->{'endTime'}; my $end_time_dateobj = $dateparser->parse_datetime($end_time_str) or die "parsing date '$start_time_str'."; $end_time_dateobj->set_time_zone('America/Los_Angeles'); my $et = $end_time_dateobj->strftime('%Y-%m-%d-%H:%M:%S %Z'); # sunny? my $forecast_str = $aforecast->{'shortForecast'}; # temperature as a number, see later for the units my $temp = $aforecast->{'temperature'}; # new scope for hash my %parsed; # store this record/prediction in our %parsed hash # keyed on this: my $key = $start_time_str . " to " . $end_time_str; $parsed{$key} = { 'date-human-str' => $period_str, # edit: added this 'date-from' => $start_time_str, 'date-from-local' => $st, 'date-to' => $end_time_str, 'date-to-local' => $et, 'date-span-hours' => ( $end_time_dateobj->epoch() - $start_time_dateobj->epoch() ) / +3600, 'date-from-epoch' => $start_time_dateobj->epoch(), 'date-to-epoch' => $end_time_dateobj->epoch(), 'forecast-string' => $forecast_str, # we append temp unit to the key, e.g. 'F' 'forecast-temp-' . $aforecast->{'temperatureUnit'} => $temp, }; my $refparsed = \%parsed; $ordered[$index] = $refparsed; $logger->info( perl2dump $refparsed); ++$index; } __END__

      Log excerpt:

      2020/05/01 08:59:21 INFO ./1.6.weather.pl 2020/05/01 08:59:21 INFO $VAR1 = { '2020-05-01T08:00:00-07:00 to 2020-05-01T18:00:00-07:00' => { 'date-from-epoch' => 1588345200, 'date-span-hours' => '10', 'date-to-local' => '2020-05-01-18:00:00 PDT', 'date-to-epoch' => 1588381200, 'date-to' => '2020-05-01T18:00:00-07:00', 'date-from-local' => '2020-05-01-08:00:00 PDT', 'date-from' => '2020-05-01T08:00:00-07:00', 'forecast-string' => 'Chance Rain Showers', 'date-human-str' => 'Today', 'forecast-temp-F' => 65 } }; 2020/05/01 08:59:21 INFO $VAR1 = { '2020-05-01T18:00:00-07:00 to 2020-05-02T06:00:00-07:00' => { 'date-from' => '2020-05-01T18:00:00-07:00', 'date-from-local' => '2020-05-01-18:00:00 PDT', 'date-to' => '2020-05-02T06:00:00-07:00', 'date-to-epoch' => 1588424400, 'date-to-local' => '2020-05-02-06:00:00 PDT', 'date-span-hours' => '12', 'date-from-epoch' => 1588381200, 'forecast-temp-F' => 50, 'date-human-str' => 'Tonight', 'forecast-string' => 'Rain Showers' } }; 2020/05/01 08:59:21 INFO $VAR1 = { '2020-05-02T06:00:00-07:00 to 2020-05-02T18:00:00-07:00' => { 'date-from-local' => '2020-05-02-06:00:00 PDT', 'date-to' => '2020-05-02T18:00:00-07:00', 'date-from' => '2020-05-02T06:00:00-07:00', 'date-span-hours' => '12', 'date-to-local' => '2020-05-02-18:00:00 PDT', 'date-from-epoch' => 1588424400, 'date-to-epoch' => 1588467600, 'forecast-temp-F' => 60, 'forecast-string' => 'Rain Showers', 'date-human-str' => 'Saturday' } };
      You still need to figure out how to convert the received data to machine-readable format, re: "sunny", and, most importantly, figure out how to store received data for easy access over time (SQLite database?, csv files?, json files? perl variable files which you can later eval? - in order of my preference) - what search keys, what identifies a unique record? etc.

      When I get to the part where I'm keeping count for real, I think a SQLite database is the right tool.

      Then you need to figure out how to create the sequence of "sunny"->"cloudy"->... given the time periods of the forecast etc.

      You know, it takes me all week to figure out a script that is written at a level that stretches my vistas, and then figure how I'm going to "make it my own," maybe extend it, and how I'm going to write it up. During this week and change, it has happened several times that I've been in situations where it was sunny and raining. I think we have to start somewhere other than where I suggested in the original post.

      But it looks ;ole I missed something because you should be training your model with actual data and not forecasts. But that's trivial to correct hopefully, just change your urls.

      That's just it. I've been spending as much time and effort as I can to find actual data, but the most I see is GUI representations as opposed to JSON. I'm also realizing that conditions at Troutdale Airport might be completely different than here on Hogan Butte. Troutdale is right in the mouth of the Columbia Gorge, which is a very interesting formation for the way it channels wind, for example.

      Q2 Can anybody see a way to get current conditions in JSON form?

      I found more reading re sunlight v cvirus here: a bbc article.

      Thanks all for comments, but especially:

      for (@responses){ [bliako]++; }
Re: using a stochastic matrix to simulate weather conditions
by bliako (Abbot) on Apr 23, 2020 at 22:22 UTC

    regarding crunching the numbers, I am not sure a Markov process can help you. If you want to have a prediction, then the NOAA will be more able to predict the weather I am sure. If you want to measure the energy fallen onto the gloves - locally, the maximum temperature reached etc. then some real-time temperature recording station is needed. There are some Monks who can dispense good advice on that.

    In any event, if you do insist to use the MC to make a prediction, then arrange an input file with your observations like below and use the code I posted at n-dimensional statistical analysis of DNA sequences (or text, or ...) (in the meantime there are some un-publish revisions so if you are interested I can post it again or point you at a repository).

    cat input.txt sun clo clo sun clo clo sun sun sun clo sun clo

    that's like sunny, cloudy etc. use other symbols but the separator must be space (' '), then analyse the input:

    analyse_text.pl --input-corpus input.txt --ngram-length 2 --output-sta +te mystate --separator ' '

    results in this, probabilities of a state given a previous state:

    { "counts" => { "clo|clo" => 2, "clo|sun" => 3, "sun|clo" => 4, "sun|s +un" => 2 }, "cum-twisted-dist" => { clo => ["clo", 0.4, "sun", 1], sun => ["clo", 0.666666666666667, "sun", 1], }, "dist" => { "clo|clo" => 0.181818181818182, "clo|sun" => 0.272727272727273, "sun|clo" => 0.363636363636364, "sun|sun" => 0.181818181818182, }, "N" => 2, }

    and do a 30-time-unit prediction:

    predict_text.pl --input-state mystate --separator ' ' --num-iteration +s 30 --seed 'clo'

    results in this:

    /home/andreas/usr/bin/predict_text.pl : starting with seed 'clo' ... clo sun clo clo sun clo sun clo clo clo clo sun sun sun clo clo sun clo sun clo sun sun clo sun sun sun clo clo sun clo sun sun sun clo sun clo sun clo clo sun clo clo clo sun clo clo sun sun sun clo clo sun clo clo sun clo sun clo clo clo clo sun sun clo sun sun clo sun clo sun sun clo sun sun clo sun sun clo sun clo sun sun clo sun clo sun sun sun sun clo sun clo clo sun clo clo clo sun sun clo sun

    But what's the meaning of this? good question!

Re: using a stochastic matrix to simulate weather conditions
by rizzo (Curate) on Apr 23, 2020 at 22:19 UTC

    Hi Aldebaran,

    a far less sophisticated and elegant, but probably more reliable approach could be the use of the water cooling system for disinfection, given it can be accessed more or less easily. The temperature is not only predictable, it is measurable and will be around 90°C at normal operation.
    As I've learned lately, 70°C for 30 minutes "kills" covid19 reliably. So if the outer parts of the water cooling system are above that, it could probably work.

    Drawback:
    • presumably tortous to implement
    • no Perl ;-)