Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re: creating a webpage to fetch/calculate weather data

by Util (Priest)
on Jun 21, 2020 at 03:21 UTC ( #11118299=note: print w/replies, xml ) Need Help??


in reply to creating a webpage to fetch/calculate weather data

1. One of the Atlanta.pm members was raving at the last meeting about how useful Mojo is for web munging (scraping, parsing, and re-publishing as an internal "API"). I have no personal experience with Mojo, but I trust his ravings.

2. You can get elevation from lat+long from National Weather Service (NOAA)
or The National Map Elevation Point Query Service (USGS).

USGS has high granularity and accuracy:

$ curl -sS 'https://nationalmap.gov/epqs/pqs.php?units=Meters&output=j +son&x=-83.65&y=41.37' | jq '.USGS_Elevation_Point_Query_Service.Elevation_Query.Elevation' 212.77
NOAA has the elevation of the 2.5km grid polygon, which will be lower granularity, but might (I'm just guessing) be helpful for water, where the elevation varies:
$ curl -LsS 'https://api.weather.gov/points/41.37,-83.65' | jq '.properties.forecastGridData' "https://api.weather.gov/gridpoints/CLE/16,52" $ curl -LsS 'https://api.weather.gov/gridpoints/CLE/16,52' | jq '.properties.elevation.value' 213.0552

Replies are listed 'Best First'.
Re^2: creating a webpage to fetch/calculate weather data
by Aldebaran (Curate) on Jun 23, 2020 at 22:58 UTC
    1. One of the Atlanta.pm members was raving at the last meeting about how useful Mojo is for web munging (scraping, parsing, and re-publishing as an internal "API"). I have no personal experience with Mojo, but I trust his ravings.

    Then Mojo it is. Thanks so much for your post, Util, it helped me get farther in a lot of ways. I was able to get this to work on the command line, but I couldn't get Corion's curl converter to recognize a -S option, nor could I see it from man curl. The command seems to give the same output without it:

    $ curl -s 'https://nationalmap.gov/epqs/pqs.php?units=Meters&output=js +on&x=-83.65&y=41.37' | jq '.USGS_Elevation_Point_Query_Service.Elevat +ion_Query.Elevation' 212.77 $

    Unfortunately, I couldn't get this value in the perl script which I'm posting below. (Partial credit is great for me.)

    NOAA has the elevation of the 2.5km grid polygon, which will be lower granularity, but might (I'm just guessing) be helpful for water, where the elevation varies:

    I was thrilled to see this solution at the same server where I had been fishing! Furthermore, your example along with the jq command to see where the good data is in the haystack was exactly what I needed to get some numbers out of this. First I'll post the log then the source.

    2020/06/23 15:26:07 INFO ./5.2.elev.pl 2020/06/23 15:26:07 INFO { "USGS_Elevation_Point_Query_Service" => { "Elevation_Query" => { "Data_Source" => "3DEP 1/3 arc-second", "Elevation" => -1000000, "Units" => "Meters", "x" => "41.37", "y" => "-83.65" } } } 2020/06/23 15:26:07 INFO 41.37 -83.65 0.208 2020/06/23 15:26:07 INFO ============== 2020/06/23 15:26:07 INFO https://api.weather.gov/points/41.37,-83.65 2020/06/23 15:26:08 INFO CLE 2020/06/23 15:26:08 INFO 16 52 2020/06/23 15:26:08 INFO ============== 2020/06/23 15:26:08 INFO https://api.weather.gov/gridpoints/CLE/16,52 2020/06/23 15:26:08 INFO ============== 2020/06/23 15:26:08 INFO 213.0552 wmoUnit:m

    Source:

    #!/usr/bin/env perl use strict; use warnings; use 5.016; use Log::Log4perl; use Mojo::UserAgent; use open ':std', OUT => ':utf8'; use Mojo::Util qw(dumper); # get rid of old log my $file = '/home/hogan/Documents/hogan/logs/4.log4perl.txt'; unlink $file or warn "Could not unlink $file: $!"; my $log_conf4 = "/home/hogan/Documents/hogan/logs/conf_files/4.conf"; Log::Log4perl::init($log_conf4); #info my $logger = Log::Log4perl->get_logger(); $logger->info("$0"); # weston, OH 41.37,-83.65 my $lat = 41.37; my $long = -83.65; # the API docs says you must identify yourself, please make this somet +hing legit my $name = '(example.com, contact@example.com)'; my $url2 = "https://nationalmap.gov/epqs/pqs.php?output=json&units=Meters&x=$lat& +y=$long"; my $ub = Mojo::UserAgent->new; $ub->transactor->name($name); # get JSON response my $qv = $ub->get($url2)->res->json; $logger->info( dumper $qv); my $elev = .208; # 676 ft = .206 +.02 for observer #km $logger->info("$lat $long $elev"); my $url1 = "https://api.weather.gov/points/$lat,$long"; $logger->info("=============="); $logger->info($url1); my $ua = Mojo::UserAgent->new; $ua->transactor->name($name); # get JSON response my $pv = $ua->get($url1)->res->json->{properties}; my $station = $pv->{'cwa'}; my $gridX = $pv->{'gridX'}; my $gridY = $pv->{'gridY'}; $logger->info("$station"); $logger->info("$gridX $gridY"); $logger->info("=============="); ## new transaction ## find elevation my $uc = Mojo::UserAgent->new; $uc->transactor->name($name); my $url3 = "https://api.weather.gov/gridpoints/$station/$gridX,$gridY" +; $logger->info( $url3 ); my $json = $uc->get($url3)->res->json; #my $props = $json->{properties}; my $elev2 = $json->{properties}->{elevation}->{value}; my $unit = $json->{properties}->{elevation}->{unitCode}; $logger->info("=============="); #$logger->info(dumper($props)); $logger->info( $elev2 . ' '. $unit); __END__

    It takes me time fiddling with these things with perl and jq in order to get my head around them.

    Q1) How do I get a valid value for the usgs query in lexical perl?

    Thanks again for your post,

      > recognize a -S option, nor could I see it from man curl.

      It seems to be a recent addition. See -S in man curl.

      -S, --show-error
      When used with -s, --silent, it makes curl show an error message if it fails.
      -s, --silent
      Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute. It will still output the data you ask for, potentially even to the terminal/stdout unless you redirect it.

      I doubt this option influences what gets downloaded.

      map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
Re^2: creating a webpage to fetch/calculate weather data
by Aldebaran (Curate) on Jun 25, 2020 at 01:57 UTC

    Big day for perl and you, Util; I thought your presentation was awesome, but I could not ask any questions, as the only way I could hear anything was going to the livestream with youtube.

    Then I hauled out an older machine with a newer ubuntu and now can hear the zoom proceedings. Maybe others had similar problems or not...I could be quite smart and be the local dunce in such matters.

    I've always been interested in the weather and trees of every variety. bliako has posted several examples recently that are fascinating, but I had forgotten about HTML::TreeBuilder, which I used to use quite frequently.

    You made a good case for curl'ing once:

    $ curl -o 1.weather.html 'https://www.wunderground.com/weather/us/or/p +ortland' % Total % Received % Xferd Average Speed Time Time Time + Current Dload Upload Total Spent Left + Speed 100 847k 0 847k 0 0 487k 0 --:--:-- 0:00:01 --:--: +-- 487k $

    I find your script quite elegant and readable:

    $ ./1.wund_wind.pl N5 $ cat 1.wund_wind.pl #!/usr/bin/env perl use 5.016; use HTML::TreeBuilder; my $tree = HTML::TreeBuilder->new_from_file('1.weather.html') or die; my $wind = $tree->look_down( _tag => 'div', class => qr/^condition-wind /, ); say $wind->as_trimmed_text( extra_chars => '\xA0' ); $

    Q1) What other fields are worth looking at at wunderground?

    Q2) Any idea what the KORPORTL966 part of this url signfies?

    https://www.wunderground.com/hourly/us/or/portland/KORPORTL966

    (This has nothing to do with the parent response.) Q3) Why can't I crack this with Mojo?

    #!/usr/bin/env perl use Mojo::UserAgent; use 5.016; use warnings; # Fetch website my $uaname = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 ( +KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36'; my $ua = Mojo::UserAgent->new; my $res = $ua->get('1.weather.html')->result; # Visit all nodes recursively to extract more than just text for my $n ($res->dom->descendant_nodes->each) { # Text or CDATA node print $n->content; # Also include alternate text for images print $n->{alt} if $n->type eq 'tag' && $n->tag eq 'img'; }
    $ ./1.mo_tree.pl Can't connect: Name or service not known at ./1.mo_tree.pl line 9. $

    Thanks everybody for presentations on a successful first day. As if covid could keep perl down....

      "Why can't I crack this with Mojo?"

      Alternatively just get the URL using Mojo::UserAgent, in exactly the same way as all of the other Mojo based examples you've been provided with.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11118299]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (1)
As of 2021-10-21 04:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My first memorable Perl project was:







    Results (82 votes). Check out past polls.

    Notices?