Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: polishing up a json fetching script for weather data

by marto (Cardinal)
on May 12, 2020 at 20:51 UTC ( #11116731=note: print w/replies, xml ) Need Help??


in reply to polishing up a json fetching script for weather data

As requested, a fairly rough and ready proof of concept using Mojo:

#!/usr/bin/perl use strict; use warnings; use feature 'say'; use Mojo::UserAgent; use DateTime::Format::Strptime; use open ':std', OUT => ':utf8'; my $url = 'https://api.weather.gov/stations/KPDX/observations/latest'; # the API docs says you must identify yourself, please make this somet +hing legit my $name = '(example.com, contact@example.com)'; my $ua = Mojo::UserAgent->new; $ua->transactor->name( $name ); # get JSON response my $json = $ua->get( $url )->res->json->{properties}; # for each cloud.... foreach my $cloud ( @{$json->{cloudLayers}} ){ say "$cloud->{amount}, $cloud->{base}{value}"; } # JD from JSON timestamp my $format = DateTime::Format::Strptime->new( pattern => '%FT%T%z'); my $dt = $format->parse_datetime( $json->{timestamp} ); say 'Julian Day: ' . $dt->jd; my $pturl = 'http://www.fourmilab.ch/cgi-bin/Yoursky?z=1&lat=45.5183& +ns=North&lon=122.676&ew=West'; # you wanted Julian date so it looks like date should be '2' from the +source. my $tx = $ua->post( $pturl => form => { utc => $dt->jd, date => '2' } +); my $sunrow = $tx->res->dom->at('center:nth-of-type(3) table tr:nth-of- +type(3)'); # output say 'Name:' . $sunrow->children->[0]->all_text; say 'Altitude: ' . $sunrow->children->[4]->text; say 'Azimuth: ' . $sunrow->children->[5]->text; say 'Visible: ' . $sunrow->children->[6]->text;

Output:

FEW, 760 SCT, 3350 BKN, 7620 Julian Day: 2458982.28680556 Name: Sun Altitude: 61.520 Azimuth: 21.289 Visible: Up

There may be some typos in there, chalk that one up to the time of day here. Let me know if there are any problems.

Update: for the 'Name' switch to all_text from text.

Replies are listed 'Best First'.
Re^2: polishing up a json fetching script for weather data
by Aldebaran (Deacon) on May 23, 2020 at 07:57 UTC
    As requested, a fairly rough and ready proof of concept using Mojo:

    Your script works perfectly, right out of the gate. I have tons of questions but don't want to muck up this thread by overreaching on this nice, neat result. I have been trying to imitate and extend it in other ways, but these results are slow to attain, but I don't want you to think I'm not trying:

    $ ls *mo* 1.1.mojo_api.pl 1.monk.tag.pl 2.3.mojo_json.pl 2.monk.tag.pl 1.1.mojo_json.pl 1.mo_pm.pl 2.4.mojo_json.pl 3.1.mojo_weathe +r.pl 1.2.mojo_api.pl 1.mo_pm.txt 2.5.mojo_json.pl 3.mojo_weather. +pl 1.mojo_json.pl 2.1.mojo_json.pl 2.mojo_json.pl 1.mojo_tree.pl 2.2.mojo_json.pl 2.mojo_weather.pl $

    Some of them didn't work for the strangest reasons. I had to get used to the strangeness in other cases. I will have run these hundreds of times of times before it dawned on me that the reason I never saw a value of rainfall other than undef is that no one on god's green earth has any idea what that value could be. (I wonder why it's there. A stubout for guesses?)

    I do want to pursue these ideas further, though, and have started another thread which generalizes this result: getting Sun info with Astro::Coord::ECI::Sun

    Thanks so much for lending your considerable expertise to the matter at hand....

      I don't understand the reason behind the various approaches rather than keep things clean and just sticking with the Mojo based solution for everything. I thought there would be a better, local way to get whatever data you are looking for, but I'm not an astronomer. If Astro::Coord::ECI::Sun does this then I don't see any problem in just calling that as many times as you require.

        I don't understand the reason behind the various approaches rather than keep things clean and just sticking with the Mojo based solution for everything.

        The approaches I was using are ones I understand. First with WWW::Mechanize, which I've used on that site for years. I haven't worked up the facility to do it with Mojo yet, despite many attempts. What happens is that I try to extend the script in a small way and can't quite get there. You said if I ran into the weeds again, let you know, so let's try this one, because it's where I was/am stuck with mojo.

        I'm trying to enter the site at the beginning and mechanize my way through to the table I need after the initial sky map has been made. I tried commands like this to tell me something about the page:

        $ mojo get https://www.fourmilab.ch/yoursky/ '*' attr id skylat skynsn skynss skylon skyewe skyeww horlat hornsn hornss horlon horewe horeww feedback $

        I see 2 of the fields I need to populate here, but I don't know how to deal with the button in the mojo context. In your successful upthread script, you seem not to use a click_button event to get it done, so I tried to squeak by the same way:

        $ ./3.mo_scrape.pl ./3.mo_scrape.pl 45.5 -122.4 0.346 Can't call method "children" on an undefined value at ./3.mo_scrape.pl + line 45. $ cat 3.mo_scrape.pl #!/usr/bin/env perl use Mojo::UserAgent; use 5.016; use warnings; use Mojo::UserAgent; use DateTime::Format::Strptime; use open ':std', OUT => ':utf8'; use Mojo::DOM; use Log::Log4perl; # get rid of old log my $file = '/home/hogan/Documents/hogan/logs/4.log4perl.txt'; unlink $file or warn "Could not unlink $file: $!"; my $log_conf4 = "/home/hogan/Documents/hogan/logs/conf_files/4.conf"; Log::Log4perl::init($log_conf4); #info my $logger = Log::Log4perl->get_logger(); $logger->info("$0"); ## important...no trailing zeroes else 301 error my $lat = 45.5; my $long = -122.4; my $elev = .346; #km $logger->info("$lat $long $elev"); my $guess = 2458960; #Earth day 2020 in julian days my $west_long = -$long; # the API docs says you must identify yourself, please make this somet +hing legit # my $name = '(example.com, contact@example.com)'; # can't I pretend to be a browser like any other? my $uaname = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like G +ecko) Chrome/40.0.2214.93 Safari/537.36'; my $ua = Mojo::UserAgent->new; $ua->transactor->name($uaname); my $pturl = 'https://www.fourmilab.ch/yoursky/'; my $tx = $ua->post( $pturl => form => { skylat => $lat, skylon => $west_long +} ); $tx = $ua->post( $pturl => form => { utc => $guess, date => '2' } ); my $sunrow = $tx->res->dom->at('center:nth-of-type(3) table tr:n./2th- +of-type(3)'); # output say 'Name:' . $sunrow->children->[0]->all_text; say 'Altitude: ' . $sunrow->children->[4]->text; say 'Azimuth: ' . $sunrow->children->[5]->text; say 'Visible: ' . $sunrow->children->[6]->text; $

        Q1) How do I lose this one? (I'd like to see the equivalent of $mech->uri) If I need the button it would have to look something like submit_button:

        %= submit_button 'Ok!', id => 'foo'

        Q2) What gives with the %= symbol on the left?

        I thought there would be a better, local way to get whatever data you are looking for, but I'm not an astronomer.

        I took Stellar Evolution my final semester in physics. I was doing alright until the final turned out to be a computer project that was to integrate a star given a certain composition (no idea what syntax). I was caught completely flatfooted and had somewhat of a meltdown, where I decided to continue my degree with mathematics instead. The failure has stung through the decades.

        If Astro::Coord::ECI::Sun does this then I don't see any problem in just calling that as many times as you require.

        Copy that.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11116731]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (2)
As of 2021-01-17 06:08 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?