LWP::UserAgent is extremely well established, so yes, you could just stick with that if you figure out the correct options*. Other than that, I'd recommend HTTP::Tiny, or, if you want to step it up a bit, Mojo::UserAgent, which has the advantage that it gives you easy access to JSON and XML/HTML parsing features via Mojo::JSON ($ua->get($url)->result->json) and Mojo::DOM ($ua->get($url)->result->dom).

* LWP is not my area of expertise, and you also don't say exactly in which way the requests are going wrong (SSCCE, How do I post a question effectively?). Tracing a good request and comparing with a bad one using Wireshark might be very useful here.

See polishing up a json fetching script for weather data, Super Search for similar discussion.

Before you switch your web "getter", make your script more resilient to this failure case. Clearly you can't trust the date in the data anymore, therefore you can't trust the data. Once you can inspect the data and detect "old data" (specific case of bad data), then you can retry. You may even wish to keep statistics on when this occurs and its duration. It's clear that their data producing code has its own issues, but that doesn't mean you just have to accept what they give you. Some simple QC in the loop will allow you to detect this and correct it (again via retries).

Thanks for the input.

Sticking with LWP::UserAgent for the time being, in hope that NWS solves all the issues sometime soon.

I have been using DarkSky for years and there was seldom any issues like this, but that is going away in the future (thanks Apple).

I reported the issues to NWS, and got added to a large support ticket with many other people having similar issues, which confirms it's not me. Have added Retry coding for all the 500's and 502's, and trying to figure out the best way to check and retry when old data is returned, but I'm not sure an immediate retry will help any. Adding some code downstream to not update to new data if out of date and use slightly older (cached) data instead.

It's a nightmare. Getting data for 30 cities is taking almost 2 minutes now from Tier-1 Data Center (files are only about 13KB each). I can get the same from DarkSky in seconds with essentially the same code. OpenWeather's API works for some things, but does not provide some of the forecast wording that DarkSky and NWS provide.

Not many good weather API's left, they all keep going away. AccuWeather is about twice what I am paying for DarkSky, that's my next option but trying to avoid paying that much if I can.

Again thanks for the help and wisdom!

Update from NWS: "Since an upgrade API isn't behaving optimally, the 503 errors are a known issue protecting the application. A fix for the application's database is expected no later than late this summer. We apologize for the inconvenience. This ticket will remain open until the issue is resolved. If anyone would like to be removed from the distribution list, please let us know via reply email. Otherwise, you will remain on the distribution list and updates will be provided when available, perhaps in several weeks. (CJJ)"

So the issues are not going to get fixed anytime soon, so AccuWeather API it is.

I think marto's reply to a similar question from me is relevant: Re: polishing up a json fetching script for weather data.

# the API docs says you must identify yourself, please make this somet
+hing legit
my $name = '(example.com, contact@example.com)';
my $ua = Mojo::UserAgent->new;
$ua->transactor->name( $name );

# get JSON response
my $json =  $ua->get( $url )->res->json->{properties};
[download]

I wonder if we're looking at the same API...are you shy to post your source?

I said it was the NWS API. Not hiding anything.

https://api.weather.gov/

I am working with 'Forecast' data. In the referenced Marto's reply post, they are trying to work with 'Latest Observations' which has even more issues than the Forecast data. I am sticking with the Latest Observations XML data from the NWS that comes from a different source than the NWS API. It has been much more reliable. Unfortunately they don't do the forecast data in XML like the latest observations, or I would use it instead right now.

The NWS API is newer but it has been out for a few years now. It should be much more stable than it is at this point.