Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re: LWP::UserAgent Alternatives for API data

by perlfan (Vicar)
on Jul 06, 2020 at 00:50 UTC ( #11118945=note: print w/replies, xml ) Need Help??


in reply to LWP::UserAgent Alternatives for API data

>but it will put a future date for the file date for some reason (July 27, 2020 for example), and then it does not update anymore because the file on the server is newer (but old data)

Before you switch your web "getter", make your script more resilient to this failure case. Clearly you can't trust the date in the data anymore, therefore you can't trust the data. Once you can inspect the data and detect "old data" (specific case of bad data), then you can retry. You may even wish to keep statistics on when this occurs and its duration. It's clear that their data producing code has its own issues, but that doesn't mean you just have to accept what they give you. Some simple QC in the loop will allow you to detect this and correct it (again via retries).

  • Comment on Re: LWP::UserAgent Alternatives for API data

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11118945]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (8)
As of 2020-11-24 12:16 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?