http://qs321.pair.com?node_id=11118945


in reply to LWP::UserAgent Alternatives for API data

>but it will put a future date for the file date for some reason (July 27, 2020 for example), and then it does not update anymore because the file on the server is newer (but old data)

Before you switch your web "getter", make your script more resilient to this failure case. Clearly you can't trust the date in the data anymore, therefore you can't trust the data. Once you can inspect the data and detect "old data" (specific case of bad data), then you can retry. You may even wish to keep statistics on when this occurs and its duration. It's clear that their data producing code has its own issues, but that doesn't mean you just have to accept what they give you. Some simple QC in the loop will allow you to detect this and correct it (again via retries).

  • Comment on Re: LWP::UserAgent Alternatives for API data