Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: Speeding things up

by marto (Cardinal)
on Apr 15, 2018 at 09:12 UTC ( [id://1212913]=note: print w/replies, xml ) Need Help??


in reply to Speeding things up

"I haven't programmed in perl since the last century and things have moved on"

They sure have. You could consider using something like this (Mojo:::UserAgent/Mojo::DOM based). Also, you'll want to ensure you're not violating the terms or service/use. Sites the size/nature of Argos will likely automatically block/throttle you for such rapid fire requests.

Replies are listed 'Best First'.
Re^2: Speeding things up
by Anonymous Monk on Apr 15, 2018 at 13:14 UTC

    Thanks. I'll explore the Mojo thingy to minimise thumps.

    I'd hope that 100 hits a minute to a site that size wouldn't trigger and DDoS alarms but you're right, I should have a look at their robot policy.

      I find the mojo route quick, easy to code and maintain, selectors make this sort of thing trivial. Post a reply if you have problems, I'll take a look.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1212913]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others surveying the Monastery: (3)
As of 2024-03-28 13:57 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found