Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: Are there any memory-efficient web scrapers?

by tmaly (Monk)
on Aug 15, 2011 at 13:51 UTC ( [id://920302]=note: print w/replies, xml ) Need Help??


in reply to Are there any memory-efficient web scrapers?

If you are scrapping a site that supports gzip encoding, you could set your agent to something like that of a Firefox Mozilla value. I have done this before and I do get gzip encoded responses that would cut down on your memory footprint.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://920302]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (4)
As of 2024-03-29 01:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found