good chemistry is complicated, and a little bit messy -LW |
|
PerlMonks |
Re: Are there any memory-efficient web scrapers?by tmaly (Monk) |
on Aug 15, 2011 at 13:51 UTC ( [id://920302]=note: print w/replies, xml ) | Need Help?? |
If you are scrapping a site that supports gzip encoding, you could set your agent to something like that of a Firefox Mozilla value. I have done this before and I do get gzip encoded responses that would cut down on your memory footprint.
In Section
Seekers of Perl Wisdom
|
|