Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling

Re^2: LWP slow downloads on windows (50 times faster?)

by robobunny (Friar)
on Dec 04, 2008 at 16:54 UTC ( #728027=note: print w/replies, xml ) Need Help??

in reply to Re: LWP slow downloads on windows (50 times faster?)
in thread LWP slow downloads on windows

I saw roughly the same speed differences as everyone else, both using the external URL in your script and using the internal file that I started with. After running a profiler on LWP, it looks like most of the time is spent appending the new chunk it just downloaded to the existing buffer (HTTP::Message, add_content method, line 142 in version 5.814):
$self->{_content} .= $$chunkref;
add_content is called in a loop by the collect method of LWP::Protocol. I was able to reduce the download time for a 12MB file from 36 seconds to 5 seconds by changing the collect method to use an array for the buffer and then stick everything together at the end, instead of calling add_content repeatedly:
my @total_content; if (!defined($arg) || !$response->is_success) { # scalar while ($content = &$collector, length $$content) { if ($parser) { $parser->parse($$content) or undef($parser); } LWP::Debug::debug("read " . length($$content) . " bytes"); push(@total_content, $$content); $content_size += length($$content); $ua->progress(($length ? ($content_size / $length) : "tick"), +$response); if (defined($max_size) && $content_size > $max_size) { LWP::Debug::debug("Aborting because size limit exceeded"); $response->push_header("Client-Aborted", "max_size"); last; } } $response->add_content(join('', @total_content));
Anybody know any reason that would be a problem?

Replies are listed 'Best First'.
Re^3: LWP slow downloads on windows (50 times faster?)
by BrowserUk (Patriarch) on Dec 04, 2008 at 17:52 UTC
Re^3: LWP slow downloads on windows (50 times faster?)
by mr_mischief (Monsignor) on Dec 05, 2008 at 22:39 UTC
    Your changes indeed speed things up on my testing system so long as they don't cause additional swapping. I see only one issue. It does seem to use more memory than the stock LWP::Protocol.

    Yours hit more heavily into my swap file for a large file (703MB on a system with only 512 MB of RAM) than the stock version. The map in BrowserUk's version put my poor antiquated testing server right out of physical RAM and swap completely.

    My conclusion is that I really do need more RAM in this machine. It's something to consider if you're under similar constraints or if you're submitting a patch to the module's maintainers, though. As always, paging to swap can erase any gains you'd get otherwise and then cause even much slower execution. Sometimes optimizing for speed at the expense of memory fails, and sometimes optimizing for memory use is actually a performance win.

Re^3: LWP slow downloads on windows (50 times faster?)
by kszyska (Initiate) on Apr 05, 2009 at 23:50 UTC
    This helped greatly speeding up zip downloads when I added this fix to the collect method in LWP::Protocal under perl 5.825. Before the fix it took ~3.5h to download 28 zip files and after adding this LWP::Protocal fix it took only ~1.5h :-)

    However, now I have perl 5.8.9 (with latest libwww-perl 5.825 2009-02-16) installed *and* the zip downloads are back to taking ~3.5 hours.

    I diffed the files but they are redical different (between libwww-perl 5.825 2009 and,v 1.41 2003/10/23 19:11:32) and I don't see any hooks for a similar quick fix.

    Can anyone suggest a fix for speeding up downloads using libwww-perl (5.825 2009-02-16)...?

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://728027]
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others pondering the Monastery: (4)
As of 2023-12-02 05:47 GMT
Find Nodes?
    Voting Booth?
    What's your preferred 'use VERSION' for new CPAN modules in 2023?

    Results (13 votes). Check out past polls.