I saw roughly the same speed differences as everyone else, both using the external URL in your script and using the internal file that I started with. After running a profiler on LWP, it looks like most of the time is spent appending the new chunk it just downloaded to the existing buffer (HTTP::Message, add_content method, line 142 in version 5.814):
$self->{_content} .= $$chunkref;
add_content is called in a loop by the collect method of LWP::Protocol. I was able to reduce the download time for a 12MB file from 36 seconds to 5 seconds by changing the collect method to use an array for the buffer and then stick everything together at the end, instead of calling add_content repeatedly:
my @total_content;
if (!defined($arg) || !$response->is_success) {
# scalar
while ($content = &$collector, length $$content) {
if ($parser) {
$parser->parse($$content) or undef($parser);
}
LWP::Debug::debug("read " . length($$content) . " bytes");
push(@total_content, $$content);
$content_size += length($$content);
$ua->progress(($length ? ($content_size / $length) : "tick"),
+$response);
if (defined($max_size) && $content_size > $max_size) {
LWP::Debug::debug("Aborting because size limit exceeded");
$response->push_header("Client-Aborted", "max_size");
last;
}
}
$response->add_content(join('', @total_content));
Anybody know any reason that would be a problem?
