Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Re^2: LWP slow downloads on windows

by robobunny (Friar)
on Dec 03, 2008 at 20:16 UTC ( #727792=note: print w/replies, xml ) Need Help??

in reply to Re: LWP slow downloads on windows
in thread LWP slow downloads on windows

Thanks for the response. I'm pretty sure it's LWP. Downloading the file manually with a browser is fast, and I've seen the same behavior on every Windows machine that I've tried. I've also seen it on a number of machines that I'm sure had plenty of free RAM, so I don't think it's paging. My Windows and Linux machines are plugged into the same switch as well, so there shouldn't be any network differences.

It isn't that big of a deal in this case because the file isn't downloaded often (mostly just the first time a user runs the program), but it's been annoying today because I'm working on the download code and it's slowing testing down. The main reason I'd like to know what's going on is in case it becomes an issue in the future.

Replies are listed 'Best First'.
Re^3: LWP slow downloads on windows
by jbert (Priest) on Dec 03, 2008 at 21:41 UTC

    OK, can we get an idea what your windows box is up to during this 2 minutes?

    Is the CPU busy at all (I assume it's not busy before you start the test)?

    If the CPU is busy with another proc, that's your culprit. If the CPU is busy with your perl script, we can try CPU profiling with Devel::NYTProf or Devel::Profile, if (as is likely) your CPU is idle we can look at other causes:

    Perhaps you could try:

    1. take a network trace with wireshark.

      What you're looking at here is for significant pauses between the packets. 23 Mbyte is ~ 12000 1500-byte packets. Over 133 secs the gap we're looking for is about 133/12000 ~= 0.01 secs. If the pause is between the data packet going out and the ACK coming back, the problem is at the server (it's pausing before ACKing for some reason). If the pause is between an ACK coming back and data going out, the problem is local.

    2. Using LWP::UserAgent

      This allows you to set a callback (see the get function in the doc) which is called whenever data is available. You could add these to your stopwatch to see if rate is uniformly slow or slows down over time. Not sure what that would tell us, but you never know.

    Also, since we're in the realm of the unlikely here, maybe it is worth checking it's not the LWP version. LWP is pure-perl, so you should be able to copy the modules from one box to another and use that temp copy with 'perl -I /path/to/your/temp/copy'. Maybe you've found a regression.

    A quick grep over the current LWP source shows lots of references to $^O, but nothing which seems relevant.

    D'oh! I forgot the 1st rule of network delays: it's always DNS.

    I'll wager one xp point that your windows boxes don't have reverse DNS, and your web server is (mis-)configured to log the host name of connecting systems. It sits there in DNS timeout before sending back the data.

    If I'm right, you'll see no data at all during the timeout period, then the data shoot back in a short while. Can you 'dig -x' from your web server for the windows boxes and the unix boxes?

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://727792]
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others cooling their heels in the Monastery: (4)
As of 2023-12-10 07:48 GMT
Find Nodes?
    Voting Booth?
    What's your preferred 'use VERSION' for new CPAN modules in 2023?

    Results (39 votes). Check out past polls.