http://qs321.pair.com?node_id=727789


in reply to LWP slow downloads on windows

Simple things first. I think LWP pulls the response into memory. 23Mbyte isn't that much, but is there a chance that your windows box is working near the limit of it's RAM? (Check the 'commit change' in task manager - if its close to your physical RAM then you probably are close to paging).

If so, you could simply be seeing paging activity account for the difference.

Other than that, can you try other methods? Perhaps the network to your windows box is congested?

Does your web browser take a similar length of time to fetch the file?

It still could be LWP, but it would be useful if you could check the windows box is able to fetch the 23Mbyte quickly using other methods.

Replies are listed 'Best First'.
Re^2: LWP slow downloads on windows
by perreal (Monk) on Dec 03, 2008 at 20:14 UTC
    Also, you can compare the ping times and traceroute outputs.
[reply]
Re^2: LWP slow downloads on windows
by robobunny (Friar) on Dec 03, 2008 at 20:16 UTC
    Thanks for the response. I'm pretty sure it's LWP. Downloading the file manually with a browser is fast, and I've seen the same behavior on every Windows machine that I've tried. I've also seen it on a number of machines that I'm sure had plenty of free RAM, so I don't think it's paging. My Windows and Linux machines are plugged into the same switch as well, so there shouldn't be any network differences.

    It isn't that big of a deal in this case because the file isn't downloaded often (mostly just the first time a user runs the program), but it's been annoying today because I'm working on the download code and it's slowing testing down. The main reason I'd like to know what's going on is in case it becomes an issue in the future.
[reply]

      OK, can we get an idea what your windows box is up to during this 2 minutes?

      Is the CPU busy at all (I assume it's not busy before you start the test)?

      If the CPU is busy with another proc, that's your culprit. If the CPU is busy with your perl script, we can try CPU profiling with Devel::NYTProf or Devel::Profile, if (as is likely) your CPU is idle we can look at other causes:

      Perhaps you could try:

      1. take a network trace with wireshark.

        What you're looking at here is for significant pauses between the packets. 23 Mbyte is ~ 12000 1500-byte packets. Over 133 secs the gap we're looking for is about 133/12000 ~= 0.01 secs. If the pause is between the data packet going out and the ACK coming back, the problem is at the server (it's pausing before ACKing for some reason). If the pause is between an ACK coming back and data going out, the problem is local.

      2. Using LWP::UserAgent

        This allows you to set a callback (see the get function in the doc) which is called whenever data is available. You could add these to your stopwatch to see if rate is uniformly slow or slows down over time. Not sure what that would tell us, but you never know.

      Also, since we're in the realm of the unlikely here, maybe it is worth checking it's not the LWP version. LWP is pure-perl, so you should be able to copy the modules from one box to another and use that temp copy with 'perl -I /path/to/your/temp/copy'. Maybe you've found a regression.

      A quick grep over the current LWP source shows lots of references to $^O, but nothing which seems relevant.


      D'oh! I forgot the 1st rule of network delays: it's always DNS.

      I'll wager one xp point that your windows boxes don't have reverse DNS, and your web server is (mis-)configured to log the host name of connecting systems. It sits there in DNS timeout before sending back the data.

      If I'm right, you'll see no data at all during the timeout period, then the data shoot back in a short while. Can you 'dig -x 1.2.3.4' from your web server for the windows boxes and the unix boxes?

[reply]
[d/l]

In Section Seekers of Perl Wisdom