Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

LWP::UserAgent adding unwanted TE header

by Anonymous Monk
on Jul 05, 2006 at 10:24 UTC ( #559287=perlquestion: print w/replies, xml ) Need Help??

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I'm a newbie to LWP and trying to get my HTTP headers to properly emulate my browser. However, LWP keeps adding a TE header which I want to get rid of. So far I have:
#!/usr/bin/perl use warnings; use strict; use LWP::UserAgent; # this page just prints the headers sent by the client my $url = 'http://localhost/go.php'; # initialize browser and set user agent my $browser = LWP::UserAgent->new(); $browser->agent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)'); # create new http request, clear headers and add a new header my $request = HTTP::Request->new(GET => $url); $request->clear; $request->header('Connection' => 'keep-alive'); # print the http headers before making the request my $headers = $request->headers_as_string; print "Headers before http request are:\n"; print "$headers"; # send the http request and print the headers echoed back by $url my $response = $browser->request($request); my $contents = $response->decoded_content; print "Returned headers are:\n"; print "$contents";
The output is:
Headers before http request are: Connection: keep-alive User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Returned headers are: Connection: keep-alive, TE, close Host: localhost TE: deflate,gzip;q=0.3 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Why is LWP adding the TE header and the extra parts to the connection header? How can I stop this behaviour?

Replies are listed 'Best First'.
Re: LWP::UserAgent adding unwanted TE header
by Corion (Pope) on Oct 11, 2019 at 15:56 UTC
Re: LWP::UserAgent adding unwanted TE header
by jdtoronto (Prior) on Jul 05, 2006 at 15:06 UTC
    The TE header is used to tell the server that it can, if it wishes, use one of the methods specified in the TE header to compress content. If your browser was able to use these methods to handle compressed data it would send the info in the request header! There was much discussion about this subject a couple of years ago in relation to Net::HTTP and Net::HTTP::Methods which also touched on LWP::UserAgent.

    jdtoronto

Re: LWP::UserAgent adding unwanted TE header
by shmem (Chancellor) on Jul 05, 2006 at 10:57 UTC
    These are not headers LWP adds deliberately. These are headers returned from the remote server according to HTTP/1.1 <-- that's bull.

    TE headers are client headers (see RFC2616 for the full specification). I confused them with Transfer-encoding ...

    Uhm, let's see...

    --shmem

    _($_=" "x(1<<5)."?\n".q/)Oo.  G\        /
                                  /\_/(q    /
    ----------------------------  \__(m.====.(_("always off the crowd"))."
    ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
      Thanks for your reply. So the TE stuff is actually being added by the server because the message is HTTP/1.1? And the headers being sent by LWP::UserAgent client are actually correct? I am using the PHP function apache_request_headers() on a page on localhost to echo back the request headers.

        Uhm, no. It's not a server header (it's good to know where an RFC is, but to read it actually is far better :-) - it's a client header, and yes, the header is added by LWP because the request is HTTP/1.1. The server header would be Transfer-encoding, it would seem.

        The deflate,gzip value gets added if the Zlib library is installed. I can't see that this is somehow cofigurable for now, it's buried in LWP/Protocol/http (in _new_socket):

        local($^W) = 0; # IO::Socket::INET can be noisy my $sock = $self->socket_class->new(PeerAddr => $host, PeerPort => $port, Proto => 'tcp', Timeout => $timeout, KeepAlive => !!$conn_cache, SendTE => 1, $self->_extra_sock_opts($host, + $port), );
        Setting SendTE to 0 disables TE headers. The above config pseudo hash is passed to Net::HTTP->new it seems. If you get at the NET::HTTP object used by LWP::UserAgent, maybe you can disable sending TE headers..

        --shmem

        _($_=" "x(1<<5)."?\n".q/)Oo.  G\        /
                                      /\_/(q    /
        ----------------------------  \__(m.====.(_("always off the crowd"))."
        ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://559287]
Approved by Hue-Bond
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (5)
As of 2020-11-27 06:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?