Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re: Uncompress streaming gzip on the fly in LWP::UserAgent/WWW::Mechanize

by haukex (Archbishop)
on Dec 20, 2018 at 12:36 UTC ( [id://1227519]=note: print w/replies, xml ) Need Help??


in reply to Solved: Uncompress streaming gzip on the fly in LWP::UserAgent/WWW::Mechanize

Since I don't have access to that site, I couldn't test at first, but I whipped up the following test server which I hope emulates the server you're accessing, and it seems that the solution provided by pmqs here works. The only issue I'm still having is that the last line is cut off, probably some interaction with WWW::Mechanize, I haven't checked it out yet. First the server (run e.g. via plackup):

use warnings; use strict; use IO::Compress::Gzip qw/$GzipError Z_PARTIAL_FLUSH/; my $app = sub { my $env = shift; die "This app needs a server that supports psgi.streaming" unless $env->{'psgi.streaming'}; die "The client did not send the 'Accept-Encoding: gzip' header" unless defined $env->{HTTP_ACCEPT_ENCODING} && $env->{HTTP_ACCEPT_ENCODING} =~ /\bgzip\b/; # Note some browsers don't correctly support gzip correctly, # see e.g. https://metacpan.org/pod/Plack::Middleware::Deflater # but we're not checking that here (and we don't set the Vary head +er) return sub { my $respond = shift; my $zipped; my $z = IO::Compress::Gzip->new(\$zipped) or die "IO::Compress::Gzip: $GzipError"; my $w = $respond->([ 200, [ 'Content-Type' => 'text/plain; charset=ascii', 'Content-Encoding' => 'gzip', ] ]); for (1..10) { $z->print("Hello, it is ".gmtime." GMT\n"); $z->flush(Z_PARTIAL_FLUSH); $w->write($zipped) if defined $zipped; $zipped = undef; sleep 1; } $z->print("Goodbye!\n"); $z->close; $w->write($zipped) if defined $zipped; $w->close; }; };

And the client:

use warnings; use strict; use Data::Dump; use WWW::Mechanize; use Compress::Zlib; my $gunzip = inflateInit(WindowBits => 16 + MAX_WBITS) or die "Cannot create a inflation stream\n"; my $mech = WWW::Mechanize->new(); $mech->add_handler( response_data => sub { my ( $response, $ua, $h, $data ) = @_; my ($buffer, $status) = $gunzip->inflate($data); die "zlib error: $status" if length $status; dd $buffer; 1; }); $mech->get('http://localhost:5000');

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1227519]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others pondering the Monastery: (2)
As of 2024-04-24 15:10 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found