IO::Zlib with XML::Twig/XML::Parser::Expat

athomason has asked for the wisdom of the Perl Monks concerning the following question:

'ello folks,

I'm currently handling some XML documents that are too large to process in memory or to store on disk permanently. However, they fit well enough when gzip'ed (at about 15:1 compression). But when it comes time to process them, I would like to avoid first decompressing them fully. I thought I could solve the problem using IO::Zlib, which provides an interface much like IO::Handle; this would allow me to keep only portions of the decompressed text in memory at a time. And of course, XML::Twig is great for managing big XML documents (thanks mirod!), but it doesn't natively handle gzip'ed XML. But since XML::Parser::Expat and by extension, XML::Twig can take an IO::Handle as a document source, I thought I could string the two together. However, IO::Zlib doesn't actually inherit from IO::Handle, and XML::Parser::Expat demands that UNIVERSAL::isa($arg, 'IO::Handle') be true before it will treat the argument as a handle. I figured a simple workaround like this would work:

package IO::Handle::Zlib;

use vars qw/ @ISA /;
@ISA = qw/ IO::Zlib IO::Handle /;
[download]

which would allow me to replace my IO::Zlib objects with IO::Handle::Zlib's transparently. However, when I try this out, I come across the following error, courtesy of expat:

not well-formed (invalid token) at line 7213, column 3, byte 780490 at
+ 
  /path/to/perl/lib/5.6.1/IP27-irix/XML/Parser.pm line 185
[download]

Now that's odd, since the decompressed file ends at line 7212, and is only 780487 bytes long. One might think the file is being decompressed past the original size of the document, but inserting print DUMP <$gz>; gives a file that is identical to the original (i.e., the angle-bracket read gives a file that is also 7212 lines and 780487 bytes long). So clearly, whatever the XS part of XML::Parser::Expat is doing with the IO::Handle is not what the angle-brackets are doing. And expat itself is working, since replacing

    my $reader = new IO::Handle::Zlib;
    $reader->open( $compressed_filename, "rb" )
        or croak "could not open $compressed_filename: $!";
[download]

with

    my $reader = new IO::File;
    $reader->open( $uncompressed_filename, "r" );
[download]

eliminates the error.

Has anyone used IO::Zlib like this before? Is my IO::Handle::Zlib wrapper bogus? Anybody know how XS modules do IO::Handle reads, and why this doesn't work?

Thanks,

--athomason

Comment on IO::Zlib with XML::Twig/XML::Parser::Expat Select or Download Code

Replies are listed 'Best First'.
Re: IO::Zlib with XML::Twig/XML::Parser::Expat by PodMaster (Abbot) on Aug 23, 2002 at 08:57 UTC
Can you post a minimal test case? I'm sorry to say that yes, your IO::Handle::Zlib wrapper is bogus (doesn't work now does it ;). IO::Zlib provides an interface much like which means it's not exactly like, which is what needs to happen. You could go investingating XML::Parser to figure out all the IO::Handle methods it calls, and in turn, make sure IO::Zlib complies, cause it's not someplace. It's obviously not handling the reading of the file well. I doubt the file is being decompressed past the original size (highly unlikely), I just think the wrapper is bogus. You should try IO::Filter. update: BTW, do you pass ErrorContext to new XML::Twig? You should. Whoa, I just checked my XML::Parser::Expat's sub parse, and it has `if (ref($arg) and UNIVERSAL::isa($arg, 'IO::Handler')) {`. It's version 2.27. How stupid. http://search.cpan.org/src/COOPERCL/XML-Parser-2.27/Expat/Expat.pm confirms it. `____________________________________________________` ** The Third rule of perl club is a statement of fact: pod is sexy.	[reply] [d/l]
Re: IO::Zlib with XML::Twig/XML::Parser::Expat by athomason (Curate) on Aug 24, 2002 at 00:51 UTC
Just to follow up for posterity... It turns out that the wrapper strategy wasn't the problem at all. In fact, it's a bug in IO::Zlib... sort of. When XML::Parser::Expat calls read on a filehandle, it does something a bit unusual. You might think that it would use the read's return value to determine the amount of data actually read, but that's not what it does. Instead, it uses the `len` parameter of the `SvPV` macro (see perlguts) to determine the actual length of the buffer. But IO::Zlib's implementation of Tie::Handle's READ method doesn't touch the buffer if it's at the end of file: it just returns 0 to indicate no bytes are left. So when used together, X::P::E takes the value of the buffer from the most recent non-EOF read, and so in fact does attempt to read more than it should. What it reads is the last chunk of the file, again. Unsurprisingly, this addition of incomplete data past the closing document tag results in illegal XML, which is why I got the error I did. You could almost call this odd behavior a bug in XML::Parser::Expat instead of IO::Zlib, except for this line in read: `SCALAR will be grown or shrunk to the length actually read.` So technically, IO::Zlib's read should set the buffer to `""` before returning 0 at EOF. Solving the problem is therefore just a matter of changing `return 0 if $self->{'eof'};` to `if ( $self->{'eof'} ) { $$bufref = ""; return 0; };` [download] in IO::Zlib. I'll submit the patch later today. Relieved to be done debugging XS code with printf's, --athomason	[reply] [d/l] [select]
Re: IO::Zlib with XML::Twig/XML::Parser::Expat by Anonymous Monk on May 24, 2007 at 13:30 UTC
I came across the same problem five years later (although with IO::Uncompress::Gunzip and XML::Parser::Expat). I worked round it by using the parse_start() method of XML::Parser::Expat, which lets you push the data into the parser a line at a time rather than the parser pulling it. Stephen Turner	[reply]


Think about Loose Coupling
	PerlMonks