sgifford has asked for the wisdom of the Perl Monks concerning the following question:
I am working on a Web application implementing a RETS server. The output of most types of transactions is a fairly large XML document. Depending on what the user asks for, it can have a large number of structured elements, or one element with tab-separated data in it. Very roughly, it looks either like this:
or this:<results type='xml'> <result1> ... </result1> <result2> ... </result2> ... <result6000> ... </result6000> </results
I looked at several XML modules to generate the XML, and settled on XML::LibXML. The problem I'm running into is that this module, along with the others I looked at, won't output anything until the entire XML document is done. With 6000 fairly large entries, that can be about 18MB of data. When processing it all at once, the user sends the requests then waits up to a minute before seeing anything. In the meantime, the server is chewing up memory to store the document.<results type='compact'> RESULT1 ... RESULT2 ... ... RESULT 6000 ... </results>
The analagous problems with reading XML is solved by SAX, which lets you process the document as it comes in, without holding the whole thing in memory.
Is there a similar way to output XML documents, outputting the data immediately instead of waiting for the whole document to finish but letting a module handle most of the XML details? I would like something a bit more structured than generating the XML by hand and printing it. Ideally it would help format elements and attributes, close tags (or at least warn me if I forget) and otherwise make sure the document was correctly formatted. Thanks for any suggestions!