Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

I use SAX for this. For me it provides the best performance/maintainability ratio.

Here is a standalone program that takes a html file name as an argument and prints the output to STDOUT.

Its purpose is to strip all tags except p, div, and a tags.

The program driver sets up the SAX pipeline. It uses XML::Driver::HTML as the SAX driver and XML::SAX::Writer as the writer.

When parse is called on the driver, the Pipeline sends the driver's stream to some helper modules, to our filter module, and finally the writer.

In the custom filter, start_element is called for each tag. The code checks to see if the tag is either an a, div, or p tag, and if it is, it forwards the tag to the writer. Otherwise, the tag is ignored and removed from the stream.

The same work needs to happen in the end_element callback.

  • benefits:
    • Accepts non-well-formed html and outputs well formed xml.
    • Scales well. In theory it should use a constant amount of memory.
    • High level of control of the output document
  • drawbacks:
    • need to learn SAX
    • if you remove a node from the stream in the start_element callback you have to remove the closing tag in the end_element callback (this isn't so much of a "drawback" but more of a reminder that you have to stay on your toes).
use warnings; use strict; use XML::SAX::Machines qw(Pipeline); use XML::Driver::HTML; use XML::Filter::SAX1toSAX2; use XML::Filter::BufferText; use XML::SAX::Writer; my $output; # transformation target my $writer = XML::SAX::Writer->new( Output => \$output ); my $machine = Pipeline( 'XML::Filter::SAX1toSAX2' => 'XML::Filter::BufferText' => 'XML::Filter::HtmlTagStripper' => $writer ); my $html = new XML::Driver::HTML( Handler => $machine, Source => { SystemId => $ARGV[0] } ); $html->parse(); print $output; package XML::Filter::HtmlTagStripper; use base qw|XML::SAX::Base|; # <marker language="foo" /> # $el->{Name} == 'marker' # $el->{Attributes}{'{}language'} == language attribute # $el->{Attributes}{'{}language'}{Value} == 'foo' sub start_element { my($self, $el) = @_; if ( $el->{Name} =~ m/^(?:p|div|a)$/i ) { $self->SUPER::start_element( $el ); } } sub end_element { my($self, $el) = @_; if ( $el->{Name} =~ m/^(?:p|div|a)$/i ) { $self->SUPER::end_element( $el ); } } 1;

I know its not the most un-rocket-sciencey thing in the world, but its not too tricky, and once it clicks in your head and you realize that this is high performance xml parsing, the possibilities are boggling.

Lets take a look at a run:

$ cat striptags.html <html> <head> <title>Test Document</title> </head> <body> <p>The first paragraph</p> <p>the second paragraph</p> <hr width="75%"> <div>last modified: WHENEVER</div> </body> </html> $ perl striptags.html Test Document<p>The first paragraph</p><p>the second paragraph</p><div +>last modified: WHENEVER</div>
Enjoy, Todd W.

In reply to Re: Simplify HTML programatically by trwww
in thread Simplify HTML programatically by guha

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others wandering the Monastery: (3)
    As of 2021-02-27 02:00 GMT
    Find Nodes?
      Voting Booth?

      No recent polls found