comment on

I use SAX for this. For me it provides the best performance/maintainability ratio.

Here is a standalone program that takes a html file name as an argument and prints the output to STDOUT.

Its purpose is to strip all tags except p, div, and a tags.

The program driver sets up the SAX pipeline. It uses XML::Driver::HTML as the SAX driver and XML::SAX::Writer as the writer.

When parse is called on the driver, the Pipeline sends the driver's stream to some helper modules, to our filter module, and finally the writer.

In the custom filter, start_element is called for each tag. The code checks to see if the tag is either an a, div, or p tag, and if it is, it forwards the tag to the writer. Otherwise, the tag is ignored and removed from the stream.

The same work needs to happen in the end_element callback.

benefits:

Accepts non-well-formed html and outputs well formed xml.
Scales well. In theory it should use a constant amount of memory.
High level of control of the output document

drawbacks:

need to learn SAX
if you remove a node from the stream in the start_element callback you have to remove the closing tag in the end_element callback (this isn't so much of a "drawback" but more of a reminder that you have to stay on your toes).

use warnings;
use strict;

use XML::SAX::Machines qw(Pipeline);
use XML::Driver::HTML;
use XML::Filter::SAX1toSAX2;
use XML::Filter::BufferText;
use XML::SAX::Writer;

my $output; # transformation target
my $writer = XML::SAX::Writer->new( Output => \$output );

my $machine = Pipeline(
  'XML::Filter::SAX1toSAX2'        =>
  'XML::Filter::BufferText'        =>
  'XML::Filter::HtmlTagStripper'   =>
  $writer
);

my $html = new XML::Driver::HTML(
  Handler => $machine,
  Source  => { SystemId => $ARGV[0] }
);

$html->parse();

print $output;

package XML::Filter::HtmlTagStripper;
use base qw|XML::SAX::Base|;

# <marker language="foo" />
# $el->{Name} == 'marker'
# $el->{Attributes}{'{}language'} == language attribute
# $el->{Attributes}{'{}language'}{Value} == 'foo'

sub start_element {
  my($self, $el) = @_;
  if ( $el->{Name} =~ m/^(?:p|div|a)$/i ) {
    $self->SUPER::start_element( $el );
  }
}

sub end_element {
  my($self, $el) = @_;
  if ( $el->{Name} =~ m/^(?:p|div|a)$/i ) {
    $self->SUPER::end_element( $el );
  }
}

1;
[download]

I know its not the most un-rocket-sciencey thing in the world, but its not too tricky, and once it clicks in your head and you realize that this is high performance xml parsing, the possibilities are boggling.

Lets take a look at a run:

$ cat striptags.html
<html>
  <head>
    <title>Test Document</title>
  </head>
  <body>
    <p>The first paragraph</p>
    <p>the second paragraph</p>
    <hr width="75%">
    <div>last modified: WHENEVER</div>
  </body>
</html>

$ perl striptags.pl striptags.html
Test Document<p>The first paragraph</p><p>the second paragraph</p><div
+>last modified: WHENEVER</div>
[download]

Enjoy, Todd W.

In reply to Re: Simplify HTML programatically by trwww
in thread Simplify HTML programatically by guha

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


more useful options
	PerlMonks