http://qs321.pair.com?node_id=1231380


in reply to best approach to parse vichan-style imageboard

Hi,

You do not want to use anything with "parser" in the name

#!/usr/bin/perl -- use strict; use warnings; use Mojo::DOM; my $html = q{ <div class="post reply body-not-empty" id="reply_8735435"> (cut out for visibility) <p class="body-line ltr ">The first 3 lines were 15% bait power, but t +hen it fell to mere 5% and the last lines are literally 0%, try again + in a few days.</p> </div> (cut out for visibility) <div class="post reply body-not-empty" id="reply_8735439"> (cut out for visibility) <div class="body" > <p class="body-line ltr "> <a onclick="highlightReply('8735417', event);" href="/b/res/8735417.ht +ml#8735417">&gt;&gt;8735417</a> </p> <p class="body-line ltr quote">&gt;Reddit is a great place for discour +se and there are many active subreddits where field professionals reg +ularly answer questions on issues of health, science, engineering, et +c</p> <p class="body-line ltr ">Yeah, as far as content goes, Reddit kicks 8 +chan's ass. They have some great boards for serious academic discussi +on.</p> <p class="body-line empty"> }; my $dom = Mojo::DOM->new( $html ); for my $e ( $dom->find( 'div.reply' )->each ){ print $e->{id},"\n", $e->text, "\n\n"; } __END__ reply_8735435 (cut out for visibility) reply_8735439 (cut out for visibility)
  • Comment on Re: best approach to parse vichan-style imageboard html (dom/css xpath)
  • Download Code

Replies are listed 'Best First'.
Re^2: best approach to parse vichan-style imageboard html (dom/css xpath)
by lis128 (Acolyte) on Mar 18, 2019 at 07:14 UTC

    I must admit that your approach is way clearer and understandable than mine.
    Once done with current problem i'll try to read plain HTML using given method :)
    One thing i don't get is statement "avoid parsers": why so?
    I was convinced that parsers were ment to extract data from more complext structures

    On the other hand i've managed to access chans using JSON interface (following https://github.com/vichan-devel/vichan-API/), but my question remains valid for HTML-only imageboards

Re^2: best approach to parse vichan-style imageboard html (dom/css xpath)
by karlgoethebier (Abbot) on Mar 18, 2019 at 16:20 UTC
    "You do not want to use anything with "parser" in the name"

    At least the magic word is in the description of Mojo::DOM:

    "Mojo::DOM is a minimalistic and relaxed HTML/XML DOM parser with CSS selector support. It will even try to interpret broken HTML and XML..."

    «The Crux of the Biscuit is the Apostrophe»

    perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help