http://qs321.pair.com?node_id=11141637


in reply to Re^2: Splitting multiline string into words, the stuff between words, and newlines
in thread Splitting multiline string into words, the stuff between words, and newlines

That is because \b{wb} matches between those signs.

This seems to solve the issue:

my @fragments = grep length, split /(\b{wb}\w.*?\b{wb}|\n+)/, $book;

But my knowledge of Unicode and the \b{wb} semantics is rather limited so that may have other issues.

Replies are listed 'Best First'.
Re^4: Splitting multiline string into words, the stuff between words, and newlines
by LanX (Saint) on Feb 25, 2022 at 10:22 UTC
    Not sure 'cause that's 'bout words also including non \w characters.

    And some of 'em even start on apostrophe ;)

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

      Well, specifically for the apostrophe, \b{wb} doesn't seem to take initial ones as part of words. It breaks your sample sentence as follows: {Not} {_} {sure} {_'} {cause} {_} {that's} {_'} {bout} {_} {words} {_} {also} {_} {including}

      But I agree with you that there are probably other cases of words (as defined by \b{wb}) that don't start by a character matching \w.

      At the end, my conclusion is that the only way to handle the OP problem in a way fully consistent with \w{wb} semantics is to just split using it, and maybe repack non word fragments afterwards:

      my $book = "Not sure 'cause that's 'bout words also including ...\n.\n +_\n\n..."; my @fragments; my $last_was_symbol; for (split /\b{wb}/, $book) { if (/[\w\n]/) { $last_was_symbol = 0; push @fragments, $_; } else { if ($last_was_symbol) { $fragments[-1] .= $_; } else { push @fragments, $_; $last_was_symbol = 1; } } } sub show { my $str = shift; $str =~ tr/\n/$/; $str =~ tr/ /_/; print "{$str} "; } show $_ for @fragments; print "\n";
        > \b{wb} doesn't seem to take initial ones as part of words

        good catch!

        > my conclusion is that the only way to handle the OP problem in a way fully consistent with \w{wb} semantics is to just split using it, and maybe repack non word fragments afterwards

        My intuition says split on non-words like whitespace, reject "words" without \w or equivalent characters and repack the rest afterwards.

        I doubt it's possible to cover all desirable edge cases by \b{wb} this will depend on the user's perspective, especially when considering multi-language environments and unicode.

        Cheers Rolf
        (addicted to the Perl Programming Language :)
        Wikisyntax for the Monastery

Re^4: Splitting multiline string into words, the stuff between words, and newlines
by ibm1620 (Hermit) on Feb 26, 2022 at 21:00 UTC
    For my purposes, this is fine. I'm mainly interested in capturing possessives and contractions.