Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Re^4: Splitting compound (concatenated) words )

by vit (Friar)
on May 16, 2012 at 23:39 UTC ( [id://970951]=note: print w/replies, xml ) Need Help??


in reply to Re^3: Splitting compound (concatenated) words )
in thread Splitting compound (concatenated) words )

It actually works. I created a dictionary of 50000+ words sorted by popularity. The problem was that I ran your example but I did not have "concatenated" in there.
Thanks for your explanation. I will spend some time to understand this part
print for grep defined(), $s =~ /^$re2$/;
in details. Looks like genius. In terms of both, Perl flexibility and your implementation. Actually it needs
print "$_\n" for grep defined(), $s =~ /^$re2$/;
to print line-wise.

I'm curious as hell about the source of the data and the purpose of the exercise?

The dictionary is the side affect of phrases I retrieved from crawling for my applications. The purpose of the exercise is the following. I created a keyword generator which is a web application. From the logs I found that some people merge words in a seed phrase and my resulted keywords filter fails to establish similarity in these cases. So I need to split. But usually it is only a two reasonable words split so that looks like your algorithm with my dictionary works well.

I can send you a link to the tool, but I am not sure it's a right way to do it through the forum.

Replies are listed 'Best First'.
Re^5: Splitting compound (concatenated) words )
by BrowserUk (Patriarch) on May 17, 2012 at 00:24 UTC
    I will spend some time to understand this part print for grep defined(), $s =~ /^$re2$/;

    Maybe this will help:

    #! perl -slw use strict; my @w = do{ local @ARGV = 'words.txt'; <> }; chomp @w; my $s = 'couldsomeonerecommendaworkingperlmoduletosplitconcatenatedwor +ds'; my @subset = grep{ $s =~ /$_/ } 'a', 'perl', @w; my $re1 = join '|', sort{ length( $b ) <=> length( $a ) }@subset; my $re2 = "($re1)?" x 20; my @found = $s =~ /^$re2$/; print join '|', @found; my @undefsRemoved = grep defined(), @found; print for @undefsRemoved;

    Note: I've increase the number of repetitions from 11 to 20 to allow for sentences with more words. The side effect of that is that the unmatched captures now return undef, so If I just printed out @found, I get this:

    c:\test>junk Use of uninitialized value $found[12] in join or string at C:\test\jun +k.pl line 15, <> line 178691. Use of uninitialized value $found[13] in join or string at C:\test\jun +k.pl line 15, <> line 178691. Use of uninitialized value $found[14] in join or string at C:\test\jun +k.pl line 15, <> line 178691. Use of uninitialized value $found[15] in join or string at C:\test\jun +k.pl line 15, <> line 178691. Use of uninitialized value $found[16] in join or string at C:\test\jun +k.pl line 15, <> line 178691. Use of uninitialized value $found[17] in join or string at C:\test\jun +k.pl line 15, <> line 178691. Use of uninitialized value $found[18] in join or string at C:\test\jun +k.pl line 15, <> line 178691. Use of uninitialized value $found[19] in join or string at C:\test\jun +k.pl line 15, <> line 178691. could|someone|recommend|aw|or|king|perl|module|to|split|concatenated|w +ords||||||||

    So I use grep to remove any undefined values:

    my @undefsRemoved = grep defined(), @found;
    Actually it needs print "$_\n" for grep defined(), $s =~ /^$re2$/; to print line-wise.

    If you look at the first line of the code I posted: #! perl -slw, the l in the -slw has the affect of adding "\n" to print statements automatically. Saves me having to type them all the time.

    I can send you a link to the tool, but I am not sure it's a right way to do it through the forum.

    Just post the link as text, or wrap it in brackets: [ link here ] to make it an active link.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

      I sent you a private message, please check.
      I understand these steps.
      my $re1 = join '|', sort{ length( $b ) <=> length( $a ) }@subset; my $re2 = "($re1)?" x 20; my @found = $s =~ /^$re2$/; print join '|', @found; my @undefsRemoved = grep defined(), @found; print for @undefsRemoved;
      Except $s =~ /^$re2$/; How can it match $s from the beginning to the end? And when match occurs how a part of the string which already matched is not considered any more. I understand that you somehow do it by iterations with repeated (xxxx|yyy|zz)? but cannot get it.

        In the same way that this matches:

        $s = 'fred';; print for $s =~ /^(.)?(.)?(.)?(.)?(.)?(.)?$/;; f r e d Use of uninitialized value $_ in print at (eval 10) line 1, <STDIN> li +ne 2. Use of uninitialized value $_ in print at (eval 10) line 1, <STDIN> li +ne 2.

        Although there are 6 capture groups, and only 4 characters, the capture groups are optional because of the trailing '?' ((.)?).

        So long as the regex has more capture groups than the string contains matches, the regex will match as many as are available, and the remaining capture groups will (optionally) "match" nothing, and so return undefs.

        Does that clarify things?


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://970951]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others meditating upon the Monastery: (2)
As of 2024-04-26 02:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found