http://qs321.pair.com?node_id=11121776


in reply to regex, find words that occur more than once.

I finished my script like so:

use strict; use warnings; my %words; my $text = "and him him lad has him done and john has has"; while($text=~/\b(\w+)\b(?=.*\b\1\b)/g){ $words{$1}++; } foreach my $key(keys %words){ my $count = $words{$key} + 1; print "$key: $count\n"; }

Again thank you for the advice.

Replies are listed 'Best First'.
Re^2: regex, find words that occur more than once.
by GrandFather (Saint) on Sep 15, 2020 at 00:15 UTC

    Nice synthesis of the advice you got!

    In production code dealing with many searches on large strings you might want to make the .* match in the look ahead assertion non-greedy (.*?) so a search to the end of the string isn't required. Also, adding sort to (keys %words) gets the words in consistent order making it easier to find particular results if there are a lot of words.

    Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond
Re^2: regex, find words that occur more than once.
by kennethk (Abbot) on Sep 15, 2020 at 13:38 UTC
    Good synthesis, as GrandFather says. A couple of minor stylistic elements you might consider:
    • You should put spaces around your Binding Operators. Regular expressions already look confusing to a lot of people. Same for before the parentheses of your for loop, but that's less of an issue.
    • One might naively expect that the hash in question contains the counts, and you correct that right before output. That could be confusing if the two loops were more complex or separated by a lot of code. Rather than fixing it at the end, you could initialize the value to 1:
      while($text=~/\b(\w+)\b(?=.*\b\1\b)/g){ $words{$1} //= 1; # initialize empty value to one $words{$1}++; }
      See Logical Defined Or and Assignment Operators for details on what I did there.
    • I usually name my hashes based upon how the key is related to the value. In this case, for example, %hits could be natural because when you look at it on the screen, you see the number of hits you got -- $hits{$word} reads to me as hits for word in English. On the other hand, %words sounds like I'm going to get words back, not a number. With that first change, it might be %count, %repetitions or %reps. Perl more than most programming languages is supposed to read following common grammar.
    Full version with suggested modifications:
    use strict; use warnings; use 5.10.0; my %reps; my $text = "and him him lad has him done and john has has"; while($text =~ /\b(\w+)\b(?=.*\b\1\b)/g){ $reps{$1} //= 1; $reps{$1}++; } foreach my $key (keys %words){ say "$key: $reps{$key}"; }

    #11929 First ask yourself `How would I do this without a computer?' Then have the computer do it the same way.

      I completely agree with you on the use of white space.

      The defined or initialization ($reps{$1} //= 1;) is completely unnecessary code bloat, partly because the values in the hash are only marginally important. For purposes of answering the question "Has this word repeated" $words{$1} = undef; works just as well and the following increment is not required.

      The post increment ($words{$1}++;)I should have picked up in the OP. It is mindless cargo culting with no redeeming qualities. Unless logic requires otherwise always pre-increment (++$words{$1};. Pre-increment is never slower (and often faster) than post-increment - which is of trivial importance almost always. Much more important is that sticking the operator out the front makes it easier to be seen making the code easier to understand and maintain.

      To my mind the most important aspect of the words hash is that it is a list of words - the keys are more important than the values. The name of the hash tells you that. Especially where a hash is being used to collect unique instances of things, realizing that keys can be the important data and the values are incidental is a liberating break through.

      Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond

        Hopefully I properly qualified my comments since I tried to offer them as food for thought rather than some prescriptive set of requirements.

        The pre-increment is a good suggestion, and aligns with a verb-object syntax nicely. I need to break that muscle memory.

        The subject of the post agrees with your interpretation of the hash, but the usage runs counter. As soon as the goal becomes "How many times are words repeated?" the approach should have an accurate count in whatever stash you are using, as close to data as possible. Obviously the best answer is what you gave in Re: regex, find words that occur more than once., but subject to a homework-like requirement of use a regex a defined-or assignment keeps the store honest while still being reasonably succinct (and doesn't net add any lines). I'm not fundamentally opposed to communicating that the keys are the important thing, but especially when people are starting out, having the hash named after the property you are measuring (e.g., %count, %seen, %formatted) avoids %words, %others, %words2 anti-patterns.


        #11929 First ask yourself `How would I do this without a computer?' Then have the computer do it the same way.