On Athanasius

Task 2 of the current Perl Weekly Challenge is to generate a pangram from the English dictionary provided. “A pangram or holoalphabetic sentence is a sentence using every letter of a given alphabet at least once.”¹ However, for this challenge the requirement that the words form a sentence is removed: a list of unrelated words is all that is required.

The bonus challenge is to “Constrain or optimize for something interesting,” and the first suggestion given is:

Shortest possible pangram (difficult)

Ok, so just how difficult would it be to perform an exhaustive search? I don’t know the answer, but since the search space increases exponentially as words are added to the dictionary, a search for sequences (of varying lengths) drawn from a pool of 23,610 words (see below) would appear highly impractical. I haven’t attempted it.

Before we proceed, it should be noted that the dictionary can be significantly reduced in size before the search begins. Where groups of words share the same set of letters, only one word from the group (the shortest) need be kept. For example, “abdicate,” “diabetic,” and “abdicated” all have the letter set {a, b, c, d, e, i, t}, so “abdicate” is retained but “diabetic” and “abdicated” are filtered out. This reduces the dictionary size from 39,172 to 23,610 — a saving of 15,562 words!

As noted above, 23,610 is still far too many words for an exhaustive search. So we’re looking for a way to get a close-to-optimum result in a reasonable time. And as we’ve just noted, each dictionary word can be considered as a set of its component letters. So the task is to find a minimum set cover, a well-studied problem in combinatorics and computer science.² And there is an algorithm that is known to give good results in polynomial time: the greedy algorithm:

at each stage, choose the set that contains the largest number of uncovered elements.

My initial implementation of this algorithm produced the following pangram:

oversimplification ladybug hawk jazz equinox (40 letters)

I wasn’t satisfied with this, so began looking for ways to improve the search. It occurred to me that words with rare letters should be preferred over words with common letters. This immediately suggested the Scrabble³ points system:

const my %ALPHABET =>
      (
          a => 1, b =>  3, c => 3, d => 2, e => 1,
          f => 4, g =>  2, h => 4, i => 1, j => 8,
          k => 5, l =>  1, m => 3, n => 1, o => 1,
          p => 3, q => 10, r => 1, s => 1, t => 1,
          u => 1, v =>  4, w => 4, x => 8, y => 4, z => 10
      );
[download]

With this I was able to give each word a weighting based on its component letters, and prefer words with higher weightings. This produced the following pangram:

sympathized quarterbacking reflexive jawbone (41 letters)

— no improvement at all! The problem appears to be that the selected words contain too much deadweight — too many redundant letters. So would it help to adjust the weighting for each word by deducting points for redundant letters? I experimented with different values of $DUP_WEIGHT, the amount to deduct for each redundant letter. Here are the results:

Eureka! At 29 letters, the last pangram is only 3 letters above the theoretical best-possible result of 26. (For values of $DUP_WEIGHT above 4, the result does not change.)

It should be emphasized that my improved results are not an improvement on the greedy algorithm. That algorithm applies to sets, whereas my search for pangrams is a search over words, which are essentially bags because they contain multiple instances of the same letters.

For the implementation, I found the Set::Tiny module very useful. I highly recommend it for working with sets of strings.

Wikipedia¹ gives the following as a short pangram:

Waltz, bad nymph, for quick jigs vex. (28 letters)

Since all these words occur in the given dictionary, it would be possible for a search to improve on my best solution by at least one letter. Challenge: can anyone implement a search which finds this pangram (or a shorter one) within a reasonable time?

Full code listings for my solution (in Perl and Raku) will shortly be are available on the Weekly Challenge’s GitHub repository at https://github.com/manwar/perlweeklychallenge-club/tree/master/challenge-161/athanasius.

Cheers,

¹“Pangram, ” Wikipedia, https://en.wikipedia.org/wiki/Pangram
²“Set cover problem,” Wikipedia, https://en.wikipedia.org/wiki/Set_cover_problem
³ “Scrabble letter distributions,” Wikipedia, https://en.wikipedia.org/wiki/Scrabble_letter_distributions#English

Esteemed Monks,

I was looking at The On-Line Encyclopedia of Integer Sequences (OEIS) (you know, as one does), and on the Welcome page I found a list of Some Famous Sequences, of which the first is Recamán’s sequence, defined as follows:

R(0)  = 0;
for n > 0,
    R(n) = R(n-1) - n if positive and not already in the sequence, oth
+erwise
    R(n) = R(n-1) + n.
[download]

What makes this sequence interesting is N. J. A. Sloane’s conjecture that every number eventually appears.

Coding the sequence is simplicity itself; the challenge is to test Sloane’s conjecture by keeping track of the numbers that have not yet appeared in the series. My initial, naïve approach was to use a sieve, à la Eratosthenes:

But this turned out to be far too memory-hungry: for values of MAX of the order of twenty million, RAM usage on my 3GB system approaches 100%, thrashing sets in, and the script (along with the rest of Windows) grinds to a shuddering halt.

Surely, I thought, there must be a memory-efficient way to represent a sieve? And of course there is, and of course it was already implemented on CPAN. A little searching led to the Set::IntSpan module which stores runs of consecutive integers as spans, allowing large (even infinite) collections of integers to be represented very economically.

Calculation of successive terms in the Recamán sequence is noticeably slower using Set::IntSpan for lookup than it is using a hash. But, as the adage says, it’s better to be late than be dead on time. (This was the slogan of an Australian safe driving ad campaign some years ago.) For the record: I also looked at Set::IntSpan::Fast and Set::IntSpan::Fast::XS. The latter failed to install on my system, and the former actually ran slower than Set::IntSpan for this use-case.

Turns out that Set::IntSpan not only solves the memory problem, it also makes it possible to dispense with an upper bound for the sieve. How, then, to display progressive results? Well, the OEIS has a couple of additional series related to Recamán’s:

• A064228: values of R(n) that take a record number of steps to appear: 1, 2, 4, 19, ...
• A064227: the values of n corresponding to the values in A064228: 1, 4, 131, 99734, ...

So I recast the script to output successive values of these two series:

14:20 >perl recaman.pl
1 <-- 1
2 <-- 4
4 <-- 131
19 <-- 99734
...
[download]

Here is the new script:

use strict;
use warnings;
use sigtrap handler => \&int_handler,   'INT',
            handler => \&break_handler, 'BREAK';
use Set::IntSpan;
use Time::HiRes qw(gettimeofday tv_interval);

   $|       = 1;
my $t0      = [gettimeofday];
my $min0    = 1;
my $n       = 0;
my $r0      = 0;
my $missing = Set::IntSpan->new( '1-)' );

print "$min0 <-- ";

while (++$n)
{
    my $r = $r0 - $n;
       $r = $r0 + $n if $r < 0 || !$missing->member($r);

    $missing->remove($r);

    if ((my $min1 = $missing->min) > $min0)
    {
        print "$n\n$min1 <-- ";
        $min0 = $min1;
    }

    $r0 = $r;
}

sub int_handler
{
    printf "\nn = %d, elapsed time: %.1fs\n", $n, tv_interval($t0);
}

sub break_handler
{
    int_handler();
    exit 0;
}
[download]

This script was developed under Windows 8.1, 64-bit, using Strawberry Perl:

14:20 >perl -v

This is perl 5, version 22, subversion 0 (v5.22.0) built for MSWin32-x
+64-multi-thread
[download]

The two signal handlers allow the script to be interrupted as follows:

• Control-C causes the script to display the current value of $n and the total running time of the script so far.
• Control-Break causes the script to display the same information and then exit.

My takeaways from this meditation?

First, we all know that micro-optimisation is pointless until you have first selected the best algorithm(s). But optimising an algorithm may actually consist in optimising its underlying data structures. Obvious? Yes, but still worth a reminder now and then.

Second, CPAN is awesome! But you knew that already. :-)

Cheers,

It’s Christmas! Ring the Monastery bell,
As we with joy recall that first Noël
When shepherds, watching through the lonely night,
Were over-awed by heaven’s glorious light.

With cheer, we’ll greet our brother monks, and sisters,
And amnesty extend to Pythonistas,
And other erring souls; foreach, today,
We wish bright blessings in a long @array.

And tho’ at times, alas!, we make a %hash
Of our Monastic Rule, with insults rash
And words unkind: today we start anew
And map our way with kinder goals in view.

A maid needs help with syntax? We’ll not fail ’er.
A hill of bugs is in our way? We’ll $scalar.
Tho’ all the year in Perl we script — and well, —
Today it’s “Per”, in honour of Noël.

© 2014

Merry Christmas!

Happy Monkday!!
You’ve been here 1 thrilling year.

Is it really a whole year since I joined the Monastery? I’m so glad I did, and so very grateful to all the monks whose enthusiasm and dedication make PerlMonks the special place it is.

So this seems like a good opportunity to post a rather long poem I wrote to meet the challenge of finding rhymes for all the monastic levels, from Initiate to Pope.

Enjoy! :-)

Cheers,

© 2012

© 2012