http://qs321.pair.com?node_id=285065

A Pattern of Misunderstanding: Fetishizing the Gang of Four

Over the last few weeks, after reading the first two articles in Phil Crow's "Perl Design Patterns" series on Perl.com, (1, 2) I've been troubled by a feeling that I couldn't quite put my finger on.

Finding myself with some unexpectedly-free time a few days ago (which is to say, while the blackout kept me offline) I paged through some of my patterns books (by candlelight) and thought about what had made me uncomfortable.

By the time the lights came on, I'd concluded that the Perl.com articles did not seem to have connected with the essential intent of the patterns literature. Moreover, the author's reading of the "Gang of Four" book was downright hostile and fetishistic: rather than responding to the content of the book, the focus was on dismissing or deflating the hype that has built up around it.

The reason the Gang of Four book is such a famous reference in the software world has a lot to do with the history of the patterns movement (and with the way fads develop and play out), but to focus on the patterns cataloged in that book, to the exclusion of the rest of the patterns literature, gives a lop-sided impression of what "software patterns" are all about.

What are Software Patterns?

In an attempt to refine a clearer definition of the software patterns field, I revisited a handful of books by different authors:

  • Design Patterns: Elements of Reusable Object-Oriented Software (E. Gamma et al, aka "the Gang of Four")
  • Analysis Patterns: Reusable Object Models (M. Fowler)
  • AntiPatterns: Refactoring Software, Architectures, and Projects in Crisis (W. Brown et al)
  • A System of Patterns: Pattern-Oriented Software Architecture (F. Buschmann et al)
  • Smalltalk Best Practice Patterns (K. Beck)

Although there's a fair degree of variation between the styles and subjects of these books, there did seem to be a handful of areas of consensus; I identified five shared characteristics, explained below along with some quotes from the original materials:

Patterns are A Type Of Documentation: As used in the software methodology field, patterns are a form of best-practices documentation. Their intent is to communicate the observations, analysis, and recommendations of experienced developers to others in an effective way.

"Patterns and pattern languages are ways to describe best practices, good designs, and capture experience in a way that it is possible for others to reuse this experience."
The Hillside Group, hillside.net/patterns
"Patterns are a recent software engineering problem-solving discipline that emerged from the object-oriented community... The goal of the pattern community is to build a body of literature to support design and development in general. There is less focus on technology than on a culture to document and support sound design."
J. Coplien, hillside.net/patterns/definition.html
"A pattern is a decision an expert makes over and over. For example, naming an instance variable after the role it plays is a pattern. Even though all the results (in this case, all the variable names) are different, they have a quality that is constant."
K. Beck, Smalltalk Best Practice Patterns, Preface

Patterns are Contextual Solutions: Each pattern is expressed as a written description of a design, solution, or technique that other developers have found repeatedly useful, along with a discussion of the context and challenges that the solution is attempting to address. (Anti-patterns are similar, but shift the focus to the problems they describe instead of the solutions.)

"Each pattern is a three-part rule, which expresses a relation between a certain context, a problem, and a solution."
C. Alexander, The Timeless Way of Building, p. 247
"A pattern for software architecture describes a particular recurring design problem that arises in specific design contexts, and presents a well-proven generic scheme for its solution."
F. Buschmann et al, A System of Patterns, p. 8
"This is a book of design patterns that describes simple and elegant solutions to specific problems in object oriented software design. "
E. Gamma et al, Design Patterns, Preface

Patterns are Both Observations and Recommendations: Patterns play a dual role as both descriptive and prescriptive documents. On the one hand, they recognize and name some technique that developers are already using. On the other hand, the pattern acts as a recommendation to others: if you're trying to solve the problem described in the pattern, consider using some variation of the solution it describes.

"The pattern is... at the same time a thing, which happens in the world, and the rule which tells us how to create that thing, and when we must create it. It is both a process and a thing: both a description of a thing which is alive, and a description of the process which will generate that thing."
C. Alexander, The Timeless Way of Building, p. 247
"Design patterns capture solutions that have developed and evolved over time... in a succinct and easily applied form."
E. Gamma et al, Design Patterns, Preface

Patterns are Interconnected: Patterns are collected in interconnected networks, called pattern systems. (A pattern system that aspires to fully cover some topic can also be called a pattern language, although various communities use these terms differently.) It's common for a pattern description to suggest other patterns that work with it (add an Iterator to your Composite!), or alternate patterns that might solve the problem differently. In concrete terms, this means that pattern-oriented books and sites are organized as a interconnected network of writeups, rather than a rigid sequential procedure or a hierarchical outline.

"Patterns do not exist in isolation -- there are many interdependencies between them... A pattern system ties its constituent patterns together. It describes how the patterns are connected and how they complement each other."
F. Buschmann et al, A System of Patterns, p. 360
"There is a structure on the patterns, which describes how each pattern is itself a pattern of other smaller patterns. And there also rules, embedded in the patterns, which describe the way that they... must be arranged with respect to other patterns."
C. Alexander, The Timeless Way of Building, p. 185

Patterns are Distinctly Named: Each pattern within a pattern system is given a distinct name. The name can serve as an abstraction that helps the developer think at a higher level: instead of seeing a soup of a dozen methods spread out over a handful of classes, you can think in terms of "a Publish-and-Subscribe interaction" or whatever. The name also facilitates communication between developers who share the same patterns: a developer familiar with the Observer pattern can guess what $target->add_observer( $self ) is trying to accomplish.

"The pattern name is a handle we can use to describe a design problem, its solutions, and consequences in a word or two. Naming a pattern immediately increases our design vocabulary. It lets us design at a higher level of abstraction. Having vocabulary for patterns lets us talk about them with our colleagues, in our documentation, and even to ourselves."
E. Gamma et al, Design Patterns, p. 3

What Software Patterns Aren't

Sadly, the idea of software patterns was popularized in a superficial way, such that many people have heard of them but lack a solid understanding of what they are. Thus, as originally pointed out by Mark Jason Dominus, what many people think are design patterns, aren't.

Inverting the consensus definition I came up with earlier, we can state that:

Patterns aren't Language Features: Patterns are a type of documentation that describe how experienced programmers use those languages, not optional features that the languages might or might not possess. At best, a programming language facilitates useful patterns, such that dozens of lines of Java can be replaced by a few lines of idiomatic Perl.

Patterns aren't Universally Applicable: Patterns are contextual solutions that describe how a particular problem fits with a particular solution. The Gang of Four patterns focus on challenges in object-oriented C++ code (with secondary consideration given to Smalltalk); that these might not be directly applicable to Perl programmers who prefer procedural code should come as no surprise.

Patterns aren't Platonic Ideals: Patterns are real-world observations from a particular development community. The patterns described by the Gang of Four aren't free-floating abstractions that somehow exist outside of time and space and were waiting for us to come and discover them; they're practical responses to various day-to-day challenges, including the limitations of C++. An attempt to document Perl Design Patterns should be looking at the best practices of the Perl development community, not just the parts that resemble a ten-year-old collection of patterns from a different language.

Patterns aren't Total Solutions: Patterns are interconnected with other patterns that support or compete with them; no pattern stands on its own as a silver-bullet for your software development challenges. The description of each pattern should review the context in which it is used, acknowledge the tradeoffs involved, and refer to other patterns that offer an alternative approach.

Patterns aren't Generic Concepts: Patterns are distinctly named so that developers can use that name as an abstraction and in communication with others. The Gang of Four knew that C++ had arrays and loop operators like for and while, but they weren't trying to write a general reference about all forms of looping. Trying to gather all of the looping-related techniques into one pattern would result in something too general to be of much use.

Patterns Mistaken

Re-reading the Perl.com articles with these ideas in mind, I was struck by the differences between the articles and the book they critique.

Here are three examples of confusing or contradictory references to patterns, taken just from the Iterators section of the first article. (Please understand that I don't mean to pick on the author of the Perl.com articles; such confusion is widespread and this is far from the worst offense.)

"If a pattern is really valuable, then it should be part of the core language." On the contrary, software patterns typically aren't "part of" a programming language, and it's often not clear what that would even mean. For example, Template Method is a pretty valuable pattern, but how would you make it "a part of Perl" in some concrete way?

"The inclusion of iteration as a core concept represents Perl design at its finest... Perl incorporates this pattern into the core of the language." This description confuses two related terms: "iteration" is a nearly-universal programming concept (cf. HQ9+'s 9 operator), whereas "the Iterator Pattern" is a specific object-oriented design solution (cf. the Gang of Four). The author goes on to describe a Perl pattern that is quite different from the Gang of Four's Iterator objects; one might instead call it the "Return Aggregates as a List" pattern. Blurring the distinction between these different concepts reduces the communication value of the names they've been given.

"To see that foreach fully implements the iterator pattern, even for user-defined modules, consider an example from CPAN: XML::DOM." To see an instance of what the Gang of Four meant by "the iterator pattern," consider another example from CPAN: DBI. Sure, there are times when you can simply call $sth->fetchall_arrayref() to put all of the matching rows into a list, but in some circumstances you want an iterator object (aka a "cursor") that lets you loop through the records one by one with while ( @row = $sth->fetchrow_array ) { ... }. That's clearly an instance of the "Use an Iterator Object" pattern rather than the "Return a List" pattern.

More broadly, the idea of describing best practices in a general, reusable way seems to have been lost somewhere along the way. For example, I would expect a presentation of the Template Method pattern to show how the master method defined in the superclass calls different internal methods depending on the subclass of an individual object. Having the subclass insert the methods into the superclass' namespace, such that you can not define different behaviors for different subclasses, is not a widely reusable approach, and is certainly not the type of functionality envisioned by the Gang of Four Template Method pattern.

Towards a Perl Iterator Pattern

So if software patterns, such as using iterator objects, do play a role in Perl, albeit a reduced one due to its dynamic nature and widespread use of the other patterns like "Return a List," what might a description of such a Perl pattern look like?

The Gang of Four spend fifteen pages discussing their Iterator pattern, and I'm not going to embed an equally long writeup, but here's a very rough draft of what I think such a pattern might look like, in pseudo-perldoc format. I've used closures in the below examples, in response to the Perl.com article's insistence that "objects are overkill," but a complete pattern writeup would also illustrate the use of classes of iterator objects. See the PerlDesignPatterns IteratorInterface page for a further effort in this direction.

NAME
Iterator Pattern - Use a separate entity to facilitate a loop.

PROBLEM
* A complex data structure may need to allow looping over its contents.
* There are several types of data structures that you need to loop over.
* You might need to loop over a data structure in different ways.
* You might be fetching data elements incrementally to avoid memory bloat.

SOLUTION
Create a new entity which is responsible for selecting the elements in order. Modify your loop code to create the proper iterator, then request the elements from it.

RELATED PATTERNS
If your collection is not particularly large or complex, you might prefer to "Return Aggregates as Lists".

DISCUSSION
There are many ways of implementing this pattern in Perl.

The iterator can be a closure or a full-blown class hierarchy.

The iterator's interface can be "active" or "passive":
* active or external iterator: allows the caller to step through the loop and perform its operation at each step;
* passive or internal iterator: controls the loop and allows the client to pass in an operation to be performed on the items.

EXAMPLES
For example, consider the following code, which uses a foreach loop to print out the keys and values in a hash:

    sub hash_pair_printer {
	my $hash = shift;
	foreach my $key (keys %$hash) {
	    print "$key\t$hash{$key}\n";
	}
    }
    
    # The normal use we intended to support
    my %hash = ( foo => 'Foozle', bar => 'Bozzle' );
    hash_pair_printer( \%hash );

Obviously, the foreach loop shown above is a perfectly fine piece of code, and if that's all there is to the task, nothing need be done to it. However, if we want it to be more flexible, or if our requirements change, it'll need to get more complicated.

For example, we might want to be able to print out either the contents of several different kinds of data structures or objects. In some cases, we might be able to just extract the relevant values and place them into a hash to be passed in, but sometimes other constraints prevent this. We could write specialized variations of the function that operated on different kinds of objects, but much of their logic would be duplicated, and we'd need to add another one every time someone asked us to support another data type.

The Iterator pattern suggests a way of decomposing this operation into separate "loop control" and "for each item" capabilities that can be varied independently, allowing users the flexibility of recombining software in new ways. The examples below show simple iterator implementations using code refs, but object-oriented solutions are also common.

External Iterator: It's easy to create an external iterator using a closure:

    sub pair_printer {
	my $iterator = shift;
	while ( my ($key, $value) = $iterator->() )
	    print "$key\t$value\n";
	}
    }

    sub hash_iterator {
	my $hash = shift;
	my @keys = keys %$hash;
	return sub { 
	    scalar @keys or return; 
	    my $key = shift @keys; 
	    return $key, $hash->{$key} 
	}
    };
    
    # The normal use we intended to support
    my %hash = ( foo => 'Foozle', bar => 'Bozzle' );
    pair_printer( hash_iterator( \%hash ) );
In return for the added complexity of having separated the iterator into a separate entity, we get the flexibility to reuse this code in new ways:
    my $dbh = DBI->connect( ... );
    my $sth = $dbh->prepare('select id, name from students');
    pair_printer( sub { $sth->fetchrow_array } );

Internal Iterator: It's also easy to create an internal iterator:

    sub pair_printer {
	my ($key, $value) = @_;
        print "$key\t$value\n";
    };

    sub hash_iterator {
	my $hash = shift;
	my $action = shift;
	foreach my $key (keys %$hash) {
	    $action->( $key, $hash->{$key} );
	}
    }
    
    # The normal use we intended to support
    my %hash = ( foo => 'Foozle', bar => 'Bozzle' );
    hash_iterator( \%hash, \&pair_printer );
In return for the added complexity of having separated the iterator into a separate entity, we get the flexibility to reuse this code in new ways:
    my $headers = HTTP::Headers->new( ... );
    $headers->scan( \&pair_printer );

COMMENTS
Every Perl hash is equipped with its own built-in iterator, accessible through each %hash, but its use is somewhat prone to failure; for example, the following attempt to add an iteration count accidentally causes an infinite loop instead because calling keys also resets the iterator used by each.

    sub hash_pair_printer {
	my $hash = shift;
	while ( my ($key, $value) = each %$hash) {
	    print "$key\t$value\n";
	    warn "Iteration ", ++ $counter, " of ", scalar keys %$hash;
	}
    }

What Next for Perl Patterns?

Although discussions of software patterns in the context of Perl have come up regularly over the last handful of years, they haven't seemd to play a significant role in the way most people approach Perl programming.

Indeed, on the surface, the Perl world's reaction to the patterns movement has been dominated by indifference, flecked with pockets of hostility. But on a closer look, it seems that the hostility is aimed less at the concept of design patterns, than at the hyperbole surrounding the Gang of Four's "Design Patterns" book.

In fact, the documentation practices of the Perl community seem to share some characteristics with patterns work, with web sites like this one, and books like "Effective Perl Programming" and the "Perl Cookbook", organized as collections of contextual solutions.

I believe that there's value in documentation efforts to collect good Perl practices across a range of scales and organized along a patterns model, and I'd like to imagine that such a Perl Patterns Repository could serve as a useful community resource, operating as a kind of FAQ for information about Perl developer techniques, designs, and idioms.

However, for one reason or another, current efforts in this direction haven't generated much momentum. Since a key aspect of writing descriptions software patterns is exposing them to discussion, feedback, and improvement by other developers, aspiring Perl patterns authors might be better off contributing to existing repositories like the Perl Monks Categorized Questions and Answers.

Regardless of whether patterns documentation becomes trendy or not in the Perl community, I have been seeing more comments indicating curious interest or quiet enthusiasm for the topic, and I think we would be best served by remaining mindful of the original intent of the software patterns movement.

Further Reading

Perl Pattern Collections
* www.perldesignpatterns.com -- A large but somewhat jumbled collection with some nice advanced tricks.
* www.patternsinperl.com -- An older site; seems to be down.
* panix.com/~ziggy/perl_patterns.html -- Lists some Gang of Four patterns implemented in Perl.

More About Patterns
* perl.plover.com/yak/design -- The "Design Patterns Aren't" presentation.
* www.samag.com/documents/s=1280/sam02010001 -- An article about applying patterns.
* www.norvig.com/design-patterns -- A presentation on patterns in dynamic languages like Perl.
* hillside.net/patterns -- The industry association of the patterns community.

Some Perl Monks Threads
* Design Patterns Considered Harmful (Dec 20, 2001)
* Are design patterns worth it? (Aug 28, 2002)
* Perl Design Patterns Book (Oct 03, 2002)
* Perl Design Patterns (Jun 14, 2003)

edited by ybiC: fixed "logout" PM links