http://qs321.pair.com?node_id=366279

I used perl's flexible object system in the past to create programs that support plug-ins. This can be a very powerful feature, depending on how your program is meant to be used.

Recently, this subject came to my mind again when I participated in the Movable Type plugin contest. So I thought, why not let everybody know what can be done with perl to write pluggable programs? I wrote this tutorial, which is meant for people with some understanding of perl and packages, and I'd like to know what you think before I suggest it to the gods.

Well, here goes:

Writing Plugable Programs With Perl

Why plug-ins?

A plugin is just a way for people to extend your software without you having to care, or even know about it. It allows you to write an infrastructure over which people will have the freedom to do what they need. A good, well known example of plugins use is in web browsers (such as Mozilla) that just can't display every kind of content that exists in the web, so they allow plugins to do this for them.

The example program: complain.pl

In this tutorial we will attempt to solve a problem that we all faced one time or another, after spending a lot of time reading posts: How can we prove our superiority to the highest extent possible (find problems in the code) while wasting as little time and effort as possible?

The example program, complain.pl, will do this for you. It'll do more than that. It'll handle complain-fasion. You know, today you complain about 'use strict', tommorow it's efficiency, who knows what next? So for this purpose, finding a new reason to complain will only involve adding a new plugin to complain.pl - a plugin that somebody must have written already in a lively comunity such as Perl Monks!

so, let's introduce our program:

################################## # complain.pl - find what's wrong with others' code fast! # input should come from STDIN. # options: --plugin <name> - use the named plugin # --verbose - display some stuff use strict; use warings; use Linux; use Mozilla; # OK, ommit this one :-) # General tool stuff use Getopt::Long; my @plugins; my $verbose; GetOptions('plugin=s' => \@plugins, 'verbose' => \$verbose); # Get input program. my $snippet = eval {local $/; return < >}; print $snippet if $verbose;

What is a perl plug-in?

Essentialy, a plugin is just a perl package that complies to certain rules declared by the author of the pluggable program. These rules could be as simple as: "Sit in the plugins/ directory", "Put a line describing yourself in /etc/myprog.conf" or quite complex.

For our program, we'll decide that plugins reside in the plugins/ directory, and that's it. We'll also take a shortcut and assume that this directory in below the current directory. Doing the right thing here is left as an excersize...

Loading a plugin at runtime

Perl has a way of loading packages selectively on runtime: the 'require' keyword. unlike the familiar 'use', which is unconditional, happens before the program starts, and dies if it can't happen, require allows you much more flexibility. So you could write:

foreach (@plugin_list) { require $_ or warn "Plugin $_ failed to load"; }

And this is what we'll do in our programs:

# Load plugins @plugins = grep { eval{require "plugins/$_.pl"} ? 1 : (warn("Plug-in $_ failed to lo +ad.\n"), 0) } @plugins;

So we required the plugins and filtered out those who for some reason misbehaved. Which brings us to an important point: You can't let a plugin fail your entire program. That's why we used 'eval' here.

What require really does

require() is very similar to use(). They both let you include a module in this form:
require Acme::Current; # or use Acme::Current;
Except for the differences I already mentioned, there's one more important difference, in how the two are called:
use Module LIST; require EXPR;
As you see, require takes an expression. So we can put in, for example, require "plugins/$_.pl". If you use the form require Some::Module, then Some::Module is a bareword that is translated to a string.

Declaring the plugin interface

A plugin is only usable if the main program knows how to call it. So it is important to declare the way your plugin is used, so the plugin author could implement the required behaveiour.

For example, say we want our program to allow the plugins to affect it's initialization process. We'll declare that the plugins should implement an 'init' subroutine, and do this in our main program:

foreach (@plugin_list) { $_->init; }

This is sometimes called 'program events' that the plugin should react to.

For our program, we'll just execute the 'evaluate' method:

# Do the stuff foreach (@plugins) { $_->evaluate($snippet); }

You'll notice that there's a problem here: What if a plugin does not implement a certain feature? This will certainly die with "Can't call bal bla bla..." For 'evaluate' this isn't much of a problem, since if a plugin does not implement it, we'll drop him from our command line options. But say we wanted to let plugins declare their version on init time?

'can' to the rescue

You don't want to allow one misbehaving plugin to ruin your entire program, so we'll use a mechanism built into perl that allows us to check if a package can handle a certain method or not:

# Initialization: foreach (@plugins) { $_->init if UNIVERSAL::can($_, 'init'); }

The UNIVERSAL package is the 'top object' in perl, but it's also useful for dealing with simple packages, like we did here. the 'can' method of UNIVERSAL is what did the checking for us.

Now it keeps quiet.

Now we know what we expect our plugins to do and maybe we've even written some doccumentaion for others. Now let's write one. The 'bench' plugin will tell us exactly how slow a program is.

############################# # plugins/bench.pl package bench; use Benchmark; sub evaluate { my $pkg = shift; my $snippet = shift; print timestr(timeit(400000, $snippet)), "\n"; } 1;

You can try it now:

$ ./complain.pl -p bench < snippet.txt

Comunication between the plugin and the program

That print statement above must have caused you to scream to high heavens. A plugin should not decide on where and how to output its results!

OK, we need a way for the plugin to communicate the results back to the program. In our program, we'll keep a list of results, and declare the addOpinion subroutine for use by our plugin.

But we need to make our plugin be able to find this method. So what we'll do, we'll create a 'context object' - an object that is passed to the plugin and tells it useful stuff about the using program.

We'll add this simple package to the top of complain.pl:

package Context; my %versions = (); # The returned hash will store for each plugin the complaints it regis +ters. sub new { my $class = shift; return bless {}, $class; } # This will be optional. sub declareVersion { my $self = shift; my ($plugin, $version) = @_; $versions{$plugin} = $version; } # That's for our program sub getVersion { shift; $versions{shift()} } # This is how the plugin will tell us what he thinks, as many # times as it wants to. sub addOpinion { my $self = shift; my ($plugin, $opinion) = @_; $self->{$plugin} = [] unless exists $self->{$plugin}; push @{$self->{$plugin}}, $opinion; }

Now we'll just change the way we issue program events, so that a context object will be passed to the program. This is the new code, that we shall put right after the modules are loaded:

# Start a new context; my $ctx = Context->new; # Initialization: foreach (@plugins) { $_->init($ctx) if UNIVERSAL::can($_, 'init'); } # Do the stuff foreach (@plugins) { $_->evaluate($ctx, $snippet); }

And we'll fix the 'bench' plug-in to take advantage of the new and improved API :)

sub evaluate { my $pkg = shift; my ($ctx, $snippet) = @_; $ctx->addOpinion($pkg, timestr(timeit(400000, $snippet))); }

Wrapping it up

Now all we have to do is use the opinions expressed by our complaining plug-ins to generate a report. We'll add this final part to our program, in which we use the context object to check what happened:

# Simple output. while (my ($plugin, $opinions) = each %$ctx) { my $ver = undef; my $verstr = ($ver = Context->getVersion($plugin)) ? "(Version $ver) " : ''; print "The plugin $plugin ${verstr}says:\n"; print join '\n', map { "\t$_\n" } @$opinions; }

Just for fun: A plugin that checks if you used strict

This is the most fasionable complaint. Let's write a simple checker for this:
############################### # plugins/grammar.pl package grammar; sub init { my $pkg = shift; my $ctx = shift; $ctx->declareVersion($pkg, '0.1'); } sub evaluate { my $pkg = shift; my ($ctx, $snippet) = @_; if ($snippet =~ /use\s+strict;/) { $ctx->addOpinion($pkg, "Uses strict. OK"); } else { $ctx->addOpinion($pkg, "Does not use strict; Heathen swine!!") +; } } 1;

Note that this plug-in makes use of declareVersion, which is very considerate.

And that's all there's to it

Of course, this is just the most basic program I could think of. Better implementations will probably do a lot of things different, like creating a plugin object that plugins must inherit from, or do other fancy object-oriented stuff. But this is not the place for it.

You might want to take a look at the plugin interface of MovableType.

The distinction between use() and require() is documented in perldoc -f use and perldoc -f require. Thanks chromatic and tilly for your comments on this.

If anybody knows other good examples, please share.


perl -e'$b=unpack"b*",pack"H*","59dfce2d6b1664d3b26cd9969503";\ for(;$a<length$b;$a+=9){print+pack"b8",substr$b,$a,8;}'
My public key

Replies are listed 'Best First'.
Re: Draft - Writng pluggable programs with perl.
by chromatic (Archbishop) on Jun 13, 2004 at 17:52 UTC

    A few small nits. Be careful with this:

    foreach (@plugin_list) { require $_ or warn "Plugin $_ failed to load"; }

    As your next example demonstrates without explaning, perl evaluates require EXPR (where EXPR is not a bareword) to requiring a filename, not a module name.

    Also this code can be wrong in certain cases:

    # Initialization: foreach (@plugins) { $_->init if UNIVERSAL::can($_, 'init'); }

    Consider if I'd implemented my own can() method:

    #!/usr/bin/perl -w use strict; package Foo; sub new { bless {}, shift; } my $bar = sub { return 'Bar!'; }; sub can { my ($self, $method) = @_; return $bar if $method eq 'bar'; return $self->SUPER::can( $method ); } package main; my $foo = Foo->new(); if (UNIVERSAL::can( $foo, 'bar' )) { print "UNIVERSAL thinks Foo can bar\n"; } if (my $bar = $foo->can( 'bar' )) { print "Foo can actually ", $foo->$bar(), "\n"; }

    I realize that this is a silly example, but there are very valid reasons to override can(). I'd rather write your snippet as:

    # Initialization: foreach (@plugins) { eval { return unless $_->can( 'init' ); return $_->init(); } }

    If you document that init() should always return true, you can put a guard after that eval to see which plugins initialized successfully.

    Alternately, if you want to see that an object you've somehow created or received performs specific behaviors, you could use Class::Roles.

      Yes, good points. except for the 'can' thing. I usually assume (especially for a tutorial) that nobody is trying to fool me. If someone did repplace can(), he must know what he's doing.

      I'll fix the explanation about require.


      perl -e'$b=unpack"b*",pack"H*","59dfce2d6b1664d3b26cd9969503";\ for(;$a<length$b;$a+=9){print+pack"b8",substr$b,$a,8;}'
      My public key
        If someone did repplace can(), he must know what he's doing.

        Yes, but your code won't let him do it. If I'd written a plugin with a custom can(), to make it work with your program, I'd have to override UNIVERSAL::can()! Putting specific code to make a subclass work in a superclass (let alone the ultimate superclass) is bad design.

        I understand that your code tries to ensure that everything in @plugins actually represents a valid invocant. That's well-worth doing, but I think you're forcing behavior of the plugins beyond the minimum you need to do what you're doing.

        You've created a plugin system because you don't know what people will need to do to solve problems in the future. Don't enforce too many restrictions on how they'll solve those problems!

Re: Draft - Writng pluggable programs with perl.
by revdiablo (Prior) on Jun 13, 2004 at 20:15 UTC

    This is very similar to a plugin system that I built for (wait for it...) an IRC bot. I needed to be able to unload plugins, though. So instead of using require, I just used do. This got rid of the %INC check, which is nice, but for my purposes just a hassle. I then used delete_package from Symbol (this is a core module) to remove the plugin. Also, instead of running the subroutines immediately, I store the coderef returned by can() in a dispatch table.

    Obviously my plugins work a little differently due to their different purpose, but perhaps some others will find this information useful.

Finding plugins
by crenz (Priest) on Jun 13, 2004 at 19:54 UTC

    When faced with a similar problem not long ago, I created Module::Find to let your application find installed plugins automatically, without the user having to specify them on the command line. In my GUI app, I then let the user select the plugin he wants to use.

Re: Draft - Writng pluggable programs with perl.
by perlfan (Vicar) on Jun 13, 2004 at 22:13 UTC
    I can't offer any technical critique, but I do want to thank you for doing this. I have been wondering about how to create a plug-in system for sometime, and I will definitely use this resouce.

    Thanks, again.
Re: Draft - Writng pluggable programs with perl.
by dragonchild (Archbishop) on Jun 14, 2004 at 12:43 UTC
    Personally, I think your system is overly complex. A "plug-in" is basically an entity that conforms to a given API that can be activated at runtime. I call this concept "intelligent classes". For futher info, see Re^3: Style or Clarity?. Basically, this is an advanced version of a dispatch table.

    As for your passing information back and forth ... for the most part, this doesn't require a context object. It merely requires that all the plugins return the same data structure back from a given API function. A context object should only be used when there is actual context required. For further info, see what I did with PDF::Template and Excel::Template. I actually implemented a plug-in architecture for those classes, allowing for user-defined classes to be defined at runtime in order to handle custom tags. (The API is a little rough and mostly unpublished, but the architecture is still correct.)

    In other words, it's good to bring this concept into the fore, but you made it more complicated than it needed to be.

    ------
    We are the carpenters and bricklayers of the Information Age.

    Then there are Damian modules.... *sigh* ... that's not about being less-lazy -- that's about being on some really good drugs -- you know, there is no spoon. - flyingmoose

    I shouldn't have to say this, but any code, unless otherwise stated, is untested

      Well, of course I used a context object just for the point of showing what it means. The obvious place for it will be in recursively parsing some nested data, like in templates. I actually relied on what I saw in MT::Template::Context (from MovableType) when I wrote that part.

      If I only wanted to write a complaining program, it could probably be golfed quite a lot if reduced to the bare essentials (anyone wants to give it a shot?), but then It wouldn't teach us about much plug-ins, but rather about golf.


      perl -e'$b=unpack"b*",pack"H*","59dfce2d6b1664d3b26cd9969503";\ for(;$a<length$b;$a+=9){print+pack"b8",substr$b,$a,8;}'
      My public key
        You're missing the point. I don't care about your code - your design is too complex. There is no need for most of the trappings you're using. All you need is a set of classes / objects that all conform to a given API. They don't even have to be constricted to that API, or even do the same things when called.

        As for context ... context objects, imho, should be used only when there is no other option. They are basically global variables, with all the pitfalls that entails without any of the benefits. Plus, there's a ton of bookkeeping involved and every single object needs to know about it, and possibly every single method. It is messy, dangerous, and ugly.

        My point is that you're taking a system designed for a very specific purpose and extrapolating it out to a generic concept. That's a dangerous thing to do and you're doing it poorly.

        ------
        We are the carpenters and bricklayers of the Information Age.

        Then there are Damian modules.... *sigh* ... that's not about being less-lazy -- that's about being on some really good drugs -- you know, there is no spoon. - flyingmoose

        I shouldn't have to say this, but any code, unless otherwise stated, is untested

Re: Draft - Writng plugable programs with perl.
by tilly (Archbishop) on Jun 18, 2004 at 01:40 UTC
    Your update comment based on chromatic's comment is wrong. chromatic's comment was right, but you misunderstood it.

    It isn't that require doesn't take a module name, in fact it can take a module name as an argument, exactly like use does. However it only does that when presented with a bareword. If you don't have a bareword in your code, then the expression is interpreted as a filename.

    This is explained in the documentation for require. Look for the phrase, "If EXPR is a bareword".
      Yes, of course. how negligent of me. I'll elaborate on that.

      perl -e'$b=unpack"b*",pack"H*","59dfce2d6b1664d3b26cd9969503";\ for(;$a<length$b;$a+=9){print+pack"b8",substr$b,$a,8;}'
      My public key
Re: Draft - Writng plugable programs with perl.
by eric256 (Parson) on Jun 14, 2004 at 21:36 UTC

    Did I miss the part where you showed us the completed system?


    ___________
    Eric Hodges

      Since I'm not the master of literate programming (and in fact haven't tried it), and the program is pretty easy to assemble (and is mostly here line by line), I didn't think there is a need.

      The working code is available on my scratchpad.


      perl -e'$b=unpack"b*",pack"H*","59dfce2d6b1664d3b26cd9969503";\ for(;$a<length$b;$a+=9){print+pack"b8",substr$b,$a,8;}'
      My public key