The approach is a little different for each.

With Everything, there are pages with embedded code, embedded HTML, and calls to other nodes with embedded code. In the normal Engine, there's a regex that grabs token-delimited sections out and parses them accordingly, eventually finding and calling eval() on the embedded text.

A normal superdoc may end up calling eval() a dozen or more times. That can get expensive.

I have some experimental patches that pull out embedded code sections and construct an anonymous subroutine out of the whole document. It's compiled, once, and stored in a cache. (Luckily, the cache already had code to update the document text if someone edits it -- I got that for free!) All subsequent calls, while the node containing the embedded code is still in the cache, only have the overhead of a method call.

The tricky part is dealing with all of the different types of stuff. Embedded code is supposed to execute as a single unit (one entry point, returns a simple string), so I have to wrap it in eval blocks. (That's eval BLOCK not eval STRING.)

Within Jellybean, we don't allow embedded code. We have the beginnings of a scripting system where you build a method from named snippets. For example, you could open a form, display a message, use a textfield to get a parameter, show a submit button, and end a form with a recipe like the following:

    startform
    hello
    textfield
    submit
    closeform
[download]

All of a sudden, we have a new method. It doesn't have to be parsed each time. It's as fast as a built-in method. Users can't execute arbitrary code, because someone has to make it available for them to use in this method.

This is a powerful technique -- with not a few pitfalls -- but it's certainly an elegant solution for a few situations.

First, the generated code has several variable regexes, which may change during the life of the script, but not during one execution of the evalled code. For example:

      # code to match the stress regex
      $eval .= <<"      EOT";
            next unless /\$stress[$i]/o;
      EOT
[download]

    if (defined $stress[$i]) {
      # code to match the stress regex
      $eval .= <<"      EOT";
            next unless /\$stress[$i]/o;
      EOT
    }
[download]

  for (my $i=0; $i<=$#entry; ++$i) {
    # ...

    if (defined $stress[$i]) {
      # code to match the stress regex
      $eval .= <<"      EOT";
            next unless /\$stress[$i]/o;
      EOT
    }
  }
[download]

I also have an option that prints the generated code before it is evalled, to aid in debugging.

I really like generating code on the fly. It is fairly easy if you watch your back(slashes):

• generate a string with the code for a subroutine (that's the fun part)
• generate an anonymous subroutine from it $sub= eval "sub { $sub_in_a_string }";
• use the subroutine $sub->( @args);

et voila!

Here is a stupid example:

#!/bin/perl -w
use strict;

# very simple rules: s="<string>" or v=<number>
my %rules=( 's="toto"' => sub { print "toto!\n";     },
            'v=2'      => sub { print "waouh! 2!\n"; },
          );
# values to test
my @pairs=( toto => 2, toto => 1,  tata => 2,  tutu => 1);

my @rules= make_rules( %rules);                         # create the r
+ules

while (@pairs)
  { my $string= shift @pairs;
    my $value= shift @pairs;
   check_rules( $string, $value);                       # check all ru
+les
   print "\n"; 
}

sub check_rules 
  { my( $string, $value)= @_;
    foreach my $rule (@rules)
      { my( $you_like, $do_it)= @{$rule};
        $do_it->() if( $you_like->( $string, $value));  # run routine 
+if rule applies
      }
  }

sub make_rules
  { my %rules= @_;
    my @rules;
    foreach my $exp (keys %rules)
      { my $sub_str= ' my( $s, $v)= @_;' ;               # get the arg
+uments
        if( $exp=~ m/^s=(".*?")$/)                       # s="<string>
+"
          { $sub_str .= "return 1 if( \$s eq $1);" ; }   # watch for t
+he \, 
                                                         #    $s is a 
+variable in the sub
                                                         #     while $
+1 is a variable in make_rules and a constant in the sub
        elsif( $exp=~ m/^v=(\d+)$/)                      # v=<number>
          { $sub_str .= "return 1 if( \$v == $1);" ; }   # watch for t
+he \
        else 
          { die "syntax error in rule $exp"; }
        my $sub= eval "sub { $sub_str }";                # create the 
+anonymous sub
        push @rules, [$sub, $rules{$exp}];               # store it wi
+th the sub to run
      }
    return @rules;
  }
[download]

You can see an example in Ugly XML processing looking for a pure XML solution: I create a hairy regexp in make_wrapper, which is used in wrap -- calling the wrapper in wrap should really be written $wrapper{$tag}->( @_);

There are many basic approaches that I have used. Each has advantages and disadvantages.

1. You can write a little macro language which is turned into code and evalled. Damian Conway is fond of this approach. See for a random example, Class::Contract. Generally this takes a lot of work to do, and involves creating a macro language. But the result can be very powerful. (Usually I just use some subset of Perl as my macro language and use do or eval to parse and interpret it...hrm...)
2. You may have boiler-plate code to insert in appropriate places. See, for instance, AbstractClass. Outside of very rigid problems, this approach makes symbolic references look downright sane.
3. Your use of text manipulations to make code in one form turn into another is common. Particularly for mass edits, obfuscating code, so on and so forth.
4. Turning data structures into code and back is often very useful. See Data::Dumper. (I have used similar techniques to freeze data structures in one language and re-instantiate them in another.)
5. Often a large and complex script can be maintained as an auto-generated thing from a number of small ones through some sort of template. Perl's Configure shell script is maintained this way. Take a look at the make utility for some ideas on how to do that.

Hopefully this should give you some ideas for how to use automatic code generation, and a few places to look for examples of it being done.. :-)

--
$you = new YOU;
honk() if $you->love(perl)

Excellent topic! When I first saw this node, the first thing that came to my mind was compilers. Sure, they compile your programs to executables, but as anyone who's taken a compiler construction class knows, they can also work by generating code for a virtual machine (often some kind of stack machine), and then passing the generated code to a previously written implementation of the VM on the target machine (usually provided by the instructor, so you can concentrate on worrying about lexers and grammars and whatnot).

I used this approach in a Perl module implementation of The Dada Engine, which is a simple BNF-like grammar for writing out random text (example here). Instead of just writing a module that simply executes the grammar, the Perl port was written so that it translates the grammar to stack machine instructions. When these instructions are written out to an external file, along with the Perl implementation of that machine, you have a small and fast-running standalone script that outputs your random text.