At the risk of providing an inferior answer amid higher authorities, I'll try to explain the Schwartzian Transform.

First, one has to understand why it is useful. Assume you want to sort a list by, for example, ignoring the first character in each element. In other words, you want to sort 'Babc' before "Abcd". You could write such a sort like this:

my @sorted = sort { substr( $a, 1 ) cmp substr( $b, 1 ) } @unsorted;
[download]

And that works fine. But remember that the comparison routine of a sort function gets called many times. ...possibly several times per element being sorted. It's somewhat inefficient calling substr twice for each comparison, over and over and over again as the list gets sorted.

There is a better way. What if, instead of calling substr over and over again, multiple times per element being sorted, you called it once per element and stored the results? If you can create, store, and access this cached list fast enough, it will be more efficient than doing complex calculations over and over again on the same element throughout the course of the sort.

The Schwartzian Transform does this. It creates a cache of whatever complex computation you had in mind, and saves it alongside the original data, so that it's easy to keep track of the real data and its cached pre-computed sort criteria at the same time.

The most common form of the Schwartzian Transform takes a flat list, and turns it into a list of references to two-element arrays. Here's how our substr( $string, 1 ) might work in a ST.

@original       |                @transform
                |               [0]          [1]
_________       |         _________________________
abc             |               abc          bc
asdf            |               asdf         sdf
jkas            |               jkas         kas
qwert           |               qwert        wert
uiop            |               uiop         iop
[download]

So in the above example:

@original = qw/abc asdf jkas qwert uiop/;
[download]

...and...

@transformed = ( [ 'abc'  ,  'bc'   ],
                 [ 'asdf' ,  'sdf'  ],
                 [ 'jkas' ,  'kas'  ],
                 [ 'qwert',  'wert' ],
                 [ 'uiop' ,  'iop'  ]  );
[download]

Now lets look at the complete transform, as commonly seen:

my @sorted = map { $_->[0] }
                 sort { $a->[1] cmp $b->[1] }
                     map { [ $_, substr( $_, 1 ) ] }
                         @unsorted;
[download]

Look at this from the bottom up:

1. Start with a list called @unsorted
2. Feed @unsorted into map. The bottom map takes each element of @unsorted and turns it into an anonymous array with two elements. The first is the original string, and the second is the cached complex calculation.
3. Feed the list of array-refs to sort. Inside sort, dereference each array-ref to compare the 2nd elements (the cached versions).
4. Finally, feed this list of anon-array-refs (now sorted into the final order) into the top map. This final step strips away the cached information, leaving only the original strings (the array-refs get thrown away too).
5. The sorted original strings get assigned to @sorted

In actuality, this particular example may not be all that more efficient, since substr is pretty fast, and anonymous array creation and dereferencing takes time. But for longer lists, and particularly for more complex sort computations, the ST shines.

I hope this helps. :)

Where should I go from here?

The basics may extend past what you already know. Learn references. Learn map and grep.

Once you've brushed up a bit, start reading other people's code. The best/most interesting way to do this is to read Meditations, IMHO. Often times you'll find people abusing the language, which really exposes how things work. Certain people around here have a good history of writing interesting nodes. Some personal favorites:

• sexeger
• The N-queens problem using pure regexes
• May Thy Closures Be Blessed
• Parsing with Perl 6

"I heard the other day that he had rewritten eveything in ksh and deleted my work."

Quite a big surprise to me. I work in a place that Java and c are dominant, and they also have lots of ksh scripts. For every project I can decide the technical direction, I always require scripts to be done in Perl instead of ksh, and this is how Perl got introduced to the company, and slowly became popular. I believe that's the right direction. To me, Perl is a kind of programming language, but ksh is only a traditional scripting language that has very limited functionality.

Even under this direction, I will never ask anyone to rewrite exsiting ksh script in Perl, as that's a waste of time. We only use Perl for new scripts. So delete your stuff, and do a total rewrite sounds very stupid to me. Even if there might be certain things can be improved in your scripts, they should just fix it in Perl.

"Where should I go from here?"

Nobody else can judge better than you yourself, as only you know yourself and the place you work. Instead of learning for learning, why not just try to do more things in Perl, that's the best way to learn.

Also with the Perl knowledge you learned so far, come to this site more often, try to help people out, just as others helped you. Answering questions is also a good way to learn. Even give a bad answer and get corrected is a good way to learn.

Update:

Thought about it more, I actually have a question: when you said that they rewrote your Perl scripts (DBI) in ksh, what eactly did they do? (1) execute queries through something like SQL*PLUS (2) or rewrite the program in something like Java, and execute through ksh scripts? or ...

As far as I can work out, the wrappers were rewritten in ksh using calls to SqlPlus as you so sumised.

Neil

They are heading the wrong way, don't worry about it, you cannot worry about other's mistake too much.

The value of the S::T is that you do all your potentially expensive functions calls once per value, rather than many times.

Normally when you sort one list into another, the normal perl idioms go like this

my @output = sort @input; # default sort using eq to compare values
my @output = sort {$a <=> $b} @input; # numeric sort
my @output = sort some_func @input; # custom algorithm
[download]

Now lets have look at that last one - how often does some_func get called? Heres some code to show a potential worst case scenario

#!/usr/bin/perl -w

use strict;

my @input = (1..6);

my $calls;
my $fibcalls;

my @output = sort reorder @input; # how many calls to fib() ?

sub reorder {
  $calls++;
  return fib($a) <=> fib($b);
}

sub fib {
    my $val = shift;
    $fibcalls++;
    $val == 0 ? return $val:
      $val == 1 ? return $val :
    return fib($val - 1) + fib($val - 2);
}
print "called $calls times - fib calls $fibcalls\n";
[download]

Output is

cd /home/le6303/src/perl/
/usr/bin/perl -w "/home/le6303/src/perl/lister.pl" 
called 9 times - fib calls 150
[download]

OK, so a fibonacci sequence is exagerating the impact, but even so, what if some_func was doing some DBI-based lookup, using LWP or something else potential slow or expensive ?

Wouldn't it be better that we call some_func once for each unique value ?

Well we can. Looking at what we have so far, sort takes an input list (and potentially a function to operate on elements of that list) and returns a list. Wouldn't it be better if, instead of sort calling some_func repeated, often for a value it has already seen, we gave sort a list of values with some_func already applied ?

What we need is a function that takes a list, applies a function to each element once and once only, and returns a list of those function applications.

That function is map

We can write a trivial implementation of map thus

code>sub my_map { my ($function, @values) = @_; my @results; foreach my $value (@values) { push @results, $function->($value); } return @results; } </code>

Putting that together, we can show my_map applied to a list thus

#!/usr/bin/perl -w

use strict;

my @input = (1..6);

my @output = my_map(\&double, @input);

sub double {
    return 2 * shift;
}

sub my_map {
  my ($function, @values) = @_;

  my @results;
  foreach my $value (@values) {
     push @results, $function->($value);
  }

  return @results;
}

use Data::Dumper;
print Dumper \@output;;
[download]

Results are

cd /home/le6303/src/perl/
/usr/bin/perl -w "/home/le6303/src/perl/subber.pl" 
$VAR1 = [
          2,
          4,
          6,
          8,
          10,
          12
        ];
[download]

So to proceed, we want to pass the list of values that map has already processed to sort, and have sort compare those.

Like this

my @output = sort {$a <=> $b} map {fib($_)} @input;
[download]

But we have a problem now - @output is the sorted list of values that have had fib() applied to them, not the original list. If we put this together with our original example, we can see it more clearly

#!/usr/bin/perl -w

use strict;

my @input = (1..6);

my $calls;
my $fib_calls;

my @output = sort {$calls++; $a <=> $b} map {fib($_)} @input;

sub fib {
    my $val = shift;
    $fib_calls++;
    $val == 0 ? return $val:
      $val == 1 ? return $val :
    return fib($val - 1) + fib($val - 2);
}

print "called $calls times - fib $fib_calls\n";
use Data::Dumper;
print Dumper(\@output);
[download]

cd /home/le6303/src/perl/
/usr/bin/perl -w "/home/le6303/src/perl/lister.pl" 
called 9 times - fib 58
$VAR1 = [
          1,
          1,
          2,
          3,
          5,
          8
        ];
[download]

We only made 58 fib calls this time, but the output is wrong - it is the sorted fibonacci sequence, not the original list. How can we fix this ?

Map to the rescue. What we do is have the first map return a list of tuple's - in other word, pairs of values. The tuples are made up of the original value and the value with some_func (fib) applied e.g.

my @output =  map {[$_, fib($_)]} @input;
[download]

cd /home/le6303/src/perl/
/usr/bin/perl -w "/home/le6303/src/perl/lister.pl" 
$VAR1 = [
          [
            1,
            1
          ],
          [
            2,
            1
          ],
          [
            3,
            2
          ],

          [
            4,
            3
          ],
          [
            5,
            5
          ],
          [
            6,
            8
          ]
        ];
[download]

See how we have the pairs (value, some_func(value)) ?

So what we want now is for sort to use the second value in the pair to compare, but use the first value in generating the output list. Well sort wont do that, but we can use map again - it takes a list, does a manipulation to each element and returns a list of those manipulations. Code for that is

my @output = map {$_->[0]}              # extract just the original
                                        # value from the tuple
             sort {$a->[1] <=> $b->[1]} # sort based on the
                                        # computed value
             map {[$_, some_func($_)]}  # create (value, computed)
                                        # tuples
             @input;
[download]

Output is as required

HTH...

This was my way and I'm really happy with it. First, self-teaching (kinda falling in love) then good books (kinda getting to know closer and closer and eventually marriage).