Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

Templating algorithm - pass in variables vs callback?

by Tanktalus (Canon)
on Mar 01, 2005 at 00:03 UTC ( #435241=perlquestion: print w/replies, xml ) Need Help??

Tanktalus has asked for the wisdom of the Perl Monks concerning the following question:

This node started out as a reply to Re^2: Using DateTime together with Template::Toolkit. Then I decided it was sufficiently different from that thread...

I'm really curious as to why template modules take hashrefs. I notice most, if not pretty much all, template modules take this view. And it's driving me crazy. Perhaps I really am crazy to want something a bit different, and perhaps someone can set me straight on this.

Most template modules want you to pass in a hash of keys/values, or keys/AoH-refs for looping. My problem with this is that I much, much prefer lazy evaluation. For example, in some code at work, I need to figure out variables for thousands of iterations through this type of evaluation since I took the same idea, figuring that if everyone else was doing it, there was a reason. But only a hundred or so strings needed a lookup, and even then, only one or two of the variables needed to be there at the time. The effort in creating the hash was mostly wasted.

Here's an example of what I'm talking about. I have a list (whether on disk or in a database or in an xml file, whatever - it's been extracted to a list) of files to do something to. However, these files are libraries and executables. And I'm running cross-platform. So, say I have a list like this:

my @files = qw( [%libprefix%]foo[%libsuffix%] Foo[%exesuffix%] foo[%batsuffix%] );
Here, we have a shellscript/batchfile foo which is a wrapper around the Foo executable, which loads the foo library. Each filename is unique - at least on the platforms that I'm familiar with. So we have to come up with each of the variables:
my %params = ( unix => { libprefix => 'lib', exesuffix => '', batsuffix => '', }, aix => { libsuffix => '.a', }, hp => { libsuffix => '.sl', }, linux => { libsuffix => '.so', }, windows => { libprefix => '', libsuffix => '.dll', exesuffix => '.exe', batsuffix => '.bat', }, ); my %real_param; if ($platform =~ /windows/i) { #including Win/ia32, Win/ia64, Win/x86- +64 %real_param = %{$param{windows}}; } elsif ($platform eq 'AIX') { %real_param = (%{$param{unix}}, %{$param{aix}}); } # ...
Rather than doing things this way, I'd rather have a callback. I can go and retrieve the parameter from a database, for example, if, only if, and when it is required. No earlier, and no need to do so if the parameter isn't required. My callback can also use Memoize to cache the value to become faster. I did things this way for Re: String expansion script and am really liking it.

Now, I understand that in some cases, the overhead of calling a sub is greater than the overhead of creating a hash to pass in. But it's not really that difficult to use both. Perl6's multi-subs may make this faster, but even in perl5, checking that you have a CODE ref rather than a HASH ref and doing the right thing is not a significant amount of code.

Is this premature optimisation? Even if it appears this way, is it not more flexible? Any comments?

Replies are listed 'Best First'.
Re: Templating algorithm - pass in variables vs callback?
by Fletch (Bishop) on Mar 01, 2005 at 00:10 UTC

    This similar to the way Ruby does many things; not specifically code refs of course, but any method invocation can provide an associated block of code. The method in question can call back into this block using yield, passing it parameters and receiving the return value the block produces. Some methods behave differently depending on the presence/absence of a block. I've heard anecdotal tales of many Perl people toying with Ruby starting to use similar techniques (with coderefs, of course) in their Perl.

    So no, it's not that far out.

Re: Templating algorithm - pass in variables vs callback?
by lachoy (Parson) on Mar 01, 2005 at 01:29 UTC

    Since many templating systems allow you to pass in an object, couldn't you just bundle your desired functionality into it, pass that to the template and route all your dynamic, lazy loading calls through it? That seems a lot more straightforward than creating a separate way of passing parameters to a template.

    M-x auto-bs-mode

      It's not that I'm well-versed in many template systems because it has been a while since I looked at them all, but just going with the one I use the most, HTML::Template, for a second, I remember a method like this. Unfortunately, it scares me too much. It says that you can associate an object that has a param method like its own, one behaviour of which is to list all the parameters it has. And my dynamic method may accept hundreds or thousands of parameters, and looking them up is exactly the overhead I was trying to avoid.

      If HTML::Template said that it only needed the aspect of param which is to retrieve the value, that would be perfect. Even if that's all the code does (I've not checked), I would not feel comfortable about it until the docs were updated to give me some sense of security about the long-term stability of the requirement.

      Are the other templates out there doing something less invasive? I understand that this was intended to allow you to put your CGI object directly into the template to fasttrack some parameters, so the desire to change it would be quite low by the developers.

        No, that's not what I'm talking about. Systems like Template Toolkit allow you to pass an object to a template as the value of one of the hash keys you pass in. You can then call methods on the object from the template and they're called just like normal method. For instance, given the dummy class:

        package Foo; sub new { return bless ( { count => 1 } ) } sub count { $_[0]->{count}++ } 1;

        And the following template code:

        use Foo; use Template; my $template = Template->new(); $template->process( \*DATA, { bar => Foo->new() } ) || die "Cannot process: ", $template->error(), "\n"; __DATA__ I am calling methods on the object named 'bar': Call: [% bar.count %] Call: [% bar.count %] Call: [% bar.count %]

        You'll see:

        I am calling methods on the object named 'bar': Call: 1 Call: 2 Call: 3

        M-x auto-bs-mode

        Actually, HTML::Template allows you to pass in subroutines which will get called to get the parameter values:
        # with a subroutine reference that gets called to get the value # of the scalar. The sub will recieve the template object as a # parameter. $self->param(PARAM => sub { return 'value' });
        However, since these subroutines cannot be parameterized from within the template, I am not sure how much use this is.

        I'd love to be able to do the following (which is not supported, however):

        $tmpl->param( blah => sub { my ($a, $b, $c) = @_; return 'something'; } ); <tmpl_var name='blah(1,2,3)'>

        I think HTML::Template should have some simple kind of expression language that would allow you to these things, as well as drilling down into hashes and arrays, accessing objects (which should eliminate the need for callbacks), and doing simple arithmetics .

        HTML::Template::Expr is doing some of these things already.

        (Like everyone else, I have started to implement my own templating system with these ideas in mind. It is mostly a project to teach myself about Parrot, but if you are interested: )

Re: Templating algorithm - pass in variables vs callback?
by merlyn (Sage) on Mar 01, 2005 at 11:47 UTC
    In template toolkit:
    my $lazy = do { my $result; sub { unless (defined $result) { ... do expensive computation ... ... assigning to $result ... } return $result; }; }; my %vars = { lazy => $lazy, easy => 2 + 2, }; $template->process('', \%vars);
    From the template, you use both "lazy" and "easy" the same way. Except that "lazy" is a coderef. That's what's cool about TT: the call is the same.

    I seem to recall a module that does this for you. Yes, Data::Lazy.

    -- Randal L. Schwartz, Perl hacker
    Be sure to read my standard disclaimer if this is a reply.

Re: Templating algorithm - pass in variables vs callback?
by metaperl (Curate) on Mar 01, 2005 at 00:32 UTC
    I'm really curious as to why template modules take hashrefs. I notice most, if not pretty much all, template modules take this view.
    HTML::ELement::Library which is the main support library for HTML::Seamstress has mainly API functions based on callbacks and very few based on complete data.

    Per conversation with Fergal, Petal also offers callbacks.

Re: Templating algorithm - pass in variables vs callback?
by perrin (Chancellor) on Mar 01, 2005 at 04:51 UTC
    As others have mentioned, Template Toolkit lets you have it both ways. The main advantage though of pre-calculating answers is that you often know better than the template how to get the data efficiently. For example, you might use one large SQL query that joins several tables to get all the data you need at once, rather than using multiple queries called from the template. For your example with the thousands of iterations, the solution would be to only calculate the ones you need before running the template, rather than doing all of them.

    The downside of this is that it tightly couples your code to the template.

Re: Templating algorithm - pass in variables vs callback?
by ikegami (Patriarch) on Mar 01, 2005 at 02:19 UTC
    For modules that require a hash ref, you could tie the hash ref to get the functionality you desire.

      Almost. Some of the templating modules (e.g., HTML::Template) will query all keys to compare to all the parameters in the template to make sure of a one-to-one match which I find equally annoying, but that's for another rant - the error message from the check is at least quashable. I'm unsure of whether the parameter that quashes the error message also quashes the check.

      That said, HTML::Template also only takes a hash, not a hash ref, so your comment doesn't quite apply there - which modules take a hash ref? I would think that most modules wouldn't since they need to allow you to set the parameters multiple times to add new parameters from various sources. Yes, you could do that yourself and pass the whole thing in, but that's quite a bit less extensable. And then it would take the hash you give it, tied or not, and copy it into itself. It's sounding a bit more work than submitting patches to allow callbacks :-)

Re: Templating algorithm - pass in variables vs callback?
by talexb (Chancellor) on Mar 01, 2005 at 04:51 UTC

    Well, with a wave of one's wand, a hashref can become an object, thus an expensive result can be cached after the first call. That's something I certainly take advantage of as I use Template::Toolkit for my web application.

    In your case (above), where you want to do something different based on the platform you're running on, it seems you'd know what platform you were on before you got to the template stage .. unless I'm missing the meaning of your example.

    Alex / talexb / Toronto

    "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

Re: Templating algorithm - pass in variables vs callback?
by sgifford (Prior) on Mar 01, 2005 at 06:23 UTC
    When faced with a similar problem, I found the best approach was to ask the template for a list of all variables it was going to use, pull in just those values to a hashref, then display the template with that hashref. For an SQL backend, this mostly meant adding in the right tables and columns to a SELECT statement.

    This was using custom template code; not sure whether your system will allow this or not, but it seems like pretty basic functionality.

    As for the popularity of the hashref approach, I think it's intuitive, and it's also quite hard to screw up from a security perspective. It seems like callbacks would be much harder to get right in this way; it seems like a small mistake or oversight on the part of the template code's author could easily let the template call arbitrary code.

Re: Templating algorithm - pass in variables vs callback?
by fergal (Chaplain) on Mar 02, 2005 at 00:09 UTC
    It's been mentioned already but I thought I'd give an example in Petal
    my $t = Petal->new($template_file); print $t->process( user -> $user, );

    Here $user is an object that has many methods that return strings or arrays or other objects. In a Petal expression "user/name" will result in a call to $user->name. "user/account/balance" will result in a call to $user->account->balance. Since these are methods, they can be as lazy as they like, they can pull things out of databases at the last moment, perform complex calculations etc.

    In Petal the syntax for hash access is the same as method call (basically it calls a method if it can otherwise it treats it as a hash key) so it's easy to switch between them. Particularly handy when you're protyping and you want to pump some sample data into your template to make sure everything's OK.

Re: Templating algorithm - pass in variables vs callback?
by dragonchild (Archbishop) on Mar 01, 2005 at 13:55 UTC
    There are actually some formats that, while not requiring it per se ... it just really helps to know about everything ahead of time. I know it helps me in PDF::Template to have all the data ahead of time. This is because it's a paginated format, which means I need to calculate page numbers and the like. Knowing how much more data there is allows me to calculate the final page number, so you can do things like
    Page 1 / 45
    and have it look right. (I'm sure there's other ways to do it, but that's what I've got so far.)

    As for HTML::Template, you might want to look at HTML::Template::Expr if you want expanded functionality. Or, just take the plunge and go to Template Toolkit.

    Speaking as a template author, callbacks can be ... problematic. They're not easy to write an API around. Just a few problems I have with them:

    • Are all CODE references to be considered callbacks?
    • Are callbacks allowed in any type of variable? Scalars are easy, but what about loop variables that need to return a hashref for the current iteration?
    • What happens if a callback dies? Should I call the callback within an eval block? What do I do with $@?
    • Should I allow for parameters to be passed to the callback? What is the interface within the template for that?
    • Do I allow for eval within the template? How does that play with mod_perl and other persistent environments? What about safety concerns - templates are sometimes under the control of another group from the Perl developers ...

    Not so simple ...

    Being right, does not endow the right to be rude; politeness costs nothing.
    Being unknowing, is not the same as being stupid.
    Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
    Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

      Hmmm - granted, some data needs to be determined ahead of time. But, to answer your questions:

      • CODE references which are being interpolated into a string should be considered callbacks 100% of the time. If you want to actually put in "CODE(0x1234578)", stringify the coderef first. Perhaps in other contexts, having a code ref is more questionable (say, serialisation), but I am having a hard time figuring on when passing in a code ref is going to be anything other than a callback. The callback may not be immediate - it may just be stored for future use (e.g., in Tk), but it will be called back iff the situation arises, which is no different than this - if there is no variable in the template, the callback doesn't get called.
      • In Re: String expansion script, I had a callback which could return a list. Perhaps you want to give your users an API to fill that says all returns are scalars - lists and hashes aren't supported. Perhaps they're scalars, but lists must be returned as array refs, and hashes as hash refs. Whatever it is, I do think that this is something that we, as a community, should try, then iron out. Like many other idioms, which sprang to existance from trial and error, I think we could end up with a very flexible and clean interface here. Heck - it could be based on XPath! :-)
      • If a callback dies? What happens if a hash sets a value to undef? The callback can return undef, too, I suppose, so it's not quite the same. I would think it completely reasonable to expect the callback to catch all exceptions and deal with them, or, if the callback propogates the exception (that is, doesn't use eval), it would be completely reasonable for the template to propogate it as well. Which is not significantly different from operations today.
      • Allow parameters? Hey, that's cool. But let's take it one step at a time, ok? :->
      • Eval from within the template? I'm not talking about allowing the template author to embed perl in the template (there are template managers for that already). I'm talking about straight variable substitution. If the developer who is using the template manager is concerned about safety, and doesn't think that the callback is safe, don't use it. I can't see how that would be the case - all the safety would be inside the callback. The developer then gets to tell the HTML author(s) what tags are allowed, and what they do. If the template code allows for parameters, but the callback doesn't, then the perl developer won't tell the HTML authors about parameters. If parameters are allowed, it's up to the callback to ensure the safety. If the perl developer allows a variable substitution of eval("system(q(rm -rf /))"), that is not the template manager making things unsafe - that's the callback developer making things unsafe. Treating the variables as tainted is almost as recommended here as it is for dealing with end-user text from a web form. I would hope that the html template authors would be on the same team as the perl developer, though ;-)

      Problems aren't often simple - but their solutions often are ;-)

        • I can accept that all code references are callbacks. That's not an expensive check to do.
        • What should the template engine do if you pass a callback in for a loop variable and the callback returns a scalar after 10 iterations?

          Actually, upon thinking about this, I just realized I have a bug in my templating engine in that I don't check to see if the value in the looping array is a hashref. Thanks! :-)

          So, if the callback doesn't give the engine what it needs, just treat it like a regular failure. Ok ...

        • I'm actually thinking that a callback dying is your damn fault, not the engine's. So, the engine shouldn't catch the errors.
        • Parameters ... the big problem isn't passing them around, it's figuring out how to specify them. I think it's better if you pass in a CODE ref that already has the parameters for your callback specified, at least for now.
        • I think that eval shouldn't be allowed, period.

        Note: While researching this response, I discovered that HTML::Template will do the right thing when passed a CODE reference for a TMPL_VAR or TMPL_IF, just like you want. But, it won't allow a CODE ref for a TMPL_LOOP. I'm sure we could provide a patch for Sam, but I have my own engine to patch. *sighs*

        Being right, does not endow the right to be rude; politeness costs nothing.
        Being unknowing, is not the same as being stupid.
        Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
        Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://435241]
Approved by Fletch
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (3)
As of 2022-05-20 20:54 GMT
Find Nodes?
    Voting Booth?
    Do you prefer to work remotely?

    Results (76 votes). Check out past polls.