Masem has asked for the wisdom of the Perl Monks concerning the following question:
Now, reading through a recent entry at perl.com on the new Template module, I got to thinking how I may be able to take advantage of this: I certainly could use the ability to use templates to produce various results of the DBI output that I'm getting. However, I'm concerned about speed -- with preliminary scripts running under a mod_perl-enabled virtual host of apache, I seem to be getting no speed benefit, yet. Mind you, two possible slow effects are occuring: I'm using fetchrow_hashref to get the values out from the DBI call, which might have some overhead, and after getting the row data, I do additional processing (more DBI or file reading) for supplimentary stuff. And furthermore, some of the formats for the Templates that I have planned will not use all this data. The data is simple rows, no complex data structures outside a few join'ed arrays.
In other words, I have a tradeoff between a concise and modular function that does the results display but runs slow, and to use several different functions for each list type but most likely a benefit in speed. I'd rather avoid the latter case, but I'm finding right now that even with queries of 5 results, the results take a long time to process. Is Template just not the right tool here, and should I fall back on just using something similar to here documents for each row?
|
---|
Replies are listed 'Best First'. | |
---|---|
Re: Speed of Template
by chromatic (Archbishop) on Feb 09, 2001 at 10:18 UTC | |
Besides that, though I have seen Template Toolkit slow things down, the TT2 does a compilation/caching stage. In a persistent environment, you'll only pay the penalty of caching templates once, then it's amortized over the rest of the calls to that template. It may not run quite as fast as a heredoc, but it has the potential to do many more complicated things. If that weren't enough, it's much easier to change a template than it is to change a heredoc. An hour's worth of my time would buy extra memory. A new hard drive if I have to wear a tie. I would recommend that you use Template for its many benefits, only then worrying about speed. It's not *that* slow, if you do it right. The difference between 5 hits a second and 10 hits a second may not be important when you realize how many hundreds of thousands of hits per day you'll get by that point. | [reply] |
Increasing Performance of mod_perl/CGI/MySQL programs
by dkubb (Deacon) on Feb 12, 2001 at 02:26 UTC | |
mod_perl/CGI/MySQL programs are interesting applications to attempt increasing performance of, there are so many moving parts, you need to take a more holistic approach. There's generally a combination of several factors affecting performance. Warning: This will be a long post (and possibly slightly OT), because I am fairly interested in this subject =) Below is an outline of the areas I believe you need to consider when attemping to increase the performance of your perl database interface. Some many not be possible depending on the resources/time you have available, but I wanted to outline everything I could think of, and allow you to choose what is possible: So, you're database and web server are running at peak performance. Your perl code is optimized and profiled. Want more speed? Now you need to start looking at what happens after the information leaves your server. Whew! Sorry for the length of this post. Once I started writing I couldn't stop. Perhaps I should put this into a tutorial. Anyone have other performance improvement suggestions? | [reply] [d/l] |
by tilly (Archbishop) on Feb 12, 2001 at 02:49 UTC | |
To this I would like to add a couple of more relevant links off of my home page. First of all Code Complete has a sample chapter online on Optimization that is worth reading. Secondly those who want to get into great depth on how to design web-servers and network protocols for maximum performance may find many things of interest in The C10K Problem. Another good reference is this rant on Latency vs bandwidth. Many slow applications are slow because of heavy duty interactions across something with bad latency, and not because of lack of bandwidth. | [reply] |
Re: Speed of Template
by PsychoSpunk (Hermit) on Feb 09, 2001 at 10:26 UTC | |
Have you tried comparing the results of the same calls using fetchrow_array? Template is a great tool, but don't buy into it if it's simply because you read so-and-so say that it's the best thing he's ever used. Buy into it if it's going to make your life easier. That said, I bought. I'm trying it out on a new project at work, and I am pretty pleased with the results. And yes, I'm using the DBI Plugin that comes with it. I found that I was able to wipe out my server when I attempted to do some 250+ SQL commands in one script, but I guess that's to be expected. :) (don't ask, I was just playing around with something to find out if it would) Is there a marked improvement? I doubt it, at least if you're currently running mod_perl and your scripts use Apache::DBI or even plain ol' DBI. But coming from straight use CGI; and use DBI; stuff, I see improvement. Update: yes, chromatic has a good point from the DBI book, but I'm going to guess that the point of that statement in the DBI book is related to anyone who wonders why the server mysteriously locks while pulling thousands of rows back for data. (Once again, it was just to see what my limits were :) Jeez.) ALL HAIL BRAK!!! | [reply] [d/l] [select] |
by Masem (Monsignor) on Feb 09, 2001 at 17:28 UTC | |
The other reason that I'd want _hashset is that I'd like to toss some more things into the hash that I get back from it after the db, and as such, I can just take that hash ref from this call and not worrying about creating a new hash until the next row. But again, it comes down to the fact that running a query with 1-5 results seems to take as long as a 1000 result one based on how my code is called, so I very much doubt the DB is the bottle neck, and instead Template being the problem. I've made sure to put the template calls outside of the loop (that is, make an array of the hashrefs from above, then let Template handle the for-next'ing it). From chormatic's post, it also suggests that caching of the templates should be going on if the envirnoment is persistent -- while I've folled the mod_perl docs and the like to modify the Apache conf, I don't have an easy way to test if I have created the right persistent environment. | [reply] |
by arturo (Vicar) on Feb 09, 2001 at 20:58 UTC | |
As far as getting the column titles out correctly goes, I'd suggest something along the lines of putting as much of your DBI stuff in a single module (an OO one if you like) as possible. That allows you to 'encapsulate' (love that buzzword) your database code. In that module, or elsewhere, you could use the constant pragma to alias column numbers to memorable tags. You'd only have one place you'd have to change, and you'd get the speed advantages of fetchrow_array over fetchrow_hash. e.g. :
Or some such. Just a thought. Philosophy can be made out of anything. Or less -- Jerry A. Fodor | [reply] [d/l] [select] |
Re: Speed of Template
by perrin (Chancellor) on Aug 15, 2001 at 10:23 UTC | |
| [reply] |
Re: Speed of Template
by Anonymous Monk on Feb 10, 2001 at 07:39 UTC | |
It generally ends up looking something like this:
Or something like that, the only downside being: if I change the SQL query in my dbh->prepare() call, I'll have to make the corresponding changes to the order hashref above. This saves the DBI overhead of having to check the column names for each query, plus you get the benefits (IMO, at least) of pseudo-hashes over plain hashrefs -- wether or not you like them is up to you, here is one small discussion on the matter: Are pseudo-hashes worth the effort? | [reply] [d/l] |
Re: Speed of Template
by EvilTypeGuy (Initiate) on Feb 10, 2001 at 02:38 UTC | |
| [reply] |
FOLLOWUP
by Masem (Monsignor) on Feb 14, 2001 at 23:34 UTC | |
First, I did find that SQL queries were slowing me down : I was trying to be cute with INNER JOINS, but after recognizing that key searches that I'd want to do would require 2 or 3 inner joins to do it, and this ate up alot of time. I figured that most of the data that the frequent searches will be done with is static except when *I* change it (infrequent or as necessary), that I'd create a 'cached' column of data of what I'd normally get out of the JOINs, usually as a /;/ list, so that I can then use REGEXP to get the data that I need instead. I can then use secondary SQL calls to get any additional data out of the table. It's arguable that these secondary calls may increase runtime, but it increases the modularity of the CGI/perl stuff, since I use similar calls at other non-searching parts of the site. (As another optimization point, I would think that if I kept around a global variable for the $sth for these utility functions, since the query format doesn't change, only one placeholder, I'd get a bit more optimization... must try that tonight... :D) Template is *wonderful* to work with. It allows me to have minimal HTML code, even through CGI.pm, in the perl code, and thus provides yet another degree of model-controller-view. And yes, once I worked out the SQL problems, it was sufficient fast. Plus, Template has enough logic possibilities built into it that I can use more general results as outputs from my modular perl functions (refs to arrays and hashes as opposed to strings) and decide how to deal with them when I tighten down the presentation issues of my site. For example, one of the results of the faux JOIN above is a list of id number, each which, in a separate table, has a name; these can lead to individual pages which can be linked too, however, I may not always want to link to such in my code. I *could* write two functions in perl, one for the non-linked list, one for the linked list; I could supply the extra parameter in the function and handle it as such, but I think the most generic way to go is to return a hash, as I can still grab the list of names via a join(values()). But since Template can iterate through hashes, I'll just pass it to the template file, and then handle it there. Oh, but the possibilities..! I'm glad that the suggestions here made me stick with it -- the time of development right now is much faster than if I stuck to only perl and CGI for code generation. I'd be happy to share specific details of my experiences if anyone is in a similar position. | [reply] |