If you were using HTML::Mason, it comes with a caching mechanism which lets you specify a sub that determines when a pre-existing copy can be used and when it needs to generate a new version.

Also the mod_perl Developer Cookbook (ISBN 0672322404) has examples of handlers which create cached copies.

Generating static HTML can be a big win for a high-traffic site that gets many more hits than it does updates.

For a very elegant example of how to generate a static site from dynamic content, download and examine Moveable Type, which is a blogging package written in Perl (by btrott). It builds a static site on demand whenever dynamic content is published.

One of the sites use the approach that it will regenerate the appropriate pages when the administrator updates something - this can involve adding a new news item, or adding a new object for customers to view. This has pretty low frequency, and for all practical purposes, there is only one administrator, so there are no concurrency issues either (although safety has been built in in case they would suprisingly expand a lot or something). All the changes made are saved into a database, and upon pressing "publish", recently changed pages will be regenerated. Since visits are really plentiful when compared to the updates, this is an approach that works really well. Matter of factly, this kind of approach could mean you could have a site on a host with no dynamics at all - you generate the pages from somewhere else and put them there via ftp, scp, rsync or whatever is appropriate. :) In this case though, there are dynamic parts, which includes a browsable web map and customer logins. These parts are purely dynamic, partly for session stuff, and partly because this is still low traffic parts.

The other approach is an hourly update from the article data base, to the web shop. Not the same company though. When a user buys something, or searches etc, dynamic pages are generated directly from the database, but when they are just surfing around, they see static content. This has some issues when dealing with discrepancies between static and dynamic content upon rare occassions, and we are not satisfied with the solution. However, we picked a bad platform for the job, it couldn't handle the huge load any other way, and we hopefully know better now. Mistakes are good for learning, if not for confidence... *grin*. And no, I am not allowed to fix it. :(

Anyhow, given the right circumstances, those two approaches really works well when you are worried about heavy load.

Your third suggestion was to query the DB, or disk, every time someone wanted a page and either return a cached page, or generate and return a new one, right? Well, I still advocate updating on change instead of polling for changes, given that changes are more rare than views. Then the views are without any load at all, while the fewer updates have some load (but not a poll for changes).

However, depending on the expected load, I am not sure that any static HTML approach is fitting for a forum. If load is high enough, you will have lots of concurrency problems, and you will need to lock the files on disk and other such things - effectively slowing things down a lot again, while they wait for each other. If load is low enough for this to not be a problem, then load is probably not a problem for generating everything on the fly either.

It would still depend upon the forum, or whatever, but the smaller the difference between updates/reads, the worse is a static HTML approach imo, no matter what the expected load is. And forums are generally something that gets updated pretty much compared to just viewed - unlike a news site or similar. :)

If you have the time and possibility - try both approaches (static and dynamic). Otherwise, try to think what you can gain/risk/lose on either approach or a healthy combination.

This probably won't help much. I still hope it will. :)

What I mean by that is design your system so that you can tell when a change has been made, so you always know when you need to regenerate your HTML.

So to translate this to your system, when a thread has an addition, generate the HTML code, store it, and mark the thread as clean.

Next time that thread is viewed, you only need to make a single database lookup to decide if you can re-use the HTML or not. If it's still clean, go for the cached version.

When additions are made to the thread, simply mark the thread as 'dirty', and the next time it's viewed, the HTML will be generated again.

Issues to think about here are concurrency, and also making sure everything that can affect the thread updates the dirty flag.

And as for optimizing my mysql queries, can you point me to a good reference or give me some tips?

First, joealba had mentioned to make sure that your database tables have good indexes and keys - and he is right.

From experience (of not doing this the first time around) I would add to make certain that your database is properly normalized BEFORE it gets large and painfully slow to query.

There are plenty of good places to look for info on this. Go to Google and search for 'MySQL database normalization' -- for that matter, look at mysql.org for hints.

You know that you are in good shape if you can get whatever you are looking for with an absolute minimum number of query statements using table joins. One query is obviously ideal. If the query is slow, you may want to rethink your keys/indexes.

The performance benefits of a persistent database connection has already been addressed. If you can, I would suggest using it.

my $.02

This is where you look at a performance analysis of the item. It depends on your your caching scheme and the nature of your data. There are a few basic principles behind a cache mechanism that you need to take note of (this is midly off topic). Keep in mind always that a cache is the epitome of the classical space/time tradeoff. With that in mind:
• Is the data that you are generating dynamically mostly static or mostly dynamic? This is really important to consider. If you are generating basically the same thing over and over again, then it's fairly cache-safe. If it changes a lot (has the times the poster last logged in, how many posts they've made, or replies come in often), then your stored information has the probability of being "dirty", and thus useless to store somewhere.
• Is the overhead of the caching worthwhile? Are you typically going to do a lot of mucking with the data to store it? Are you putting the cache on disk or in a database? These are questions you will have to answer for yourself to come up with the right answer, but performance tuning will be involved. Start the caching, and see what hits the cache and what misses it.
• Can you afford the space/time tradeoff? Do you have the space available to store the cache? Do you have a mechanism in place to clean the cache once it becomes to old (to keep it efficient)?


Those reflections should be able to answer your questions for yourself. If you're looking into perl modules:
• HTML::Mason
• IPC::ShareLite (if you're brave)
• Storable (can I get an AMEN!)
Or you can roll your own solution taylored to meet your needs. Have fun with the project

    --jb

The Cache::FileCache (part of the Cache::Cache family) module is worth looking into as well. Most modern filesystems have very efficient algorithms for keeping MRU data in memory, making it very speedy and simple to boot. (IME shared memory isn't worth the effort.)

Chris
M-x auto-bs-mode