might it be a good idea to start from scratch with a new engine, with in the beginning far less features, but FAST?

No. The Everything Engine had been under development for around two years by the time Perl Monks came about. That was over three years ago. In my opinion, throwing away five years of development, when you have a working product, is a terrible idea.

no more eval()

It was a deliberate design decision that administrators should be able to create nodemethods that exist only in the database. It would be nice to have the option to specify that you're not using that feature for a big performance gain (and I'm working on that), but forbidding it altogether would be removing one of the most important features of Everything.

implement some smart caching between application and database

This is a much better idea. It's also a hard problem, with Apache and mod_perl. Threaded Apache 2.0 may help immensely. I'm also experimenting with a standalone forking server that has much more control over shared memory. Improving the existing node cache may save a bunch of time.

I doubt you have profiling data, though, so I feel at liberty to say rewriting the engine from scratch is a terrible, awful, lousy idea. Just this weekend I checked in code that improved the second-biggest timesink subroutine by 50% -- without removing a single feature.

Update: I forgot to bring up the idea of data migration. Good luck.

The speed issues may be fixable without a rewrite, but frankly the Everything code is not very well suited to its current use on PerlMonks. It has some fundamental design decisions (keeping the code in the database, doing most of the data as generic blobs of XML) which are a major cause of slowness and race conditions, but more importantly make it really hard for most people to contribute to the code and make testing nearly impossible. How could anyone profile this code effectively? Just retrieving it to run on your own system requires significant work (because you have to get it out of the PerlMonks database and test data is not available).

It's very cool that the system was designed flexibly enough to work in this way, but a more focused codebase that works specifically for PerlMonks would be able to run much more efficiently. PerlMonks is essentially a separate codebase now, since it branched off the Everything codebase a long time ago and is not able to take updates from that code unless someone manually merges them in.

I would like to believe that a gradual process of rewriting could fix these issues, but I'm not sure it will because the things that need to be changed are so fundamental to the current design. Your point about all the accumulated knowledge in this code is a very good one though, and not to be dismissed lightly. Rewriting would be a lot of work and it would be hard to get all of the current functionality right. Migrating the data would be REALLY hard.

Caching with mod_perl, on the other hand, is trivial. I gave a talk about it at OSCON last year and I'd be happy to help if you have questions about it. Tye was concerned that using shared caching anywhere other than the nodelets would make the race conditions worse, but doing the caching itself is simple. (Of course caching across a cluster is hard, but that has nothing to do with mod_perl and may not be required.)

I profiled the current CVS in single-user mode on my laptop this weekend. I'm not really concerned about any one specific site, just the framework and general behavior. If I can speed that up, I'll have met my goal.

I'm not terribly concerned about the XML, though using XML::DOM is a performance killer. That's mostly during the installation, though, so it's a low priority along the performance axis. The only place it's really used internally in the live system is in the workspacing code, and I don't think there's any of that on Perl Monks at the moment.

Caching is complicated by the fact that the current CVS has subrefs in it. That's why I'm betting my managed-forking approach will have better performance in certain circumstances.

The performance killers, as I see them:

• pages are optimized for writing -- parsing links every time, processing page templates on every hit. This is ameliorated somewhat by code caching in the 1.0 series
• nodelets are cached for the whole site or not at all -- they could be cached per user for a speed improvement
• nodes have a custom inheritance scheme to deal with nodemethods, which was between 10 and 20% of the profiled time in my tests -- this could be reduced further
• inefficient database queries, fetching hashrefs when bound scalars and explicit column names are 20-50% faster -- I'm working on this
• inefficient code -- we're better programmers now

I'm working on all of these, but it's at the tip of CVS. My goal is to make migrating to Everything 1.2 an attractive option for Perl Monks.

No, I said that making the node cache shared between processes would be bad (prevent improvements that would reduce the impact of its race conditions and reduce database server load).

I mentioned some types of caching besides nodelets and noted that they wouldn't likely be a big win. I didn't say anything about "anywhere other than".

I also note that chromatic appears to mostly be talking about load that we'd see on the web servers, which, last I checked, wasn't where the main problem is.

                - tye

In my opinion, throwing away five years of development, when you have a working product, is a terrible idea.

So Microsoft shouldn't throw away their IE code? ;-)

Also, you wouldn't have to throw away the other code at all, if someone's interested in starting a new project, go for it. If the code ever reaches the point that it's superior then use it. You don't have to decide to switch it before the alternative has arrived. Did you hear people saying "I'm going to switch to Linux" after Linus' first email on the subject?

I wouldn't go so far as to call IE a working product. Efficiency and correctness are two very separate issues.

Makeshifts last the longest.

I think the answer isn't really technical, or at the very least, not a problem with Everything, apache, or perl. All indications are saying the problem is with the database server getting overloaded. First off, mysql on freebsd isn't a happy thing, due to differences between Linux's threading model and freebsd's. The recent kernel upgrade should have helped with that, but it's still not good, just better. Secondly, trafic is increasing, possibly to the point where we need either a better DB server, or two DB servers. Two DB servers would probably require changing the way we do certian things, but not the point of rewriting everything.

Yeah, there are certian things that Everything could do better... but you should fix it, not rewrite it. Caching the code as bytecode, instead of evaling it, would help, but that's not a reason for a complete rewrite, just a fix.

There are some people (such as newrisedesigns), who turn off all of the server-intense nodelets and maybe not use Newest Nodes and Super Search. Okay, so some people are trying to be 'environment-friendly' here. My question to such people is this: are you turning off those features because you don't use them or because you're trying to help the server out a little?

The answer to this question could raise some more questions about this topic: if you're disabling these features because you honestly never use them, then good for you! You're not putting any more stress on the server than what you want to use. If it's just because you're trying to speed up the server however, that's just the wrong answer. You cannot just say "turn off all of the good features of the site and then things will run faster". The correct solution we need to come up with is not that of "remove some features", but that of "improve the mechanics of these features so that they cause less trouble".

Upon a minute or two of reflection, here's a couple of questions I have come up with:

• Nodelet caching: there are some nodelets that don't need up-to-the-minute, real-time display. Take Daily Best and Weekly Best as examples. Each time I load a page, is this list researched and populated from the database or is it quickly retrieved from a cache that is updated every 15-20 minutes or so?
• Database held in RAM. I am curious to know whether the database itself is held in memory or if it is continuosly read freom disk. When I request a page, does the hard drive spin around and around to locate certain sectgions of the database, or is the database held in memory in such a way that the hard drive is only accessed for writes to disk? One point I do not know is how big the database is at the moment. Is it so big that it would be too much to ask to add some more RAM to the system to hold any/more/all of the database in memory? I got this question from reading (one of?) Philip Greenspun's book(s) in which he had a site that was managing many requests per second, while only using about 10% of the server's resources and hardly hitting the disk. Sounds like fantasy world to me :)
• Static pages: I was thinking about this one for a bit until I realized the worst part of wanting to turn some nodes into static pages: the nodelets. I already knew it would take a lot of time and effort to turn posts into static pages instead of needing hits to the db, but I thought I would mention it. Then I realized that the whole right-hand menu would not update at all, and I thought "oh, dumb idea then". Then I thought of maybe using a CGI include for that, then I thought "no, still stupid". It has been mentioned before about using the existing static backups of the site, but how many people really use these?
• Caching, caching, caching. I think a lot of the improvements that could be made would rely on caching things. From nodelets to nodes, maybe even search results, to who knows what else, I think there are probably many areas where information doesn't have to be updated as often as it is. Perhaps I am wrong, but I'll just hope that I'm right :)

Summary: I don't know the ultimate solution to speeding up the site, but it neither involves rewriting the entire engine, nor disabling popular, usefull features.

      C:\>shutdown -s
      >> Could not shut down computer:
      >> Microsoft is logged in remotely.
    

I'm sorry to say that I turned off all the nodelets and it didn't make a noticeable difference. It could have just been slow in general at the time, but I think the slowness issue is not solved by turning off nodelets.

chromatic is dead-on. Why start all over?

Why not work on the social-end of Perl Monks, and start a movement to use the site in a manner that puts the least amount of strain on the server?

Turn off monk quips and images, don't use Newest Nodes for bouncing on the chatterbox, etc. Get your friends to do it. Make it a source of pride. "I didn't strain the server today!"

John J Reiser
newrisedesigns.com

I agree. But I will take this one step further and argue that its slowness is embarrassing. Recently I was in a discussion with a co-worker about the virtues of mod_perl. My coworker liked PHP and talked about how fast it was (the execution speed & the code development time). I was winning the argument, until he pointed out how damn slow PerlMonks was!

I am not sure if a complete redesign/reimplementation is the best solution here - but is anything else being done to solve the problem?

Of course doing a new everything would also be just plain fun.

PerlMonks is slow but it's backed up by five years of development. Why throw all this away? Because it's slow. We've learnt a lot in five years and you can't tell me that there are no legacy remnants in the Everything code that are 'too embeded' to remove. Of course there are! Caching for one seems to be a problem.

Set the code free I keep hearing that there's problems that people will be looking at when they get the time, however why not make the source open so that we can all take a look? Surely the reason why most of contribute to open source is that if everyone can post a small change then we end up with a lot of goodness added. And before you all reply telling me that I can download the Everything source, let me make the point that the Perl Monk's source is very different from the Everything. Sure you can get a grasp on how it fundamentally works, but you can't write a MonkPatch.

Parallel Rewrite What I'd suggest is a parallel open-source rewrite. We colaboratively rewrite the Perl Monks site using everything we've learned in the last five years, starting with data design, then moving to data migration then onto the actual site. Don't think of this as 're-inventing the wheel' but as 'inventing the pneumatic tyre'.

I'll see your $0.02 and raise you the same for each of my posts. Go ahead and start your rewrite. I'll give you that $0.10 if you're done rewriting Everything before I'm done refactoring Everything, if you support the same features Everything 0.80 does, and if your migration path is easier than mine.

I'll double it if you're as well or better tested.

I'll give you that $0.10 ...

I always knew open source programmers were poorly paid, but this is just getting silly!

Oh, and I vote rewrite the entire thing. Starting more projects on this site would be a good thing.

chromatic has already addressed that idea.

Makeshifts last the longest.

A very good idea, but ... it will generate (hopefully) a lot of traffic and the site would get even slower (although I must say that I never found the "slowness" of Perlmonks any problem).

CountZero

"If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law