http://qs321.pair.com?node_id=227180

This morning the database server was spending a great deal of its time waiting for disk data to be read. This meant we had lots of DB requests sitting around for long periods of time while the DB server's CPU was rather idle. This meant the site was down (too slow to be at all useful most of the time and likely to give you errors as long-hung DB queries got killed or other timeouts occurred) for a couple of hours.

I opened a trouble ticket with Pair (who graciously and efficiently hosts us though I don't think we pay them -- I'm not in on that end of things, but I'm thankful). I have not yet gotten word on what they found. But the site appears to be back to more normal responsiveness.

The site has been slower more often than "usual" the last couple of weeks. I think the main reason for this change is simply "popularity".

In the process of looking into this morning's problems, I noticed that Pair has an operating system upgrade planned for next week (FreeBSD STABLE-4.1.x to STABLE-4.6, IIRC) and they told me this has improved disk I/O and so might help site speed in general.

In the process of looking into this morning's problems, I also found evidence that the Zombie mysql on FreeBSD problem (I didn't create that user, but follow its "evil" link) may be causing us slowness even when we aren't doing long-running queries (which was why searching had to be revamped and backups were so tricky to get right). So I pointed Pair to the information and they'll be trying to upgrade MySQL to improve that situation after the operating system is upgraded.

For further performance improvements, start a fund to pay me to take a couple of weeks off my real job and I'll be sure to finally revamp the node cache and a couple of other inefficiencies I've been eyeing for a while. (: (update: that's a joke too -- I'll reply with more on this)

                - tye

Replies are listed 'Best First'.
Re: Recent slowness and outage (mea culpa)
by tye (Sage) on Jan 16, 2003 at 07:05 UTC

    Note the two "that's a joke" updates, one in the root node, one in a reply. That "(:" probably should have been a ";)".

    First, I don't want anyone to get the impression that I take the 3 fine computers Pair has provided us for granted or that I expect them to just cough up another. If the site continues to get more popular, then another machine might be a very good idea at some point. But that is probably quite a ways off.

    Second, I was just joking about a "pay tye to take a couple of weeks off and work on PerlMonks" fund. I'm flattered some have already offered to pay for my services. When I saw the first offer, I didn't even think there'd be a way that would work out.

    But if there really is significant interest in something like that, then I can probably use money to convince my wife into giving me some hours to concentrate on PerlMonks without feeling guilty so that I could complete a "bigger chunk" project like revamping the node cache.

    But someone would have to put together the logistics of collecting the money, negotiating my "rate" ;), etc. Perhaps YAS would be interested in that, though I don't know that anyone at YAS has ever heard of me.

    And it isn't that I think I'm the only one who can do the things I'm talking about. To some extent I think I'm in a bit of unique situation because I was the one who revamped the "top" for MySQL (called "mytop"; I named my version "sqltop") so that I could help figure out what was causing all of the sporadic slow-downs way, way back. In part, I've got these ideas in my head and I've explained several of them to some extent or another to some people, but so far they are still best documented in my head. But I hope to see other people making these (and other, probably better) ideas happen.

    In any case, the node cache revamping is the one that I don't yet see anyone else taking on and the one that I think is the most important for PerlMonks. That is the one that has taken a long time to solidify in my head (which it finally has) but is just too big of a chunk for me to get done without some dedicated blocks of time to work on it. So that is what I'd work on if this "fund" works out. I'll still try to work on it no matter what, of course.

    So I'm flattered that there is interest in "the fund". If it happens, great. If it doesn't, that's just fine too. But right now my wife wonders where I am and I feel guilty, so I'm off. (:

                    - tye
      Maybe that should be the next poll? I know I'm definitely game on putting 20€ or so in the fund. Heck, all the work you and the rest of the crew have done so far on the site is already more than worth paying for. I just hope it'll be easy for credit card free Europeans to make a donation. :)

      Makeshifts last the longest.

Re: Recent slowness and outage
by Coruscate (Sexton) on Jan 16, 2003 at 00:10 UTC

    Would anyone care to believe that the majority of the problems rise from 'popularity' as tye suggested as part of his post? I don't just mean popularity of the site by itself. Looking through some nodes, I've noticed many other methods of accessing the site that obviously add on to the load.

    Methods of accessing the site that I've found (undoubtedly there are several others):

    • The actual website; requesting pages (duh)
    • Chatterbox Clients: There are quite a few variations of CB clients. I see a full-page chat, framechat, a java client, and command-line tools.
    • The XML Tickers: On top of the chatterbox, some clients provide access to newest nodes, other users, the XP nodelet, etc etc.
    • cblast35: A CB log, how many of these are now setup and how often do the damians request the data?

    And there are most likely more to add to that list. Server load must be rising incredibly.

      Particularly the chatterbox clients are a much lighter load than chatting by constantly refreshing which requires a multitude of things to be done to build a page. There are five cblast35 mirrors if I'm not mistaken, and they poll the ticker only once every couple of minutes, so they definitely cause the least extra load of anything, far less of it even than anyone casually browsing the site. The newest nodes ticker similarly is very likely to be causing less load than visiting Newest Nodes - not to mention most clients for it make use of threading information in some way or other, allowing you to avoid a separate visit to notes posted in response to a new subthread you can read all at once.

      Makeshifts last the longest.

      Great observation.

      r.e. requesting pages, I've experienced problems specifically with timeouts when I've tried to log in over the last few days. A couple of times I just gave up. It seems to be happening when the number of monks romping around the monastery is up around the 40 person mark. I don't know if there is a direct correlation. It's just something I noticed (for whatever it's worth).

      Acolyte
      Studying at the feet of the masters
Re: Fund for tye..
by castaway (Parson) on Jan 15, 2003 at 19:04 UTC
    /me puts 10 Euros into the Fund..
    C.
Re: Recent slowness and outage
by perrin (Chancellor) on Jan 16, 2003 at 17:22 UTC
    Speaking of the node cache, my idea was to replace the current "one cache per process" system with a shared cache, probably using Cache::Mmap. It should improve the hit rate on cache requests quite a bit. Do you think this would help?

      My first guess is that no, it wouldn't help. It would greatly reduce memory usage on the system. If that then lead to making the cache size a lot bigger, then that would help. It would have other benefits as well.

      But it would also thwart my plan for greatly reducing the impact of the race condition. Right now, you get a node via the cache, then you make updates to it, then you request that the updates be saved to the database (via the cache). Between "get node" and "save node", there is a pretty high likelyhood that someone else also started making changes to the node. And we are stuck with optimistic locking and ignoring failures (because of the nature of the site) so the best we can do is have each process only update fields that it changed. So we have to track which fields this process changed. And there are two ways to do that:  1) note each change, ie. tie, ie. slow things down too much;  2) save a copy of the original and compare (my plan).

      So how do we save a copy of the original? In a perfect world, there would be a "I plan to change this node" version of the "get node" function and you wouldn't be allowed to save changes to nodes that you didn't get via that function. Getting from the PM world to that world would require huge numbers of changes all over the place. So I'm not shooting for that.

      So I need to have the "get node" function save a copy of the node separate from the one that gets handed to everyone and that might get changed. Well, the node cache is the perfect place for this. However, if the node cache is shared between processes, then the copy saved in the node cache is no longer going be "what this process originally got" so it isn't useful for making that comparison to reduce the race condition.

      So I don't want an inter-process node cache. Not until we have some other way to track changes to fields in nodes. That is a long way off.

      What I'd love is an inter-process nodelet and miscellaneous data (such as chatter in a few forms, approval status, etc.) cache. Nodelets are currently cached and only updated if they are more than N seconds old (where N is configured per nodelet). But each process has a seperate cache so most nodelets are updated P times every N seconds (where P is the number of httpd/mod_perl processes in use). So a shared cache reduces the load for many of the nodelets by about a factor of P (dividing by even 16 is a very nice thing to do to system load).

      Negotiating updates in such a cache gets rather complicated and varies by what you want to cache, and this all gets more complicated because you want to see updates from other computers also.

                      - tye
        Do multiple updates to the same node really happen that often? I thought people were not usually able to edit the same node.

        Maybe I'm misunderstanding, but I don't think the node cache is the right place to implement your idea about saving the original version. Don't you want to save the version that this user originally received, rather than this process? There's no guarantee that a user will even be on the same machine when their form submission is processed. Seems like you would have to do it based on session and keep it in the database.

        Having a separate cache for nodelets (or anything else) is no problem. You can have as many caches as you want.

        Finding a solution that works across all the machines in the cluster is harder but possible. It would require something fancier like messaging daemons or some database trickery. Just making the cache multi-process is enough of a challenge for now.

Re: Recent slowness and outage
by peppiv (Curate) on Jan 16, 2003 at 16:10 UTC
    Maybe the 'popularity' thing is just one of those January hiccups. You know, when I go to the gym now, there's all these New Year Resolutionists and I can see it in their face. "This is the year I'm gettin' in shape!" Well, unfortunately it doesn't always stick.

    Maybe there's lots a junior web dev's out there saying "This is the year I'm gonna learn Perl!". Hopefully it will stick.

    peppiv

Re: Recent slowness and outage
by PodMaster (Abbot) on Jan 15, 2003 at 23:32 UTC
    I'd be willing to contribute 20 USD at least ;)

    What are "other inefficiencies you've been eyeing for a while"?


    MJD says you can't just make shit up and expect the computer to know what you mean, retardo!
    ** The Third rule of perl club is a statement of fact: pod is sexy.

      jaybonci sent me patches to speed up group membership testing (which I don't think we do that much of compared to everything2.com where it was a huge difference). But it looks like ar0n might be applying those soon.

      The simple search is not a potential huge problem like it used to be, but it looks like it is still too much load. I think fixing it requires that the first stab at search be much simpler and then you get the option to "try harder" if it didn't find what you wanted.

      Good caching of "other users" and all of the places that make use of that could be a big win. Not an easy problem.

      The "navigation nodelet" keeps showing up on my radar. The solution there may just be to turn it off (or at least the parts that MySQL does a poor job of optimizing, despite them appearing extremely simple).

      I've done a lot of work toward better caching of chatter. I'm less convinced that is a big source of slowness at the moment, but it would also lead to better features.

      There is tons of room for performance improvements on the web servers (like caching the compiled code from templates), but that isn't where the bottleneck is (and adding a third webserver is a good solution since I'm not paying for it). (update: that's a joke)

      I think the newest nodes ticker never got optimized.

      There are probably others I don't recall at the moment.

      Update: Moving each user's scratchpad to a separate node. This is a performance improvement and makes features easier (like a "d/l code" link for scratchpads).

      Hide the fact that user nodes contain a "settings" and "document" records. Probably a relatively big performance improvement.

                      - tye

        Thanks for the list. Can you give us some idea of /how/ bad these things are? Would turning off the navigation and other users nodelets help things appreciably, or at least improve /my/ page load times? Would it be possible to run the DB server to log all requests for a few hours, to get some profiling info to know where to look? How work do you think it would be to go to a split-DB system, like livejournal uses?


        Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).

Re: Recent slowness and outage
by dmitri (Priest) on Jan 15, 2003 at 23:05 UTC
    I was under impression that perlmonks uses PostgreSQL.

      PerlMonks is built on the Everything engine (see the links at the bottom of any page) which uses mysql.

      --- print map { my ($m)=1<<hex($_)&11?' ':''; $m.=substr('AHJPacehklnorstu',hex($_),1) } split //,'2fde0abe76c36c914586c';