:) Perlmonks.org runs FreeBSD

Wow. That's impressive. Thanks for sharing that! Especially the specifics of your FreeBSD setup. It helps in deciding what to try first.

You mentioned heavily loaded servers...are you talking about the number and type of services running on them, or how heavy the traffic is? Or maybe their load averages?

Your server setup sounds sweet! How much of an onslaught are your FreeBSD servers able to stand up to (might need to mention hardware specs of the server to make it meaningful, I guess)? By onslaught, I'm thinking of something like getting slammed with a "Slashdotting" or digg-type peak of server request all at once.

Thanks again! FreeBSD sure sounds n--i--c--e to my geeky heart...

Actually, none of it's that impressive. That's the point. Properly tuned FreeBSD servers with spawning daemon services (such as Apache) are very capable. We separate out the MySQL and mail systems to dedicated machines (we do a lot of HTML e-mailing), and the MySQL is replicated with failover across the Internet. Likewise, the web subsystems are duplicated in several locations. We have homebrewed dynamic DNS management, which serves us well except for IE users whose crappy browsers don't obey no-cache pragmas.

What do they do well? Everything. We've had DDoS attacks, we've had load spikes, and we've had packet drops from upstream. FreeBSD has served well, with the only problems coming from admin errors, like leaving an anonymous FTP machine NFS'd to a live webserver or typing 'shutdown -h now' in the wrong terminal window.

Our machines are all Dell 1{678}50's and 850's with single or dual P3's and 1 - 2 GB of main memory. The 1x50's are all SCSI-3 with partitions and swap spread out over three disks, and the 850's are simple P3 ATA-disk machines.

We use a couple of the cheap boxes for live HDD data backups, but other than that and the system duplication, we just replace the systems with the latest low-end Dell hardware on a fairly regular basis. What we have been particular about is getting a good solid bandwidth provider. We're in One Wilshire in LA for our primary node, right next to Google and the other big guys. (www.aerioconnect.com) Clean 'Net, clean power, and good monitoring and emergency service.

Don Wilde
"There's more than one level to any answer."

Well, I suppose it depends on what exactly you need. If security is the primary concern, I'd opt for OpenBSD, since its out-of-the-box config is, IMO, the most secure of any free OS.

Of course, FreeBSD is excellent as well, and in the hands of a competent admin is every bit as secure (IMO) as OpenBSD. It also seems as though the FreeBSD crew manages to have a more up-to-date (though probably less-thoroughly-tested) repository of software and libraries.

All in all, I would tend to recommend FreeBSD except in certain niche environments.

Of course, it does help to have a FreeBSD guru * with us to make sure everything's properly installed.

One thing's for sure: once include install a machine, we never have to call him back to fix anything, and that's just great.

* - include refuses the title, but I insist on giving it to him since he knows so much more than I do about BSD systems

Benchmarks speak louder than anecdotes. The article is a couple of years old, but clearly shows the differences in scalability for several OSes, including the two *BSDs you are looking at. The code is provided, so you can re-run the tests on current versions if you wish.

For the lazy, OpenBSD did not scale well to high volumes of web traffic.

Well, for sure I don't want to start a flame war on "This is better than that". That benchmark is a fact for 2003 when FreeBSD was in it's 5.X "complicated version". Now we are in 2006 :) and FreeBSD is almost releasing it's 6.2-RELEASE. Please stay updated and tunned :)

What heppened to us on more than one occasion was that there would be a "normal, routine" update or patch or release upgrade of the OS, and suddenly a lot of perl scripts would stop working, because the version of /usr/bin/perl had suddenly changed (unannounced to the perl programmers), this somehow involved a change of "standard" paths in @INC (because /usr/lib/perl5 is organized by version, even inside site_perl), and a lot of non-core modules were suddenly not being found.

When these "transitions" happened, there would be momentary panic, then all manner of diverse, inconsistent corrective measures (TMTOWTDI showing its dark, monstrous side), and an enduring sense that the perl environment as a whole was dreadfully fragile.

(Problems were sometimes amplified when various hosts on the local network -- servers and workstations sharing the same nfs volumes where someone might run the same script in a shell on any machine -- would somehow end up with different perl versions and different module inventories. But that's more a matter of sysadmin behavior, not anything intrinsic to the choice of OS.)

Maybe it was just pilot error on the part of the guys who were playing sysadmins when the troubles happened, or maybe our group as a whole was just missing a something fundamental about managing Perl in a non-simple FreeBSD network, or maybe it really is something that's hard to get right. I wish I knew.

It's also entirely possible (maybe even likely) that none of this would be relevant or at all likely in your situation.

I agree with you, graff, that this does happen.

I am always p-o'd by things that are in site_perl/5.8.6/blah instead of site_perl/blah, but that's an application of the philosophy of upgrading that does its best NOT to overwrite configurations. That has been The Berkeley Way from Day 1, but is it only FreeBSD ports that do this? The same philosophy applies to unpacking directories when porting, so that apache-1.3.36 is not overwritten by 1.3.37, for example. One could argue that modules are runtimes and therefore should be in common directories, but they're also source, and thus -- by the philosophy -- should not.

Yes, you do have to sweep the module directories up to the generic before going into production or do a manual port upgrade of each module, and, no, that's not mentioned except in The manpage of Hard Knocks. I'd rate the power of the ports collection and portupgrade system as far and away better than any other installation system on any *NIX, but it doesn't pretend to protect admins against the need to understand what's happening with each and every package used in production.

Of course, that's why the new boxes I mentioned get upgrades and testing before they go live. We never upgrade production boxes on the fly; some of ours are still running 4.10. I can't say we have never been bitten, and certainly I'm waaaay far short of the all-knowing guru I've made it sound like I am/we have, but that's why you build and test off-line.

FreeBSD is not meant to be the idiot's way to riches, but it sure _is_ the poor geek's toolbox for Internet success.

Don Wilde
"There's more than one level to any answer."

This is why I install my own perl in /usr/local/perl-x.x.x. I have code that I use here at work to use CPAN and install all the modules we need. Then I have a link /usr/local/perl which points to the production version. I like this setup since I can install a new version of perl and modules, have users test using the self contained install. When testing is done, I point /usr/local/perl to the new version directory. I let the ports that need perl install it and let portupgrade handle the updates.

Heheh... Yeah, playing with "#!/usr/bin/perl" vs. "#!/usr/local/bin/perl" is something we've done in the past as well -- but that was a case of maintaining both a really ancient "system default" installation for solaris (that was /usr/local/bin/perl -> perl4) along side a not-too-ancient perl5 (/usr/bin/perl -> perl5.3).

It makes life tolerable, but there are still traps for the unwary -- e.g. when scripts are copied in from other places that don't happen to share this subtle distinction in perl paths. Just another of those gritty details that users of perl scripts need to stumble over now and then...

We could also opt for "#!/usr/bin/env perl" as the shebag line, and control the relative ordering of /usr/local/bin and /usr/bin in PATH. But that's not exactly a comfortable solution either; it just passes the discomfort to the user's shell environment instead of the script maintainer's choice for the shebang line, which is probably a really bad idea.

These kinds of post in this thread, sharing real world experiences and implementation details of the "stack" components (OS, database, etc.) people find work well, make us all better Perl Monks, IMO.

Right now, I'm reading up on DDOS defenses. Interesting stuff (feel free to offer suggestions and links).