Do I need to upgrade at all?

Upgrading is a change. It's a massive change, actually. Is the cost of testing this change worth the benefits? What are the benefits? Do you have a backout plan if something breaks?

Personally, I would upgrade on a case-by-case basis. If version 2.44 of the Floopy module works for you, why does it matter if the latest version is 4.55? If you never hit a bug, then you don't need the bugfixes. Plus, bugfixes might introduce bugs you're not aware of. If you need a feature in 4.55, then upgrade to it.

Furthermore, the issue isn't bugs within a given module - it's bugs between modules. Integration testing is expensive! And, testing doesn't prove the abscence of bugs - it simply shows you haven't found any yet.

Thinks for a second . . .

The biggest problem I have with your plan is that you're changing too many things at one time. Change one item, then see what happens. If nothing breaks, bake it in for a month. Then, change another thing and see what happens. Lather, rinse, repeat.

That's a good point, and one I'm laboring over.

Things are working OK now, but there are a few 'little' things wrong with some of our modules that are fixed in newer releases. Some of these modules, like Class::DBI, have many dependencies, so an upgrade involves updating other modules.

The problem with standing pat is it's hard to get help when things are broken. When we have an error, the first thing the Monks or people on a list ask is "what are you running?" Too often in the recent past, we have this strange mix of modules and versions. So the answer is "upgrade and see if that fixes it."

And this is a completely legitimate answer since it is just too hard to track interdependencies across strange combinations of module versions.

So I guess I'm hoping we can better troubleshoot things by staying more current. But maybe this isn't the case?

Another solution might be to just do spot upgrades when we have to because we run across something. The problem here is exactly what you mentioned: testing is hard and costly. If we just did it 'as needed', it would also be un-planned. We have a lot of perl code running, so it's difficult to arrange for ad hoc testing of everything. If we had an upgrade schedule, at least we could plan the testing.

I agree wholeheartly with many things you said here. Change isn't always a good thing. I strongly recommend using a test server to see what breaks when you upgrade anything before you just deploy it all over, *ESPECIALLY* if you are highly dependant on it.

In my personal experience as a system administrator, unless mandated by the need for a feature, repair of a bug or a security update, an upgrade is an unneccesary act and often times a very dangerous one. When I was new to system administration, I thought that it was important to stay "within two releases" etc as is mentioned in the initial post. I quickly realized (after a horrible problem surfaced) that my logic may have been a bit flawed. I adopted a new philosophy...upgrade when there is some need to do so, otherwise, leave it alone.

dragonchild is right about changing too many things at once as well. If something does break (and trust me...it WILL), it will be far more difficult to find and correct the problem if you changed multiple things. If you *must* upgrade many things, do it in reasonable phases where you can simplify any troubleshooting. You will thank yourself later.

Just my $0.02.


perl -e 'print reverse qw/o b n a e s/;'

I've had systems running versions of perl that were years out of date. This has only been an issue in two types of situations --

one of the developers (okay, me), needed some functionality that was only in a new version.

(I think this happened once in 9 years.)

Solution: Install the new version of perl in an alternate location. (see the discussion in Solaris, Perl 5.8 and use 5.005) Developers who want to use this new version would need to explicitly specify this new version.

Problem: may not work for mod_perl; this is a pain if you're just doing it for a module, so you don't muck with the original one in place

Problems moving scripts between systems.

Okay, this one has stung me a few of times. Sometimes it's just a different perl location in the shebang line, but there have been a couple of times when my code didn't work with more recent versions of the module. (I remember a bad time once with Net::LDAP, but I just went and backed down the version so the systems matched, and I didn't have to change my code, as it was being moved because of a system failure, and I didn't have time to spend on it). Personally, I just look at which modules are giving problems, and fix those issues, rather than going for a complete sync across the board.

So, basically, to answer your specific questions:

1. I don't, if I can avoid it. The installed default, and possibly whatever was current at the system was configured.
2. When I need some new functionality that the new module provides ('functionality' to include bug fixes). No.
3. Install modules as needed, on a per machine basis.
4. nope, as I do it by hand.
5. System documentation. No.
6. I try not to upgrade, but if I have to, I run whatever tests the development teams have given me. Which is to say, I don't run any tests. But I then leave it running on the development servers until management signs off on it being tested, and then move it to production. (If something breaks, I've got record of someone signing off in the change management system, showing they didn't test their stuff).

Yeah, I've probably read too much BOFH over the years, but well, I've been a developer, and a sysadmin ... if it's important, you write tests for it. If management drives a schedule that results in nothing being thoroughly tested, I make sure they sign off the system before upgrades.

Note: I've since changed jobs, to where I only have to deal with one production system (that's in my control... we have four replicas, distributed geographically, on hetereogeneous versions and hardware, but I don't manage them), and I get control of my part of the production schedule, within reason, so I don't have nearly so many issues anymore. It's much better than trying to deal with ~30 web, ldap and mail servers and idiot management.

Didn't we run into a problem with a module that was upgraded and changed behaviour on us? :)

Before you upgrade, you can use your development servers to make sure everything works with the new versions, so that's already a leg up on what most people have.

I definitely recommend staying away from anything that smells like early-adopter. I tend to think that old code does the same thing that it always did, and if old code is working and you have other things to think about, other things should win. Other people in Stonehenge are at the other end of the spectrum. :)

Yes, we did. I think it was the URI module. It changed how it translated some things in the uri_escape method.

This is one of my bigger fears. Module authors can change behavior, or even just change a default setting. If we have code that relied on this default, it might break. Actually, I think DBI changed a default in a somewhat recent update and I've got it on my list as something that might break.

The safe thing here is to check for the know changes before the update, but I don't relish going through the change notes for every module we have installed to see if they changed a key behavior or setting. But if we want to know what we're getting into, this may be necessary.

I have a script to report what code is using what module. For modules that only have one user, I guess I could get that developer to look into things. My problem there is that too often I was 'that developer' who recommended using a module. :)

We're in sync across our environments, so we don't have that problem right now. We've got dev, QA, and prod servers, and several of each. We also have other servers where we run perl and we've just gotten those in sync since we started deploying from version control. When we put in changes, we test at each level and roll forward if things are good.

but to get it start, here is another discussion not long ago. Smooth perl upgrades

Break your requirements into two:

1. System which happen to use Perl, such as UNIX servers which perform some other task.
2. Servers which host production Perl applications.

The first situation is very easy to deal with: Upgrade Perl when you upgrade your OS. Consider the out-of-the-box RedHat Enterprise server. It comes with Perl, and has its own mechanism for keeping Perl up to date. If you don't have custom Perl apps on that server, let RedHat's up2date handle it for you. When you upgrade from ES3 to ES4 you will get whatever version of Perl comes with the latter.

The second case can be handled as follows:

1. Recognize that this is a development issue and not an operations issue. IOW, it's controled by your developers' need.
2. Dev should set the Perl version on a per project basis.
3. Build your apps with all their libraries local. An *excellent* example of this is the Krang CMS project.

If you adopt the methodology used by Krang, you will find that you are immune to upgrade problems, and have a very solid platform for testing new versions of Perl on your apps before you put them into production.

TTYL,

-Jesse-