Distributing code to 100's of servers

perlofwisdom has asked for the wisdom of the Perl Monks concerning the following question:

I've got several hundred servers (Windows, Linux, and several flavors of Unix) that have all of our maintenance scripts (hundreds of scripts, in dozens of sub-directories), on multiple networks. As you can imagine, maintenance of this environment is difficult at best.

Since many of the scripts use common modules, any change to the source code is tightly controlled. And, because of heavy inter-dependence between the modules, changes are normally supposed to be distributed as a release bundle. Even with fairly good controls over the code, keeping everything in sync with the master copy is nearly impossible.

The real problem is that our current distribution method is undocumented and laden with obfuscated code (I tip my hat to those of you gifted in this fine art, alas I am not). To make matters worse, the guy who wrote it has left the company and now lives out of state. So, it looks like I'll have to re-write the entire process. I could use a few tips on how to do the following things remotely, and from just a few distribution points (probably one per network):

1) tar/zip the existing code on the target server.

2) In the event of failure (incomplete distribution to a server, or widespread new bugs in the new release), I need to be able to reverse the process using the saved tar/zip file from requirement #1.

3) Deploy the new distribution (which will also be in a tar ball/zip file).

4) Verify successful deployment.

I know I'm not the only one out there who has the responsibility for keeping things running in a crazy environment like this, so I come seeking wisdom from those of you who may have faced this before. Any help?

UPDATE:

This is why I love PerlMonks! You've given me some great ideas, and a lot more to investigate. I should've mentioned that we are using CVS as the repository. But, since several of you mentioned that, it opened up new avenues of thought.

I welcome any more suggestions on how to implement your solutions with CVS.

Thanks again.

Comment on Distributing code to 100's of servers

Replies are listed 'Best First'.
Re: Distributing code to 100's of servers by BrowserUk (Patriarch) on Oct 31, 2007 at 01:55 UTC
Seems to me that a distributed source control system like subversion or SVK would be ideal for this. Store the modules in a repository on your test machine and mirror that on your production machines. When changes pass your testing procedures they get committed to the master repository. The production machines then sync and update to get the latest code. With SVK, the changes can be 'push'ed to the mirrors, though I think the update can only be performed locally? If anything goes wrong, you can revert back to the previous version. Either on individual machines, or at the master repository and force another update at the targets. If that idea floats your boat then you have a decision to make about what to store in your master repository: Platform independent distributions (al la CPAN): With this, you only need have one version of each new distribution at the master repository, but you end up having 4 copies at every production machine. The one in the mirror repository, the checked-out copy (al la CPAN packages directory), the 'blib' copy, and the finally installed copy. You also have to have an ancillary process of going through the make/test/install cycle at each machine. Platform-dependant 'installed images': You have to have 1 platform-dependant installed image for each platform at the master repository, but only 2 copies on each target machine. The mirror copy and the checked-out copy that is also the installed image. There is no need for the make/test/install and the update would be directly to the installed image (lib) directories. Whether this is a viable option will depend upon how consistent your production machine installations are. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply]
Re^2: Distributing code to 100's of servers by roboticus (Chancellor) on Oct 31, 2007 at 13:35 UTC
I second BrowserUk's suggestion. I've built a system built on CVS to manage code distribution on a number of systems, and it works well. I didn't build modules like absolut.todd suggests, we just have a directory hierarchy that we check out as necessary. Basically, it works like this: Each morning, the computers compare CUR_PROD_VERSION.sh with their local stored copy. If it's different, they check out CUR_PROD_VERSION.sh, which is a trivial shell script that checks out the latest production version (using a tag). ...roboticus	[reply]
Re: Distributing code to 100's of servers by andreas1234567 (Vicar) on Oct 31, 2007 at 06:03 UTC
Have you considered Cfengine? Cfengine is an automated suite of programs for configuring and maintaining Unix-like computers. It has been used on computing arrays of between 1 and 20,000 computers since 1993 by a wide range of organizations. Cfengine was implemented primarily on Unix-like operating systems, but has since been ported to Windows operating systems and MacOS X. I have used it myself in test and develepment environments for maintenance, primarily making sure I have all the required software installed in advance, the directory structures is in place, accounts are set up etc. This incredibly powerful tool is released under GNU GPL. Update:Added platform section from the reference. -- Andreas	[reply]
Re: Distributing code to 100's of servers by perrin (Chancellor) on Oct 31, 2007 at 01:58 UTC
That sounds okay for a few servers, but when you have hundreds, I think you need a more scalable approach. I would be looking at a networked filesystem (Samba/NFS) with the code on it, or remote boot servers that pull from a central point, or at least a system like RPM where you can automate the distribution of packages with existing tools like yum.	[reply]
Re^2: Distributing code to 100's of servers by absolut.todd (Monk) on Oct 31, 2007 at 03:16 UTC
Thats pretty much what im working on right at this moment. Im creating a package that will pull code from CVS, package it up into a RPM and then it will be released via yum. All will be configure via a YAML file, so the general thought is that the developer will tag the CVS module, and the release manager will update the YAML file with the tag (he has little unix experience so i am following KISS). And a script would check out the source, turn it into a RPM and then via yum update the target system. The nice thing about it is that via RPM it is possible to figure out if there have been any changes on the production system. Unfortunately I have found the RPM modules on CPAN to be a little lacking so im doing a rewrite of one of them to have a little more cleaner interface and fix the bugs. Add to that i only started a couple of days ago so i don't really have much to show for it yet apart from these comments.	[reply]
Re: Distributing code to 100's of servers by cmv (Chaplain) on Oct 31, 2007 at 02:02 UTC
perlofwisdom- My better half did some work in this area. Her stuff works on *nix and on PCs under Cygwin. Even if this isn't exactly what you're looking for, the methods they used to solve some of the problems that you're facing may be useful (I believe you can freely download this stuff, but you have to search a bit around the fnal website). Feel free to PM me if you need more info. -Craig	[reply]
Re: Distributing code to 100's of servers by dwm042 (Priest) on Oct 31, 2007 at 14:12 UTC
There are three ways of handling this, that I can think of. 1) Use a single common repository on a library machine, set up common directories on target machines and ssh trust relationships using the appropriate public keys. Keep your scripts in those common directories, with updates pushed out regularly from your library machine. 2) Use a single common repository on a library machine, and export a single directory of usable code as a NFS mount. 3) You could do what Mindspring used to do, which is use the apt package manager to package all their code, and then set up an internal repository. Then, synchronization would become: `apt-get update apt-get upgrade` [download] Update: if your code distributions would include large flat file database dumps, then I cannot recommend CVS as a distribution mechanism. I saw CVS being used once as a backup mechanism for flat file database dumps. By the time the dump reached a few hundred megabytes, it exceeded the memory of the target CVS machine and the backup was failing. Even scp would have been more robust with respect to size.	[reply] [d/l]
Re: Distributing code to 100's of servers by tilly (Archbishop) on Nov 03, 2007 at 01:08 UTC
brian d. foy gave a talk about minicpan not long ago. One of the uses that he walked through was setting up your own miniature CPAN repository with your own modules. Then configuring other machines to pull from that repository. Assuming that your target machines have the standard make infrastructure installed (Unix and Linux should, Windows can be made to, particularly if you've installed Strawberry Perl), you can then use the standard CPAN toolchain to update your target machines from your central repository. One of the nice things about this solution is that each machine can keep track of its own dependencies. So if you need a particular script, the installation process will figure out and handle all of the modules you also need to install. I don't remember the full details to make it work, but he has an article about it in the current issue of The Perl Review. (Yes, this requires a subscription. But you'll be getting access to a useful Perl resource.)	[reply]