http://qs321.pair.com?node_id=240429

mp has asked for the wisdom of the Perl Monks concerning the following question:

The basic problem is this. I am maintaining several servers, one for development and several for production, and I am fairly heavily dependent on CPAN modules for this project. I tend NOT to upgrade perl or CPAN modules because of the extensive testing required to ensure that everything still works, though at times I do upgrade particular modules to the most recent version on CPAN. When installing a new module that has not been used on the project previously, I generally want to install the current version that is on CPAN.

Because I may need to bring another server online later to keep up with load or to replace a server that has crashed, and because that server may be running a more recent linux installation, I need a good way to manage old CPAN sources so that I can always rebuild and reinstall all CPAN modules that are being used (for many modules, this will be older versions) from sources with identical version to what are currently being used on the existing servers.

The way I see it, I have two primary problems to solve. One is keeping track of which versions are currently (and should be) installed for a particular server. I think there may be a way to do keep a master list as a bundle file, but I'm not sure exactly how to do that, so any pointers would be appreciated. What I'd like to be able to do is have a bundle that tells what versions of all modules to use, what order to install them in, and what version numbers, then be able to just tell CPAN to install that bundle. Is that possible?

The second problem is creating and maintaining an archive of the CPAN module sources that I'm actually using. I really don't need all of CPAN, just certain modules, so a sparse mirror containing only those modules would be sufficient.

I have experimented with saving the "*.tar.gz" files off of CPAN for just the modules that I use, then building manually from these (perl Makefile.PL, make, make test, make install). While this is adequate for the perl source itself, it does not seem like the best approach because of the number of modules in use and the time it takes to build them "manually".

I have also thought of keeping a complete CPAN mirror that I only update when I need the latest copy of something (and then possibly only update that particular module while always keeping old versions), then just use CPAN.pm to install from that as needed. It seems that it should not be too difficult to write something (possibly borrowing from merlyn's mini-cpan-mirror maker) to fetch new modules into the archive on demand without disturbing modules that are already there.

Rather than maintain a full mirror, it seems like what would be ideal would be to maintain a sparse (in the sense that not every module is present) CPAN mirror and have a script that can fetch a module source on-demand to add to that mirror. I could then point CPAN.pm there and use it for installations on all my servers. Has anyone tried this? Any gotchas?

Has anyone else come up with a good solution for all or parts of this? Am I missing something basic that already solves this problem? I'm looking more for general direction that specific code at this point.

  • Comment on Maintaining CPAN sources for later installation across multiple servers

Replies are listed 'Best First'.
Re: Maintaining CPAN sources for later installation across multiple servers
by hsmyers (Canon) on Mar 04, 2003 at 21:04 UTC
    I don't know that there is a 'right' answer to your question/s, but I do know what I would do if I were in your shoes. Seems to me that you have two problems, one is replicating a working solution and the other is updating 'parts' as they change. To handle the first I would go ahead with your list of '.gz' files, except I would use it to power a perl script to automate the install procedure. It would do for each .gz the usual make this that and the other thing---all in the correct location probably from a tempory directory, then cleaning up afterwards. There may be some 'hand' work left afterwards, but if it is invariant, then script it back into your build process. Generalize this so that you can point at whatever box you need to and you should have a leg up on the first problem. The second would require monitoring the CPAN published list looking for hits---from which you could do the appropiate thing, dl, etc. I'd also do all of this with an eye towards doing as little as possible in order to get the job done, as it seems to me that in your case, 'least change' is probably 'best change'...

    --hsm

    "Never try to teach a pig to sing...it wastes your time and it annoys the pig."
[reply]
Re: Maintaining CPAN sources for later installation across multiple servers
by lshatzer (Friar) on Mar 04, 2003 at 21:08 UTC
    You might want to read this.
[reply]
Re: Maintaining CPAN sources for later installation across multiple servers
by demerphq (Chancellor) on Mar 04, 2003 at 23:25 UTC

    Perhaps you have too many modules for this to be efficient, but source code control can be used to deal with this problem. During the development phase you keep everything new or added under source control, and later for install all you do is run the standard perl install (for me this means installing a given version of AS perl) and then a "get latest version" into the approprate location and presto your done. Source control also gives you the easy ability to look for unauthorized changes, snap shots, etc. For instance config files and the like can be kept under source control to make tracking errors and changes easier.


    ---
    demerphq


[reply]
[d/l]
Re: Maintaining CPAN sources for later installation across multiple servers
by valdez (Monsignor) on Mar 04, 2003 at 21:12 UTC

    At my new workplace, we are trying this approach: all Debian Linux boxes and one custom mirror with all debs we need plus locally packaged modules that aren't available officially. When we need a replica, we install a base system and set /etc/apt/sources.list to our mirror: installation ends in minutes without problems (almost, we are still testing and it is not our first priority).

    Ciao, Valerio

[reply]
Re: Maintaining CPAN sources for later installation across multiple servers
by IlyaM (Parson) on Mar 05, 2003 at 11:45 UTC
[reply]
      Thanks for the link, that thread was helpful. Thank you all for your suggestions.

      The approach I'm leaning toward at this point is to just mirror CPAN locally, use either a bundle file or write a script that uses CPAN.pm to do the installation of all the modules of specified revisions from a list, and write several scripts, 1) to compare listed modules against the repository to see if new revisions have been released and 2) given a module that is out of date in my local repository, be able to pull the new version down from CPAN into the local repository.

      Something else mentioned in the mod_perl thread was conversion of modules to RPMs for deployment. That approach looks interesting and provides the ability to query the rpm database to find out what versions of what modules are installed and the ability to uninstall modules, but it also adds a complexity that I think is beyond what is needed at this point in this project.

[reply]

Back to Seekers of Perl Wisdom