Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

How do you manage module deployment?

by fce2 (Sexton)
on Jul 13, 2005 at 03:28 UTC ( #474448=perlquestion: print w/replies, xml ) Need Help??

fce2 has asked for the wisdom of the Perl Monks concerning the following question:

We use Perl on many machines here, and have been suffering from different versions of modules being installed on development and production boxes. We're looking to do something about it, and I wanted to get some insight from others who may have tackled this.

We use Subversion fairly extensively, and we roll code changes into production via an export, so we figured it was a natural extension to store module source in a repository too. Then, when installing/updating a machine, we go simply export an additional repository (third-party modules) in addition to the existing ones (internal modules, application code).

We've also patched some third-party modules to add or change functionality to suit our purposes. Where possible, we've contributed these changes back, but in some cases this hasn't been possible. Because of this, our desire is to keep unpacked distribution source in Subversion rather than a tarball.

The packages are actually installed into a seperate directory rather than into the various system Perl locations (with all our code using an appropriate "use lib" line). We do this because we're an applications group and don't get root access to our machines.

We figure that we'll have the following tasks to do:

  • Install a new module
  • Upgrade an existing module (which may mean a later version of a core module).
  • Reinstall an existing module (eg after we patch it)
  • Compile and install everything (bootstrapping a new machine)

When installing/upgrading, I'd really like dependencies to be figured out so stuff is installed in the proper order. This information would be cached so that when a new machine is installed the modules are built in the correct order.

I've looked into the stuff that CPAN::Shell provides, and it looks like it knows how to do everything I need (download, calculate dependencies, build, test, install), but it doesn't really seem to be suited to this task. As far as I can tell, it can't be made to use an existing unpacked source directory, preferring to extract its own from a cached tarball. It also can't seem to calculate dependencies without trying to do the entire build, which I don't want to do if I can help it.

autobundle isn't helpful in this case, because I expressly don't want a list of core modules. I'm also not sure if it would install the exact version numbers I want. And besides, there's no scope for installing our own patches with it.

I suppose Module::Depends will figure into the equation somewhere (though I find its rather extensive list of dependencies somewhat amusing; I would have expected them to be kept to a minimum).

Another though I had is to carve up dh-make-perl. We're not using Debian here, but it seems to be able to manage to do alot of what we want.

So what have others done in this scenario? I can't believe we're the first to come up against it. Any guidance you can offer would be greately appreciated.

Replies are listed 'Best First'.
Re: How do you manage module deployment?
by CountZero (Bishop) on Jul 13, 2005 at 09:06 UTC
    Could PAR be of any help?

    It knows how to put together a working Perl-environment in a single zip file which can be deployed on another computer. If you make a "mock" script or module which uses all the modules you want to deploy, you could perhaps subvert PAR to your use.

    You can tweak some settings in PAR so it does or does not include core files.

    A presentation on PAR can be found here (Warning: it may not reflect the latest version of PAR, but it gives a good idea what it can do).

    CountZero

    "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

      pretty easy to create a zipped tarball from your source.
      system ("tar -zc /sourcedir > /mytarball.tar.gz");
      or you can use the Archive:: cpan modules

      Sounds like a disk image of your perl installation might be handy as well. Or maybe you can create your own rpm's for apt or yum. Apache ant build scripts are also great for copy/moving files around Ant.

        But that is not necessarily cross-platform and may overwrite already installed modules. PAR was meant to avoid these pitfalls. Of course if you are so lucky as to have a single architecture for all your machines ...

        CountZero

        "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

Re: How do you manage module deployment?
by jwest (Friar) on Jul 13, 2005 at 12:38 UTC
    In order to make a CPAN shell solution plausible in my environment, I use CPAN::Mini to maintain a local mini-CPAN mirror. That mirror weighs in somewhere in the area of 450MB.

    Then, as I create new or update existing modules I use CPAN::Mini::Inject to add them into the mini-CPAN mirror. Although I currently do this manually, it seems trivial to script it and run out of cron or some other scheduler.

    I serve the entire mini-CPAN mirror up over anonymous FTP. I've configured the CPAN module on each build machine to use it as its only CPAN mirror. Then I can pull down the entire project, dependencies and all, to a clean machine with zero hassle. Updates are similarly a breeze, so long as I'm disciplined about maintaining the dependency list.

    Some caveats are that the mini-CPAN mirror will only contain the latest-and-greatest CPAN modules each time it updates (though this might be configurable, I'm not sure). That works well for my environment, but may not be so hot for yours. Also, I don't do any updating of modules that I can get from the CPAN. If I were to do so, I imagine that it may require pulling it into a different CPAN namespace than that of the original author.

    --jwest


    -><- -><- -><- -><- -><-
    All things are Perfect
        To every last Flaw
        And bound in accord
             With Eris's Law
     - HBT; The Book of Advice, 1:7
    
Re: How do you manage module deployment?
by nothingmuch (Priest) on Jul 13, 2005 at 11:51 UTC
    If you are sufferring from conflicts, just install another perl for each project.

    As long as the perls don't know about each other, you have a lot of safety.

    From then on make sure people who do stuff to the perl installation directory just make a note of it, and keep the log under revision control.

    Third party modules should go into your source control for each project, should be renamed (even trivially). Renaming is crucial, don't underestimate it. If they need complex installation, you should have your own installation tree for each project.

    To sort out dependencies, you could either 'make dist' for each of your modules, and then put them in your own fake CPAN mirror and have CPAN.pm install their dependencies, or you could install the originals as part of the process. Since your modules would have been renamed, they don't interfere.

    The recipe script for installing all the dependencies should be under the project's source control. It should know how to fetch perl, build it, get all the modules, install them in the right prefix, and so on and so forth. stvn needed something similar, and that's why Verby was written. It might be useful for making this kind of installation script, but if you are going to script CPAN installations you might aswell use CPANPLUS's backend modules - they are more flexible to program.

    -nuffin
    zz zZ Z Z #!perl
Re: How do you manage module deployment?
by jimX11 (Friar) on Jul 14, 2005 at 01:19 UTC

    Have you considered adding tests in your testing suite for modules you need/expect?

    In Test::more, I believe you can test if Some::Module version 1.02 is being available with something like this

    BEGIN { use_ok('Some::Module', 1.02) }

    When you run a "make test" on a box that has a different version of Some::Module, the test above will fail.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://474448]
Approved by NetWallah
Front-paged by kvale
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (3)
As of 2022-07-03 20:58 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?