|
zerohero has asked for the wisdom of the Perl Monks concerning the following question:
There doesn't seem to be a way of having a private repository of modules, which one may install from. You can come pretty close, but not hit the target, exactly. For example, one may "run a mini-CPAN" using the mcpani tool (to do the administrative work on the mini CPAN repository), and use, for example cpanp to install. This works, but there is the overhead of having to mirror all the CPAN stuff to your local drive. It would be preferable if you could keep only your private modules in a mini-CPAN, and then when you installed, pull your private modules from your repository, and then move to the public CPAN to get the public modules.
An alternative to this approach is to use a tool like pip. The downfall here though is pip is too literal, you need to specify a filename. It's far preferable for deployment to be able to specify the module and version, and have the installer figure out where to get the actual file.
It seems like there is so much install machinery on CPAN (CPAN.pm, CPANPLUS, mcpani, minicpan, etc.), that I feel like I must be missing something. Isn't there an easy way to make a list of modules to install, some public, some private (e.g. in a Bundle::xxx), and then install these automatically (with some packages being fetched from private repos and some public). I guess this should work provided one has mirrored properly as I mention above (still waiting on all the files to get sucked down).
Not only that but there is no way to revert an install. This makes me think RPMs derived from perl modules are the way to go. Am I missing some obvious tools? |
Re: Private Module Repository
by zerohero (Monk) on Feb 28, 2009 at 03:04 UTC
|
OK, you can actually do this. The problem is that if you really know what you're doing, it's trivial to get this working, and if you don't you have to learn an awful lot about how the everything works through trial and error. This could be eliminated if mcpani autodetected an empty dir and setup the appropriate files (it only does this when you specify --mirror, which will suck down the universe).
Here's what I did:
- Inside CPAN empty dir, create authors and modules dirs.
- touch authors/01mailrc.txt.gz
- touch modules/02packages.details.txt.gz
- touch modules/03modlist.data.gz
- use mcpani to --add your pkg
- mcpani --inject -v
This will create a valid "micro CPAN" with just a couple of packages in it. For a secure server, try using sshfs and fuse to mount a remote ssh directory, and make it look like a normal directory.
cpanp needs a little help next. First you add "custom source" and use file:///yourcpan (obvious part). Next, you need to help cpanp find your file by installing "package-with-dashes-and-version". The version is seems to be necessary, otherwise it will just stick with an old version.
Does anyone know how to make cpanp realize that there is a new version of a local package (so we don't have to specify the version at the prompt)?
| [reply] |
Re: Private Module Repository
by perrin (Chancellor) on Feb 28, 2009 at 03:12 UTC
|
There are a few different versions of this concept available. There's one in Krang that can install a set of modules from their source packages. There's Shipwright. There's PAR which ships binaries all packaged up together. I think CPANPLUS has something, and possibly Module::Install too. There's also just building all your modules in a local directory and sharing them via NFS or Subversion or tarballs or RPMs. | [reply] |
|
|
Thanks for all the references. I looked through them.
I think the issue is confusion over the role of a "build system" and a "package and deployment system". A very good system, in my opinion, is the rpm/yum/repos system that RedHat uses (others may bring up apt-get and Debian which is also good). These packaging systems have the ability to do automatic dependency checking (just like CPAN). They also have the ability to do rollback, as well as package things other than Perl (unlike CPAN).
We already have enough good _build_ systems. They are as old as the hills, and things like make are still around, proving their value. Perl has it's own build systems which solve many of the same problems. Fair enough. But we don't need more solutions in this area. So Shipwright doesn't seem useful to me on first glance. No need to have another build system that is less known, has no docs, etc. Problem already solved.
There does seem to be a realization somewhere by some that it would be "a good thing" if CPAN could automatically make RPMs. Of course, the devil is in the detail on these things, and while you can automate 99% of this, there is a 1% that requires everyone who has ever written a module to do a little work. Having automatic RPMs from CPAN would be, in a word, awesome, because you can get the benefit of the rpm/yum/repos system which is goes far beyond CPAN (in terms of deployment functionality). Further, this system will only get better.
Note there is a tool, cpan2dist that does this. Give it a try and you'll see why there's a long way to go before we have full CPAN to RPM automation.
So I'll employ a hybrid deployment strategy. I'll maintain a separate perl stack, independent of the vendor perl, which is updated via a MicroCPAN installation of just my modules, mounted by fuse (so it looks like a file:///url). Everything else will be RPMs. Making perl RPMs has proven to be too much trouble (because of the many dependencies on CPAN-RPM modules).
Note, none of the tools mentioned above even comes close to providing the needed functionality (but thanks for the list). I would go a bit further and build something on top of rpm/yum/repos for repeatable deployment to a cluster of machines. But note, you need all the underlying machinery to work well for this to happen (e.g. rollback)
| [reply] |
|
|
What is the functionality you're after that none of this provides?
I worked on the Krang system and, although I now think building from source every time was a bit slow, that system does allow you to install a specific set of modules with an application without disturbing anything else on the machine. It means you can install a dozen different versions of the app, with different versions of the same modules, on one machine. RPM could maybe do that if you use relocatable RPMs, but I'm not sure it would work with XS modules.
Shipwright does that as well, and also seems to handle dependencies between modules. The dependencies are the main thing that makes the CPAN shell so attractive: tell it to install Moose and come back in 30 minutes and it will have grabbed everything needed. When using things like RPM, you have to build that yourself. I've tried it with cpan2rpm and a wrapper script that tries to look up dependencies and download them. It's hard to get it 100%. Maybe CPANPLUS/cpan2dist get closer.
| [reply] |
|
|
|
|
Re: Private Module Repository
by jasonk (Parson) on Feb 28, 2009 at 03:09 UTC
|
| [reply] |
|
|
| [reply] |
|
|
From the CPAN::Inject docs:The initial use this module was created for was to inject tarballs into the CPAN sources directory for the reserved LOCAL user, so that they can be installed via the CPAN shell, with automated recursion to CPAN dependencies. Seems to be just as you requested.
CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James
| [reply] |
|
|
|
|
