I have my own DarkPAN. Several of my projects share code that's not yet ready for the public CPAN. Some of that code may never belong on the CPAN—it's far, far too specific for the problems I'm solving.
Experienced Perl hackers often joke that 90% of their applications comes directly from the CPAN. A few thousand lines of code on my part might make use of tens of thousands of lines of code written, debugged, tuned, improved, documented, tested, and maintained by countless other people. Even better, installing and maintaining and depending on that code requires only the use of a few simple tools, including App::perlbrew, Dist::Zilla, cpanminus, and cpan-outdated.
Yet I want to share code between my own applications without having to upload it to the public CPAN. I want to use the same tools to manage and deploy this code as I do to code from the public CPAN.
In short, I want my own private CPAN containing only the code I put there.
That's why I wrote something I'm calling CPAN::Dark for now. It's a thin wrapper around CPAN::Mini::Inject which creates an empty, private CPAN into which you can add any or all distribution you like.
If you do this, you can configure your CPAN tools to use your private DarkPAN as one of multiple mirrors, so that CPAN operations will find your private code. You can also (as I do) use the dzil ScpDeploy plugin to inject your code into your DarkPAN as easily as you would release it to the CPAN.
(It would be nice if cpanplus and cpan-outdated
and other CPAN tools had a single point of configuration to manage mirror
lists, but that lack is reasonably easy to automate away for now.)
With all of my projects safely ensconced in a DarkPAN, it's even superlatively easy to test my code against any release of Perl 5 that perlbrew can manage. All I have to do is install the new version of Perl 5, then let cpanminus do its thing until it finds an error or completes successfully.
I've never had code deployment this easy before.
Your job, if you choose to accept it, is to give me feedback on CPAN::Dark. Is the name right? Does this solve a real problem? Is the documentation clear? (Does it need a mechanism to remove distributions?) Should it include configuration instructions for cpanminus/CPANPLUS/CPAN.pm?
Ironically (I think) I just coded up a *very* similar tool, though it's a bit more broad in scope, and takes a somewhat different approach (though I should see what I can steal from your code ;-)
If your project has a Makefile.PL or Build.PL, it figures out all the CPAN dependencies, right down to core modules, then builds a "micro-cpan" with the same structure as a normal mini-cpan, but containing *only* the deps for your project and nothing else.
So far, it's working pretty well for me and has allowed me to, for example, finally upgrade to Moose 2.x because the system (for reasons I won't get into) is stuck with v0.98!
I like the sound of that. One of the other tricks you can do with a micro-CPAN is to pin certain known-good versions of public CPAN distributions.
Except for the index files, cpanminus has this micro-CPAN built in with the --save-dists option. I've been toying with it recently myself for just such a purpose.
I, too, intend to use a darkpan for locking down a few module versions in addition to storing private modules.
As for removing previously injected modules,
that is probably something that should be a part of CPAN::Mini::Inject (in fact it's in that dist's TODO file).
As for module feedback in general, it seems like the purpose of the module so far is to provide an alternative CPAN::Mini. The equivalent of update_mirror() basically just creates the fairly empty template files that a CPAN has, and then uses CPAN::Mini::Inject to do the rest.
I think just doing that (initializing an empty repo) is useful,
since having a darkpan separate from a minicpan has its advantages.
I haven't come up with any other ideas of what it should do, yet.
Documentation for setting up cpan clients would be good (cpanm is easy, cpan might take more than just "o conf urllist" (I'm not quite sure)).
Thanks for pioneering this!
@perigrin - Hah! That's exactly what I'm using! The script abuses cpanm to do that part of the dirty work, then scans the dir full of dists and builds the indices (indexes?). The last thing it does is spit out a command I can copy and paste to use cpanm to build and install the whole shebang (though that part is still kinda kludgy and specific to my work environment).
I've been using MyCPAN::App::DPAN for some time now. It features a dpan script that can either rebuild indexes from scratch or incrementally add to them based on new material dropped into place in the authors/id/ directory structure. At present I have well over 500 CPAN distributions managed this way, plus a few private ones. The directory structure features authors/ and modules/ directories, complete with appropriate index files, so the DPAN can easily be added to the urllist in the CPAN shell just like any CPAN mirror.
For over 99% of the CPAN modules I have in my DPAN the index creation, via MyCPAN::Indexer, works without a hitch. For those that don't, there is a simple way to provide a manual override to what the indexer comes up with.
Adding new modules is as easy as dropping them into place, re-generating the indexes, reloading indexes in my CPAN shell, and installing. The approach is more command-line driven than CPAN::Mini::Inject appears to be, which has suited my needs fine. It seems to share the same basic initial goals as CPAN::Dark in that you don't start by mirroring anything, you only put in what you want, you control the entire layout of your DPAN, and you can incrementally rebuild indexes upon adding new material.
This is awesome though I think I'd prefer to see a name like CPAN::Private or CPAN::Local.
One other thing, that was partially already mentioned in the comments, is that
I'd like to fix the module versions from CPAN and I'd like to be able to manually(?) decide which ones to upgrade.
Especially in companies people are afraid of the latest and breakest from CPAN so they want to make sure they have full control of what version of each module they install.
Is the name right?
I was going to suggest CPAN::Private or CPAN::Local too. CPAN::Dark doesn't tell you what it does unless you happen to know what a DarkPAN is.
CPAN::Private seems more semantically correct to me, since CPAN::Local kinda implies just a local mirror.
Does this solve a real problem?
Hell yes. I have a maze of modules that have interlocking dependencies, and being able to have a "staging CPAN" that I could push releases to and then have all those other modules get the fresh version, under the various perlbrews and so on... well that'd be damn handy.
Does it need a mechanism to remove distributions?
I can't think of a specific use case, but I'd say yes: it's a private repo, you need the confidence to know you can undo actions after you've taken them, without restarting the repo from scratch.
Should it include configuration instructions for cpanminus/CPANPLUS/CPAN.pm?
Absolutely. If it could be made zero-conf then that'd be even better, but realistically speaking: if it needs stuff to be done to make it work, then it needs to have instructions on how to do that stuff.
I can see this becoming a very valuable part of the toolchain for module authors.
Have you looked at CPAN::Site?
http://search.cpan.org/perldoc?CPAN::Site
This seems to do everything you're asking : maintain local copies, make it behave like CPAN, inject new private modules. For client installations of Perl, they just point their primary CPAN mirror to the one created with CPAN::Site, and then as far as they're concerned it IS their CPAN.