Hi Ken. Right now at work our darkpan purely consists of a dump using minicpan. If you wish to replicate https://metacpan.org locally you could do so, the docs for metacpan-api and metacpan-web should be useful in this regard, there's even a docker image to ease replication/installation, but this may be overkill depending on your needs/infrastructure.. Failing that you may be interested in App::lcpan. minicpan_webserver has some further documentation not mentioned in CPAN::Mini::Webserver.

Update: Should have mentioned, the self hosting suggestion for Metacpan gets around the potential lack of direct internet access darkpans may have.

G'day marto,

Many thanks for your suggestions. Some of these have already been looked into. Here's a brief summary of those.

• Work on creating a local CPAN mirror was started quite some time ago (before I joined the company). It ran into various difficulties and was never fully operational. Fixing (or finding an alternative to) this is part of my current task. For now, I've put this into the overkill bucket; although, that doesn't preclude revisiting at some future point.
• My current research looked into the option of using MetaCPAN locally; I did encounter Docker which you mentioned. While not entirely abandoned, I felt this also fell into the overkill bucket.
• I had looked at both CPAN::Mini::Webserver and minicpan_webserver. The latter does have more documentation; although, how to achieve various tasks is not shown. As an example, it uses static CSS, via CPAN::Mini::Webserver::Templates::CSS: I imagine there would be some way to either tweak what that provides, or to specify a completely different style sheet, but there is no information on how to do that.
  Compare with Pod::Html (and, by extension, pod2html) which has a fairly straightforward --css=stylesheet option. CPAN::Mini::Webserver uses Moose but I can't see any obvious 'has css' statements; the code is fairly long, and includes a number of use and extends statements, so I imagine I will eventually be able to find what I need — it's just annoying that I need to do this at all.

Thanks for pointing me towards App::lcpan [for anyone else reading this, that's a lowercase case 'L', not an uppercase 'i']. I've had a brief look at this but will need to spend more time on it. There does appear to be a lot of documentation, especially in App::lcpan::Manual; at present, a lot of that is just "TODO" but the author (fellow monk, perlancar) appears to be working on that at the moment — I see four distribution updates in the last three days.

Overall, there's nothing currently set in stone. I'll be starting a proof-of-concept tomorrow using the DarkPAN solution outlined in the OP — of course, that may change as work progresses. Given tomorrow is only a few minutes away in my timezone, I should probably get some sleep first. :-)

— Ken

A quick look at the source of CPAN::Mini::Webserver::Template::CSS shows all the css in there. It wouldn't take much work to create patch to allow reading from another file.

Hi Ken,

I've just added a tutorial page in App::lcpan::Manual: App::lcpan::Manual::Tutorial::700_SettingUpADarkPAN. If you have more questions, I'll gladly help.

I set up a Darkpan recently. It was intended only to provide some internal modules, making them available for installation using standard CPAN tooling. I didn't need presentation, so my webserver is just minimal Mojolicious::Lite app that serves the tarballs in a CPAN-style file hierarchy as static content. To build the Darkpan I used CPAN::Mirror::Tiny. It works just fine.

This solution doesn't provide a web GUI, just the barebones file structure needed by CPAN tools like cpan and cpanm. The latter is what our consumers mostly use, via Carton.

We build our distributions into a Darkpan prototype on a build server, and then rsync the files that got modified with new versions over to a mirror that has both Beta and Production level Darkpan mirrors on it. We don't delete old versions; we keep them around so that consumers can pin to a version, and so that consumers can roll back as needed.

G'day Dave,

Thanks for the information. Along with the ability to use tools like cpan and friends, a web GUI with functions including search facilities and a POD viewer will be required.

I had a look at CPAN::Mirror::Tiny. Its "WHY NEW?" section appears to promote its "no dependency on XS modules" as one of the main reasons to use it. There are no ($work) issues with XS modules; in fact, they would generally be preferred over pure Perl equivalents for the speed improvement.

— Ken

That makes sense. My use-case simply didn't involve GUI browsing. It's pretty easy to read the 02 index file either way. What I found CPAN::Mirror::Tiny to be useful for, was in taking distribution tarballs, using them to build the required file structure and index file, which is the bare essentials of a Darkpan.

I suspect if I wanted browsability, I would probably still use CPAN::Mirror::Tiny to build my darkpan, and then use a different module as the business end of a web GUI. I don't see this module as excluding the use of a GUI-providing module. But it is entirely possible there's a more suitable alternative for your needs.

Incidentally, I actually intend to talk about building a Darkpan at the online / virtual version of the Perl Conference this year. But my talk will spend some of its time on how to convert internal use modules into distributions that are suitable for an internal darkpan, and the strategies we used for building and shipping to the mirror. I hadn't intended to get into anything regarding a front end GUI.




Dave

G'day djerius,

Thanks for your response. Following my quick PM, at just after midnight this morning, I've now had a chance to look at these modules.

A "file based, rather than server based" solution is not really what I'm after. I did note that a OrePAN2::Server module exists: I've only looked at this briefly.

I came across Pinto a few years ago for an unrelated, personal task. It seemed to be a work-in-progress back then; phrases like "subject to radical change without notice" were of concern at that time. There has been no further development in almost three years.

— Ken

pinto works well. former $work* used it heavily. The pin and stack concepts are nice for having known exact versions in your system, by pinning them at the repo level rather than in your modules.

Jeffery had a hosted version at stratopan, but I think he's let that go. A shame, this is what happens when we don't support community projects, they fall by the wayside.

hint, sounds like RipZecruiter.

the API link, at the top of MetaCPAN pages, leads to a "YAPC video" which discusses MetaCPAN::API; however, that module's documentation is very clear (using large, bold, block capitals) that it is deprecated and indicates that MetaCPAN::Client should be used instead.

The "API" link at the top of the MetaCPAN pages has https://fastapi.metacpan.org/ as its target. This simply redirects to the API documentation in the github repo. I presume at that point you then followed the "watch the actual talk" link. Instead if you just read the page at github you will see it says

Perhaps the easiest way to get started using MetaCPAN is with MetaCPAN::Client.

which is indeed the current advice AFAIAA. Maybe a short PR to mention that the video (and slides) have been superseded somewhat is all that's needed to clear that up? They are from seven years ago and a lot can change in that time, after all.

G'day hippo,

"The "API" link at the top of the MetaCPAN pages has https://fastapi.metacpan.org/ as its target."

Yes, you are correct; however, given that's exactly the same target that I used for the "API" link in the second paragraph of my OP, I don't understand why you thought it necessary to state that. Perhaps I've misunderstood your intent. Just to be clear, the "API" links at the top of MetaCPAN pages and in my OP, both have the same target: https://fastapi.metacpan.org/.

The purpose of the three sentences in the second paragraph were:

1. Here's something that's changed which I knew about.
2. Here's something that's changed which I discovered as part of my research.
3. There may be other things that have changed which I don't yet know about — please tell me.

"They are from seven years ago and a lot can change in that time, after all."

That's sort of my point: a lot can change and I don't know if I'm aware of all changes.

"I presume at that point you then followed the "watch the actual talk" link."

Yes, that's the "YAPC video" link in my OP. While watching it, I decided to follow along with the MetaCPAN::API documentation to supplement what was being talked about: that's when I discovered the deprecation notice.

— Ken