Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

Pmdev documentation

by ELISHEVA (Prior)
on Aug 05, 2009 at 06:10 UTC ( #785977=pmdevtopic: print w/replies, xml ) Need Help??

Recently I compiled an annotated list of all of the database tables used by the PerlMonks website (see PerlMonks Data Model). bobf has expressed an interest in collaborating on it with me and I think that is a wonderful idea. In fact, we'd like it if we had an entire collection of pages that pmdev's could add remove, and edit as a group without godly intervention.

As anyone who checks the data model on my extra scratchpad will note, it is nearly at the 64K limit and there is still much information to add. It badly needs to be broken up into subsections, each with their own page. And of course those pages need to be communally edited. It would be even better if (a) we could see an edit history of all edits with undo, redo ability (b) we could assign pages to multiple categories and have category lists auto-generated (a la mediawiki). We have found both those features extremely useful on large collaborative documentation projects (including the mediawiki documentation itself!).

A collaborative environment for pmdev documentation could also be used for much more than just the annotated data model on my extra scratchpad. The documentation for pmdev is scattered in many places, including the wayback machine. A collaborative documentation environment would let us consolidate the existing documentation into an internally consistent corpus and make it easier to keep it up to date.

The ideal environment for collaborative pmdev documentation may require some work. In the meantime, bobf and I are eager to get to work and we already have some things that come pretty close to what we need. As an initial first step, we were wondering if we could copy the infrastructure used by SiteDocClan to create a set of internal pmdev documentation parallel to the public site documentation?

This would involve creating the following new nodetypes:

  • DevFaqlet:: similar to sitefaqlet. These would be used for group edited documentation pages. pmdev's should have full rights to add, delete, and update the pages. Deleting is important because leaving around irrelevant documentation can be very confusing. When pages are broken down into subpages, one doesn't always get the arrangement right the first time. It is important to be able to reorganize information easily without leaving the old not-so-good organization lying around. The cabal should have read access.
  • DevFaqlist: similar to faqlist. These would be used to thematically group devdoclets. pmdevs should also be able to add, remove, and edit these lists at will. The cabal should be able to read them.
  • DevFaqstring: similar to faqstring. Again, pmdevs should also be able to add, remove, and edit these at will. The cabal should be able to read them.

Using existing infrastructure also has another advantage. As we begin working on documentation together, our ideas about what we really need will take a firmer form. Once we have more experience working together, we can reopen the issue and discuss requirements with more clarity.

Best, beth and bobf

Replies are listed 'Best First'.
Re: Pmdev documentation
by jdporter (Canon) on Aug 05, 2009 at 13:12 UTC

    My recommendation is simple: Have the individuals who will be doing this work be in both pmdev and SiteDocClan. SDC is quite capable of creating group-editable nodes. And since this work is to be documentational, it even makes logical sense for SiteDocClan to be involved.

    Between the mind which plans and the hands which build, there must be a mediator... and this mediator must be the heart.

      That is certainly a good short-term solution. However, if the intention is that over time pmdev will expand and maintain this documentation, then the membership of pmdev will need to become a subset of SiteDocClan. I have no issue with that (and in fact, it may be a good idea), but I wanted to raise it for discussion.

      Some questions:

      • Is there any benefit to having two easily identifiable but separate corpuses of documentation? Having a separate node type makes it ridiculously easy to write a query that extracts only the development documentation or only the user documentation. If they all share the same node type then we can only separate them by hand tagging them.
      • Is it possible that we may want some special behavior, data or a toolkit that applies to dev documentation but not end user documentation? Having a separate end-user and dev documentation nodetypes would allow that more easily.
      • Are there any security concerns in making pmdev documentation part of site documentation? The entire monk community is a reader of site documentation, not just the cabal.

      Given these questions, I think it might make more sense to have a supertype "PmFaqlet" from which both "SiteFaqlet" and "PmdevFaqlet" inherit. Or alternatively, make "PmdevFaqlet" inherit from "SiteFaqlet". This gives us the most flexibility down the road.

      On the other hand, I think it is more important that we move forward while the enthusiasm is high. If creating separate nodetypes is going to take an inordinate amount of time (more than a week), then provided we have a way to hand tag the pmdev documentation, I'd rather get something less than ideal now and clean up the mess later.

      Best, beth

        Is there any benefit to having two easily identifiable but separate corpuses of documentation?

        Yes. The better question is, How much benefit is there, and is it the worth the costs? The subsequent question is, What is the best way to implement it? "Easily identifiable but separate" is pretty abstract. I think the end could be achieved by using the existing sitefaqlet/faqlist infrastructure; the dev docs would be kept in a separate set of faqlists, for example.

        a query that extracts only the development documentation

        Extracts? What for? Having the dev docs in their own faqlists would make it easy enough, for practical purposes. More important, I think, would be the implications for Super Search. Would we want it to search the dev docs? IMHO, probably yes. But if not, we could rig Super Search to skip the dev docs (except for pmdevils, maybe). I've already begun work on an infrastructure for making Super Search ignore faqlets which are linked in certain faqlists. It could be used for this as well. Also see my answer to the Security question below.

        Is it possible that we may want some special behavior, data or a toolkit that applies to dev documentation but not end user documentation?

        Yes. I would say that at such time as we identify such a thing, it will be easy enough to create the new node types and convert the existing docs. I'd argue for deferring just on the You Arent Gonna Need It principle.

        Are there any security concerns in making pmdev documentation part of site documentation?

        If we don't want non-cabal to read these docs, we could make them alphafaqlets. In fact, this is what I would recommend.

        But note: currently only SiteDocClan can create/read/edit alphafaqlets. I have submitted an enhancement request for alphafaqlet access to be expanded to all of Cabal; but so far, it has been tabled by the gods. Now would be a great time for them to reconsider it. :-)

        I think the bottom line — the essential fact — is that converting nodes from sitefaqlet to pmdevdoc (or whatever we call it) , at such time as that type is created, will be trivial. I agree with you - let's get started on the docs, and worry about the infrastructure later.

        Between the mind which plans and the hands which build, there must be a mediator... and this mediator must be the heart.
Re: Pmdev documentation
by ELISHEVA (Prior) on Aug 07, 2009 at 04:42 UTC

    After a discussion in the chatterbox this morning (2009-08-07, around 4AM UTC), a godly decision was made to put the pmdev documentation in separate nodetypes from the sitedoc documentation. tye gave two reasons for the decision. Using sitedoc nodetypes:

    • messes up assumptions made by 'super search'
    • messes up assumptions made by "search faq" functionality

    In the course of the discussion, we also concluded that we need to look more closely at any additional changes that may need to be made beyond adding nodetypes. ysth mentioned in particular, the edit history and htmlpages. While it may be possible to minimize or avoid any additional changes by having the new nodetypes inherit from the sitedoc node types, this needs to be investigated. tye delegated the work of investigation to the non-godly pmdevs.

    The cb conversation follows:

    • bobf: Have we reached consensus on Pmdev documentation?
    • tye: why does it have to be consentual?
    • ELISHEVA: I think bobf means do we create new nodetypes or use existing sitedoc nodetypes.
    • bobf: It doesn't. I'm just wondering what the next steps are.
    • ELISHEVA: new nodetyes requires consent and action of gods :-)
    • ysth: new nodetypes seem reasonable to me (but I should read that thread)
    • ELISHEVA: existing sitedoc nodetypes we can just do (bobf and I are both already in SiteDocClan)
    • ysth: tye can we reuse some of the ones you deleted? (just kidding)
    • tye: you expect that is possible?
    • ELISHEVA: I would prefer new nodetypes - less error prone than jdporter's suggestion that we keep track of pmdev docs manually by adding to a list.
    • ELISHEVA: (its easy to forget to add it if one is working quickly)
    • tye: sitedoc nodetypes would not be the best choice. it messes up assumptions made by 'super search' and, more importantly, the "search faq" functionality
    • bobf prefers new nodetypes, too
    • Anomynous Monk objects, but only to prove that you can proceed without consensus
    • tye: if there is code required, then pmdev, heal thyself (making a coupla nodetypes is easy)
    • Anomynous Monk: but then sock puppets never count
    • Anomynous Monk: new htmlpages needed. new case in the edit history stuff.
    • ELISHEVA: yes but only gods have create permission for nodetypes.
    • ysth belatedly depuppets
    • tye: because they can't see their own fingers
    • ELISHEVA: if the new nodetypes inherit from sitedoc node types don't they also inherit the htmlpages?
    • ysth: ELISHEVA yes, but I thought there was something there that needed changing.
    • tye: I don't know what needs changing. That isn't a requirement for only gods to figure that out, so I'm not going to. :)
    • tye: These nodetypes have been duplicated before already. The pattern shouldn't be that hard to identify.
    • ELISHEVA: I think for now the main issue is categorization. More concern is edit history (i.e. if nodetype name is hardcoded rather than just taken from the current record) - but that can be investigated.
    • ELISHEVA: and yes research is something a pmdev can do w/o being a god. :-)
    • ELISHEVA: ysth mentioned them being duplicated for pedagogues a few days ago - it that what you mean?
    • tye: as for policy, new nodetypes seems the way to go. I just remember hearing that they'd been duplicated.

    Best, beth

      elisheva, thank you immensely for posting that conversation. I wish I'd been there to be a part of it. :-) I have to wonder if there was more, or if that was the end of it. Because what I don't see elaborated are the assumptions which would be messed up by using sitedocs. Both of the concerns which you have bulleted are really the same, and both (or I should say, it) would be mitigated by using alphafaqlets, as I said elsewhere. Alphafaqlets are not searched by Super Search (and thus would not be found in a FAQ search) yet are readable by pmdev and editable by SiteDocClan. On the other hand, I don't see any problem in storing these anticipated pmdev docs in publicly visible nodes. Do they need to be any less public than, say, the Everything Bible? If it is determined that there is some knowledge which is just too sensitive for that, it could be sequestered in the pmdev HowTo wiki (e.g.)

      I want to be clear about my reluctance to creating a new nodetype for this: I am resistant to any potential notetype proliferation. I think that before a new nodetype is created, alternative solutions should also be weighed. In my opinion, this has not been satisfactorily done.

      Let us review how this all started. Elisheva produced a bunch of documentation on ELISHEVA's extra scratchpad. It was observed (by me and probably some others) that the information, once completed, would be worth preserving in an official pmdev doc; the pmdev HowTo wiki would make sense for this, but it is already nearly full, and it was noted that ELISHEVA's extra scratchpad's contents are just shy of the 64kb limit. What, then?

      As software developers, we automatically began to think about generic solutions with an eye toward scalability, and other considerations along those lines. We observed that the existing site doc infrastructure has and does what we would need for maintaining a corpus of pmdev documentation, the only serious difference being in access rights ... and this in turn is something which would be very easily (and naturally) handled by making a new nodetype for the purpose, with its proper Creators/Updaters/Deleters.

      Does that mean that this is really the only good way to solve the original problem? I don't think so. In particular, what seems to have been ignored is the fact that we are looking at about one wiki's worth of information, so far. How much more do we really antipate? Do we really think that the pmdev doc set is going to keep growing and growing? I think that's unrealistic. What we're really looking at is the immediate creation of one document's worth of info, maybe two. I just can't see making a new nodetype just for this — at least, lacking any really peculiar display/edit/access requirements, which so far haven't been demonstrated.

      So rather than take a step which does nothing more than further complicate the already complicated nodetype picture, why don't we do the simplest thing which meets the immediate need: create another pmdev-owned wiki and move ELISHEVA's extra scratchpad's contents to it?

      And, as I already tried to say elsewhere, such a measure in no way cuts off the future possibility of converting to a new nodetype if it is finally determined really to be necessary. Creating a new nodetype now would, imho, be an example of violating the You Arent Gonna Need It principle of system design.

      Btw... I raised a number of points and asked a number of (non-rhetorical) questions in Re^3: Pmdev documentation, which, so far, no one has deemed fit to address. Just because I'm not on board with your plan, doesn't mean my ideas are worthless and I should be ignored.

      Between the mind which plans and the hands which build, there must be a mediator... and this mediator must be the heart.

        I wish you had been there too. That was in fact the entire conversation, and you are quite correct, the precise nature of tye's concerns were not explored, though perhaps they should have been. Hopefully, tye will add a more detailed explanation of his concerns to this thread.

        I, of course, don't feel as uncomfortable with tye's decision as you do.

        For one, I find the idea of mixing up site documentation with pmdev documentation very confusing. Sitedocs are end user documentation. pmdev docs are developer documentation. Different kinds of people volunteer to write them. They need different editorial styles and appeal to entirely different audiences. For decades we've been shipping applications with seperate technical and end user documentation, precisely for these reasons.

        I also think you are seriously underestimating the number of nodes needed to do a good job of development documentation. If we were only talking about 4 or 5 nodes, a manually maintained faqlist would problably work well. However, I've never documented a system this complex with so few nodes. Nor have I seen any other comparable system documented well with only 4 or 5 nodes.

        But even if it were true that we can fit all the documentation into 10 or so pages of solid unbroken text (640K), cramming as much as possible onto each page would hardly be advisable:

        • Editing 64K page of documentation just isn't practical. Every save brings you back to the top of the document, scrolling back down over and over again is error prone and time consuming.
        • Technical documentation benefits greatly from rich hyperlinking and hyperlinking works best when documents cover short discrete topics. If each topic has its own document, then we can link to the topic by linking to the document. If we cram topics into a page until we run out of space, then we have to do links internal to the document. These links tend to be a lot more work to set up (we have to manually add <a name="blah"> tags. They also tend to be less stable: it is all too easy to wipe them out simply by editing the document. Link rot can also happen with document ids, but it is a lot easier to write a program to find those than it is to write a problem to verify "#blah" links.
        • Crowding too much information onto a page tends to strangle the various subtopics on the page. A topic is more likely to get full detailed treatment when it has a page of its own.
        • Moving detailed topics onto their own page makes overview pages easier to scan. They can then become learning tools that give new developers a chance to see the big picture. In a complex system overview pages are especially important because they help people understand the range of things to be learned.
        • If the documentation is well organized, overview pages also help people blackbox parts of the system that they don't have time to learn. This means that someone can get up to speed on a particular part of the system without having to understand the whole.

        Would you still consider a manually maintained faqlist a good way to manage documentation if we had 20,30,50,100 nodes? I would contend that we'd be spending a lot of time simply making sure all the nodes we created ended up on that list. On the other hand, if the nodes are part of a separate node type it just happens automatically. We can then spend the management time on something more useful.

        You suggest that wanting new node types violates the YouArentGonnaNeedIt principle. However, that principle applies to adding features that you don't have any use for (yet). Adding new node types doesn't add new features. It reuses existing features (sitedoc infrastructure, node types, inheritance) for a new purpose. By your own admission the rationale for creating new nodetypes (permissions) is legitimate. And even if we were adding new features, we'd be adding something we'll use right away, not at some fuzzy time in the future.

        Finally, in Re^3: Pmdev documentation, you suggest there is a cost to adding new nodetypes but you haven't elaborated what costs you are concerned about. It is hard to have a discussion about costs that aren't specified.

        In theory, we should be able to create a set of subtypes of SiteDoclet etc and be done with it. After all, we only want to change the permissions and the Everything engine supports inheritance for all of the rest. Unfortunately, the first few htmlcode nodes I've looked at all have "SiteDocClan", "SiteDoclet", etc hardcoded throughout. This leaves us with the following options:

        1. copy all of the relevant nodes and then do a massive search and replace. This adds yet another N htmlcode etc nodes to the mix, but it may be the least disruptive)
        2. refactor the nodes with hardcoded strings. Instead derive this information from the nodetype. This would eliminate the need to copy,search,and replace. And if we ever wanted to do this again, it would save us time in the future. On the other hand, PM doesn't really have a robust test environment, so maybe copying is safer. If we break anything, at least we'll be breaking something only a few people need to use rather than a widely used production system.
        3. copy and refactor. Eliminates risk of disrupting existing systems, but still adds N new nodes that are very similar to existing nodes. On the other hand, we could use the pmdev documentation to test the refactoring and once we gained confidence we could gradually move the sitedoc to the refactored solution and get rid of the almost alike nodes.
        4. Begin documenting in sitedoc, but work on one of the above solutions. When the pmdev nodetypes are done, change the nodetypes on the documentation.

        My own preference is for 3+4: temporarily using sitedocs while working on option 3 (make and refactor copy). I am concerned about the time involved in refactoring and the very slow run-debug cycle (since I can't apply patches or create a development server). On the other hand, I don't know whether even temporary residence in sitedocs would be a problem and that brings us back to the question that wasn't really discussed: how having pmdev docs mixed in with sitedocs affects search logic.

        Best, beth

Re: Pmdev documentation
by ELISHEVA (Prior) on Aug 12, 2009 at 18:06 UTC

    As promised at the end of last week, I've prepared a list of what has to be done to replicate the sitedoc system for internal documentation. I've also found a few possible security issues, discussed below

    Since they are spread out among several different nodes and I'd like feedback on them as a group, I'm listing them here instead of submitting several disconnected patches. A number of these changes have the added benefit of making the site doc infrastructure easier to reuse should we find a need for yet another discrete documentation collection. The document collection interface can be grouped into several subsystems, each of which has its own set of changes: (links lead inside the readmore tags.

    I've tried to check the code below for typos, but I haven't run it (or what I can of it) through a perl compiler so I expect there are many I have missed. If you have suggestions or recommendations for the best way to check code snippets before I submit patches, please let me know. And, of course, if you see something that needs to be fixed, please either message me or add a note below.

    A lot of information about the nodetypes involved in site documentation is repetitively hardcoded throughout htmlpages, accessrules, opcodes, and htmlcodes. The list below documents all the places where this happens and tries to suggest solutions that reduce or at least contain the amount of hard coding. Ideally though, this kind of data should be moved out of the code nodes and stored in database fields or settings where it can be stated once and all can access it.

    Moving the data out would allow us to treat each of these collections (tutorials, QandA, site documentation, pmdev documentation) as instances of a single type "docball". With such a common type it would be easier to patch and/or add features to all such collections at once. It would also make it easier to create new document collections. With three existing document collections (sitedoc, catqa, tut) and a new one proposed (pmdv), clearly a docball is not a singleton object. Ideally the decision as to whether or not we create new collections should be driven by the benefits of having a separate collection and not by concerns about the time it takes to find (or correctly patch) all the places that need to be changed.

    Best, beth

Log In?

What's my password?
Create A New User
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (5)
As of 2021-04-21 06:14 GMT
Find Nodes?
    Voting Booth?

    No recent polls found