Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options

Re: Pure Perl Modules, XS Modules, what's the current trends? (allowing easy choices)

by tye (Sage)
on Dec 20, 2007 at 20:36 UTC ( #658236=note: print w/replies, xml ) Need Help??

in reply to Pure Perl Modules, XS Modules, what's the current trends?

The best route is to have a pure-Perl module that provides the interface and have an XS module that requires the pure-Perl interface and is used via the pure-Perl interface (yes, this can be done efficiently and easily). That way, if you want/need the XS implementation, then you just install that one XS module (and your install tool likely installs the interface module automatically because of the declared dependency). If somebody else has difficulty installing an XS module, then they just install the pure-Perl one and they're set as well. You don't have to try to write code to get all of the varied install tools to try to ask the user whether they want the XS piece or if they want to continue installing even though the XS part didn't build or other such complex magic.

I've seen several modules go that route though I don't recall the names of any of them at the moment.

If the XS module is wrapping some external libraries, then this whole question is probably moot. Otherwise, there are usually only a few (if any) features that can't be provided without XS (most of the time, the vast majority of the features can be done in pure-Perl and the XS is just there to increase performance and the number of bugs) or just haven't been coded yet in Perl, in which case the pure-Perl replacement likely just dies/croaks (at least for now).

But you were more asking about what you should do when things didn't go this "best" route. I'd write a wrapper module that transparently uses either the XS module or the pure-Perl module and make it require the pure-Perl module. And I'd encourage the author of the XS module to be involved and to encourage users to switch to using this interface abstraction module (which, again, usually isn't terribly difficult to write while still being very efficient, though exactly how to write it will depend on the iterface being implemented) to the point of getting the XS module to require this abstraction module.

If you can also get the author of the pure-Perl implementation involved such that the interface abstraction module is actually just made a part of the pure-Perl implementation module, then you've got the original best route that I described (but the only real advantage of this last step if reducing the number of "moving parts").

Note that it is also nice to have a hook whereby the user of the abstraction module can request that the pure-Perl implementation be used even if the XS implementation is installed. This is polite but can also be important as almost always the XS implementation will have more bugs or just not work on some less-vanilla data, etc. and will certainly be harder to tweak to work around bugs, awkward design choices, etc.

Update: Here is a block diagram of the proposal for a quick grasp of the concept without trudging through the long description:

+-------------------+ |XS implementation | |Either install this| ---requires--\ +-------------------+ | V +-------------------------------------+ |pure-Perl module distribution | |Or install this | |Other tools require this distribution| | +---------------------------+ | | |pure-Perl interface package| | | |Only package used in code | | | +---------------------------+ | | +--------------------------------+ | | |pure-Perl implementation package| | | +--------------------------------+ | +-------------------------------------+ (or) +-------------------+ |XS implementation | |Either install this| ---requires--\ +-------------------+ | V +--------------------------+ |pure-Perl interface module| |Or install this | /---requires--- |Only package used in code | | |Other tools require this | V +--------------------------+ +-------------------------------+ |pure-Perl implementation module| +-------------------------------+

- tye        

  • Comment on Re: Pure Perl Modules, XS Modules, what's the current trends? (allowing easy choices)
  • Download Code

Replies are listed 'Best First'.
Re^2: Pure Perl Modules, XS Modules, what's the current trends? (allowing easy choices)
by skazat (Chaplain) on Dec 20, 2007 at 21:16 UTC

    I agree with this approach, but what I'm seeing (albeit I haven't looked at every single module that mixes in this way), that modules /used/ to be written like this, but aren't anymore, opting to go for simply an XS route and then someone comes along and writes a Pure Perl version. Which is causing me all kinds of headaches.

    For example, in MIME::Base64 2.23 (just as an example), you'll see basically what your diagram is sayin':

    eval { bootstrap MIME::Base64 $VERSION; }; if ($@) { # can't bootstrap XS implementation, use perl implementation *encode_base64 = \&old_encode_base64; *decode_base64 = \&old_decode_base64; $OLD_CODE = $@; #warn $@ if $^W; } # Historically this module has been implemented as pure perl code. # The XS implementation runs about 20 times faster, but the Perl # code might be more portable, so it is still here.

    (followed by the Pure Perl version)

    In the newest version (as of my writing: 3.07) that's gone, but out has sprouted MIME::Base64::Perl - which is probably the same implementation that *was* in Mime::Base64 to start off with.

    Seems like a step backwards.


    -justin simoni
    skazat me

      No, that first block doesn't look like what I described. That first block looks like the XS version is bundled with the interface module and they try to detect if the XS part of the one module just isn't installed. Trying to install a module distribution where you can't build the XS part is ugly. So it is nice of them to try to detect that situation but it doesn't look to me like they've really done anything to make life easy for people who have a need for that situation.

      So, splitting out the pure-Perl version into a separate package doesn't bother me one way or the other in the abstract. Splitting it out into a separately distributed module might be a step backward, I agree, though perhaps only a tiny step. I prefer to have the interface distribution be the "visible" one with the pure-Perl implementation less visible but required by (or included in) the interface distribution, since the alternate situation (people having to specifically request that the pure-Perl implementation be installed) tends to lead to less transparent fall-back.

      If the XS code were similarly split out and the XS code distribution declared that it depends on the base interface distribution, then that would be a big step forward.

      In this particular case, I suspect that part of the motivation for these changes is that this particular module looks like it was added to "core" (based on comments in the "Changes" file) and so the XS component should be built when Perl is built and so most people shouldn't have to worry about trying to build the XS part. Of course, this idea that "/it/ being in core means that everybody who gets Perl also gets /it/" is not absolute.

      For example, my primary Linux box has a Perl that is missing many core modules and that box doesn't have a C compiler installed (there at least used to be a C compiler available but it isn't easy to find nor install and I'd probably have to purchase extra hardware to do that anyway). So I don't buy the "who cares if they can't build an XS module, nearly everybody can if they just take the time" argument. Time is valuable so not having to take the time is a valuable option to provide to people. I've seen plenty of cases where time isn't the only obstacle (in my case, hardware restrictions, in many cases company policy restrictions or factors of the scale involved due to a large number of target platforms, etc).

      So it'd be nice to help these modules be better citizens toward those unfortunate enough to not have the ideal Perl build environment to work with. But my experience is that even when accomodating such cases is not complex nor difficult, many will still object to having to even contemplate the possibility at all, considering it a waste of their valuable time. So I wish you luck.

      But, yes, for this particular case, the creation of MIME::Base64::Perl looks like a backward step to me as well.

      It is sad that somebody expects one to do s/MIME::Base64/MIME::Base64::Perl/g to their code base (including to modules one didn't write) in order to allow it to work in a unfortunate environment, especially when this previously wasn't required.

      - tye        

        Thanks for the clarification and continued insights.


        -justin simoni
        skazat me

Re^2: Pure Perl Modules, XS Modules, what's the current trends? (allowing easy choices)
by chromatic (Archbishop) on Dec 20, 2007 at 20:56 UTC
    I'd write a wrapper module that transparently uses either the XS module or the pure-Perl module and make it require the pure-Perl module.

    Upvoted and seconded. Unification behind an interface that can support either version solves a lot of problems.

      I just realized that that's what they did with Text::CSV a little less than a month ago - a little bit after I did my research on Text::CSV. The world it's moving faster than I am.


      -justin simoni
      skazat me

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://658236]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (5)
As of 2022-05-23 21:00 GMT
Find Nodes?
    Voting Booth?
    Do you prefer to work remotely?

    Results (82 votes). Check out past polls.