Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

RFC -- Evolving Perl: a Decision Theory Approach to the Challenges of Perl 7

by thechartist (Monk)
on Jul 07, 2020 at 18:16 UTC ( #11119011=perlmeditation: print w/replies, xml ) Need Help??

The announcement of Perl 7 has created a lot of activity in the various techology communities I follow. It has also lead to a lot of discussion and debate among a number of subsets of the Perl community.

Last week we had a discussion on "software quality" in the Perl Programmers Facebook group. "Quality" is a term that is not defined whenever it pops up. I pointed out that "quality" for a typical "real world" programming problem can be seen as a Multiple Criteria Decision Analysis, or a Multiple Objective Optimization Problem, that attempts to balance the following:

  1. Validate that inputs are appropriately transformed into the correct/desired outputs.
  2. Handle errors, exceptions, etc.
  3. Does so in a computationally efficient manner (ie. time and space efficiency). The algorithm doesn't need to be provably optimal, but it should be suitable for the *known* needs of the application at the time of writing.
  4. Delivered in a verifiable/testable form as quickly as possible.
  5. Comprehensible to other team members.
  6. Able to be relatively easy to change, if application specification changes.

I'm sure there are other objectives I omitted. I think it should be obvious that:

  1. these dimensions need to be traded off against one another and
  2. these criteria aren't all equally weighted. Context determines what criteria are most important in a particular time.

From a formal point of view, there are generally no unique/optimal solutions, only sets of satisfactory ones (Multiple Criteria Decision Analysis).

Some of the themes of the p5p messages after the announcement included:

  1. What is the new contract with Perl users re: backward compatibility?
  2. What should be done in newer versions of Perl to attract new users?
  3. What should be done to ease the implementation of new features, and reduce the complexity of the Perl interpreter, toolchain, etc.
  4. What can be done to facilitate a better process for the community to make better decisions?

The conclusions I've come to (as a former Perl hater, now Perl convert) is:

  1. Perl as a technology, scales from small one-liner throw away scripts, to massive code bases that span decades. There aren't many other languages that can boast about that.
  2. Perl, as a community, has learned a lot of lessons in producing reliable software. These lessons become part of Perl culture, and produce better developers.

In terms of point 4 above (Perl community process), I'd like to suggest borrowing some methods from the operations research/decision analysis communities (Tools for Decision Analysis).

The criteria I see the Perl community attempting to balance are:

  1. Perl interpreter complexity. Certain things need to be changed to improve computational efficiency, ease maintainence, and enhance the language.
  2. CPAN backward compatibility. Too much breakage with CPAN, and we have a new language with no code base.
  3. Increasing use of Perl in newer areas (ie. attracting new users, especially newer developers who can hack on the toolchain).
  4. Teaching the language, and good software development practice generally.

An approach worth considering can be seen in the Ada community. Initially standardized in 1983, each revision (1995, 2005, 2012) has been accompanied by a Rationale document, that describes the changes and the reasons for them. Most of the changes have been addiitions or extensions to the language.

Any rationale for future versions of Perl will be a bit more challenging, in that some practices will need to be revised. To explain and defend such changes, an explicit, pairwise comparison of what criteria are improved, and how they are traded off vs. other objectives, needs to be considered.

I personally would like to see Perl more involved in so-called "data science." My use case for Perl is statistical natural language processing to assist in the meta-analysis or evidence synthesis of research in health care specifically, but in applied science more generally. I believe those tools could be helpful in exploring the CPAN code base to collect empirical data on what features of Perl are most used, and which could be dropped with minimal loss.

If I'm thinking about this correctly, the problem the Perl community faces is one known in the Prolog and logic programming communities as inductive specification recovery. If I were to translate the informal problem facing the Perl community into a formal statement, I'd say we are attempting to discover (induct) the Perl grammar that simultaneously:

  1. Reduces CPAN Breakages
  2. Reduces Perl interpreter code complexity

Damian Conway has shown how hard this path is via human effort Three Little Words.

To calculate these things require some metrics. I think the CPAN breakage is reasonably easy to calculate. But the interpreter complexity metric is something that should be given some considerable thought.

With an explicit decision model, I believe we can use statistical methods from reliability theory and machine learning to develop software that will ease the burden of transitioning Perl to a more sustainable future.

  • Comment on RFC -- Evolving Perl: a Decision Theory Approach to the Challenges of Perl 7

Replies are listed 'Best First'.
Re: RFC -- Evolving Perl: a Decision Theory Approach to the Challenges of Perl 7
by jcb (Vicar) on Jul 12, 2020 at 04:01 UTC

    The major problems I see are that Perl historically has been a different language with each major revision, although perhaps Perl 6 was too much of a difference.

    Could a more conservative approach to Perl 7 be useful, perhaps limiting the breaking changes to making use strict default when parsing from a file (meaning, except for perl -e ... one-liners), making use warnings default in all cases, enabling all non-experimental features from the end of Perl 5 by default, and dropping the Perl 5 experimental features that were not promoted to core?

    XS will be "fun" no matter what we do; there are already compatibility shims for XS/perl within the 5.x series. While I think that the burden of introducing "XS2" would be excessively high, it might just be workable if XS2 extensions can still be compiled into XS and used with perl 5 and XS2 gives significant benefits with Perl 7.

    The key here is not breaking backwards compatibility without a very good reason and "PBP says that is bad" is not a good reason.

Re: RFC -- Evolving Perl: a Decision Theory Approach to the Challenges of Perl 7
by perlfan (Priest) on Jul 12, 2020 at 15:06 UTC
    Thank you, nice write up thechartist. I agree with the fully rational approach. Regarding data science, I am sure you are aware of PDL and BioPerl. You could probably throw together a blag post or even meta module on CPAN that ties in Perl interfaces to the littany of Apache projects and others that seem to always get mentioned with Data Science. An effort like that would always identify what is missing and create a hit list. I've often thought about doing the same thing with machine learning and computer vision (e.g., OpenCV support on parity with Python), even though it's only something I reach for very rarely I know that it needs someone to be the driver. One day maybe I will have time and need for that.

    That said, Python has likely consolidated enough of this information to create a nice target list of things to support. The trick is finding others in our community that would contribute to creating the critical mass to create a Data Science "community". Sounds like you definitely have the energy to blaze a trail in that regard.

    But to make a reference to the confusion on why Roku was not as big of a hit as it was, my only comment is that it's not language features that get a use domain excited about a language; it's the availability and maturity of the tooling they need. So if there is a domain that someone wishes to see Perl or Roku being used as the primary medium for tools, then a substantial effort needs to be made to create the tooling. Sometimes this happens organically (most things in Perl, in particular PDL). Some things are astroturfed into a community by the full weight of the US Government (e.g., SciPy and friends - and most modern things tbh).

      Hi Perlfan,

      I've been going through a number of 'Meditations' here on Perl Monks, some of Larry Wall's historical statements on the motivation of Perl, watched Sawyer X's discussion on Perl 7, and had a number of discussions on the Perl Programmers Facebook group on a "wishlist" for Perl 7. Synthesizing these things deserves a separate Meditation on its own.

      FYI -- there have been a number of discussions in other forums RE: Making Perl more competitive for 'data science' applications. I think PDL is a great option currently, but the documentation and tutorials need development. I'm looking into what has been done previously with Jupyter; I'd like to have Perl PDL integrated with Jupyter notebooks to draw some attention to the current capabilities of Perl. I know there is a CPAN Jupyter Kernel, but I haven't tested it yet.

        This thread was TL;DR and I'm probably missing the point ... but regarding

        > Jupyter notebooks

        Johann Rolschewski gave a lightning talk on integrating Perl in Jupyter on the German Perl Workshop 2020 (the biggest real Perl Workshop 2020 ;-)

        His slides are available in English

        http://jorol.de/talks/2020-GPW-Jupyter/#10

        I think there must be a video too on YT, tho in German.

        Cheers Rolf
        (addicted to the Perl Programming Language :)
        Wikisyntax for the Monastery

A reply falls below the community's threshold of quality. You may see it by logging in.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://11119011]
Approved by Tommy
Front-paged by Tommy
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (3)
As of 2020-09-19 01:45 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    If at first I donít succeed, I Ö










    Results (114 votes). Check out past polls.

    Notices?