Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Meditations

( #480=superdoc: print w/replies, xml ) Need Help??

If you've discovered something amazing about Perl that you just need to share with everyone, this is the right place.

This section is also used for non-question discussions about Perl, and for any discussions that are not specifically programming related. For example, if you want to share or discuss opinions on hacker culture, the job market, or Perl 6 development, this is the place. (Note, however, that discussions about the PerlMonks web site belong in PerlMonks Discussion.)

Meditations is sometimes used as a sounding-board — a place to post initial drafts of perl tutorials, code modules, book reviews, articles, quizzes, etc. — so that the author can benefit from the collective insight of the monks before publishing the finished item to its proper place (be it Tutorials, Cool Uses for Perl, Reviews, or whatever). If you do this, it is generally considered appropriate to prefix your node title with "RFC:" (for "request for comments").

User Meditations
Organizational Culture (Part VII): Science
4 direct replies — Read more / Contribute
by eyepopslikeamosquito
on Sep 14, 2021 at 00:02

    Publish or Perish

    I remember my dad lamenting his scientific publish or perish workplace culture, lightheartedly explaining to me one day how promotions were awarded:

    1. Each applicant's publications are placed on a weighing scale
    2. The promotion is awarded to the candidate with the heaviest weight

    Using that simple method, I doubt that Albert Einstein would've gained a promotion in 1905, due to the (meagre) weight of his Annus Mirabilis papers:

    • On a Heuristic Point of View Concerning the Production and Transformation of Light External (Photoelectric effect). 16 pages.
    • On the Movement of Small Particles Suspended in Stationary Liquids Required by the Molecular-Kinetic Theory of Heat External (Brownian motion). 11 pages.
    • On the Electrodynamics of Moving Bodies External (Special Theory of Relativity). 30 pages.
    • Does the Inertia of a Body Depend Upon its Energy Content? (E=mc**2). 3 pages.

    In fact, Albert did not score a promotion in 1905, as detailed in his 1905 Performance Appraisal:

    This is a patent office, Albert. Your job is to transform written patent applications into clear and precise language, and to study applications and pick out the new ideas of an invention. These are the priorities. Where does it say that your priorities are rewriting the rules of the Universe, unifying space and time, unifying radiation and matter, or demonstrating the existence of atoms?

    Regrettably, I had to put you down as "poor" for "works well with others" and "shares credit appropriately". You had no co-authors on your five papers, and your citations were quite skimpy: no citations at all in your June and September paper, only one citation in your April paper, and not much better on the others. You wrote that your special theory of relativity came to you after a discussion with your friend Michele Besso. But you didn't even acknowledge him in your June paper. This is an area for improvement.

    You seem to lack a flare for self-promotion. Lucky for us our PR department stepped in and changed your L/c2 equation into the much more marketable E = mc2.

    Based on his performance as a patent clerk, I cannot recommend Albert for a promotion at this time.

    -- from Einstein's Patent Clerk Third Class Performance Appraisal of 1905

    Curiously, Einstein was passed over for promotion for three long years, until he "fully mastered machine technology", remaining a lowly Patent Clerk Third Class at the Swiss Patent Office until 1 April 1906, when he was finally promoted to Technical Expert Second Class.

    See also: Performance Appraisals from the Agile Imposition series.

    The Sad Story of Giordano Bruno

    In 1600 Giordano Bruno was found guilty of heresy by the Roman Inquisition. He was then humiliatingly paraded naked through the streets of Rome (with his tongue lashed to prevent him speaking) and finally burned at the stake, with his ashes thrown in the Tiber river. His crime? Declaring that the stars are distant suns surrounded by their own planets.

    I'm still amazed at how far ahead of his time Bruno was, his bold conjecture only recently verified by exoplanet detections by the Kepler space telescope. If there is a heaven, I hope Bruno smiles every time Kepler finds a new exoplanet and I look forward to paying my respects to him at his statue, located at the site of his execution.

    Galileo was only placed under house arrest for the lesser crime of suggesting that the Earth and planets revolve around the Sun (rather than the other way around).

    Scientific Culture

    The process in the scientific method involves making conjectures (hypotheses), deriving predictions from them as logical consequences, and then carrying out experiments or empirical observations based on those predictions. A hypothesis is a conjecture, based on knowledge obtained while seeking answers to the question. The hypothesis might be very specific, or it might be broad. Scientists then test hypotheses by conducting experiments or studies. A scientific hypothesis must be falsifiable, implying that it is possible to identify a possible outcome of an experiment or observation that conflicts with predictions deduced from the hypothesis; otherwise, the hypothesis cannot be meaningfully tested.

    -- from Scientific method (wikipedia)

    Thankfully, Science has come a long way since the days of Galileo and Bruno. Today, the scientific community expects:

    • Rigorous scrutiny. In science, all ideas (especially the important ones) must stand up to rigorous scrutiny. The culture of science does not value dogma.
    • Honesty, integrity, and objectivity.
    • Credit where credit is due. Scientific research articles always provide a list of citations, crediting other scientists for ideas, techniques, and studies that were built upon by the current research. The number of citations a paper receives can help indicate how influential it was, since important research influences how other scientists think about a topic and will be cited many times in other papers.
    • Adherence to ethical guidelines.

    Update (see erix response below): the four Mertonian norms (often abbreviated as the CUDO-norms) are:

    • Communism: all scientists should have common ownership of scientific goods (intellectual property), to promote collective collaboration; secrecy is the opposite of this norm.
    • Universalism: scientific validity is independent of the sociopolitical status/personal attributes of its participants.
    • Disinterestedness: scientific institutions act for the benefit of a common scientific enterprise, rather than for the personal gain of individuals within them.
    • Organized skepticism: scientific claims should be exposed to critical scrutiny before being accepted: both in methodology and institutional codes of conduct.

    Because it undermines science, scientists take misconduct very seriously. In response to misconduct, the scientific community may withhold esteem, job offers, and funding, effectively preventing the offender from participating in science. There are also strict rules around scientific publications, as detailed at publishing process and responsible referencing:

    • After erasing all signs of your identity, your paper will be sent to 2–3 other scholars who have done research in the same field for review; they will not know who you are or who the other readers or reviewers are.
    • The reviewers will spend a few months reading your submission, double-checking your methods and or claims/citations from other source; they will then send it back to the manuscript editor with commentary and one of the following votes: accept for publication; revise and resubmit (the most common outcome); reject for publication. The reviewers may further write some commentary, for example, a need to redo the experiment using slightly different equipment, or ask you to read through certain books or articles others have published and incorporate that into your work.
    • The reviewers have to judge the work based on what you have written and what you use as evidence, not who you are or whether you have a fancy degree.

    Though the traditional Scientific method has served us well, it seems we need new methodologies to tackle the urgent problems facing us today.

    Crisis Disciplines

    On an evolutionarily miniscule timescale, cultural and technological processes transformed our species’ ecology. These changes that have transpired over this period have come about largely to solve issues at the scale of families, cities, and nations; only recently have cultural products begun to focus on solutions to worldwide problems and wellbeing. Yet we lack the ability to predict how the technologies we adopt today will impact global patterns of beliefs and behavior tomorrow ... social interactions and external feedback make it difficult, if not impossible, to reason about cross-scale dynamics through argument alone (i.e., these are complex adaptive systems).

    Humanity faces global and existential threats including climate change, ecosystem degradation, and the prospect of nuclear war. We likewise face a number of other challenges that impact our wellbeing, including racism, disease, famine, and economic inequality. Our success at facing these challenges depends on our global social dynamics in a modern and technologically connected world. Given our evolved tendencies combined with the impact of technology and population growth, there is no reason to believe that human social dynamics will be sustainable or conducive to wellbeing if left unmanaged.

    Other crisis disciplines thrive on a close integration of observational, theoretical, and empirical approaches. Global climate models inform, and are informed by, experiments in the laboratory and the field. Mathematics describing disease dynamics suggest treatment paradigms in medicine, which can be tested and validated.

    A consolidated transdisciplinary approach to understanding and managing human collective behavior will be a monumental challenge, yet it is a necessary one. Given that algorithms and companies are already altering our global patterns of behavior for financial reasons, there is no safe hands-off approach.

    -- from Stewardship of global collective behavior (cited by erix)

    As argued convincingly above, the traditional scientific method, requiring laborious peer review for example, is too slow for crisis disciplines and better alternatives must therefore be urgently sought. On a more positive note, we've been forced to do this sort of thing before - the Manhattan Project and rapid vaccine development during our current COVID-19 pandemic spring to mind.

    Other Articles in This Series

    References

    Historical References

    Updated 26-Sep-2021: Added paragraph on Mertonian Norms.

Similarities of Perl and Python?
8 direct replies — Read more / Contribute
by Bod
on Aug 30, 2021 at 06:07

    Learned and wise Monks...

    Within my business we use the usual front-end web technologies of HTML/CSS/native Javascript/AJAX and the back-end is entirely Perl. With a very small amount of Java for two Android apps.

    Here in the UK there is currently a government scheme to get young people into work. The Kickstart Scheme pays the young person minimum wage for 25 hours per week for 6 months. We have applied to take on two young people under the scheme. Most applicants have played with React at most but I am interviewing a graduate tomorrow. On his CV he has C, Python and R which has been done as part of his Physics degree plus Python and Shell Scripts on a Raspberry Pi as a hobby.

    I know absolutely nothing about R.
    From my very limited knowledge of Python, I believe it is broadly in the same group of languages as Perl. Therefore, the skills this applicant already has in Python should be relatively easily transferrable to Perl
    - is that a fair assessment???

    Any suggestions for me bearing in mind the applicant has had no real workplace experience?

RFC: Mock::Data::Regex
No replies — Read more | Post response
by NERDVANA
on Aug 27, 2021 at 04:13
    A few weeks ago I asked about iterating the characters of a character class, as part of a Yak-Shaving exercise for my not-yet-released Mock::Data module. Well, I went further, and built a regex-reverser! (generates random strings that match a user-supplied regex) I'd like some feedback on what people think of the API.
R.I.P. Charlie Watts
No replies — Read more | Post response
by kcott
on Aug 25, 2021 at 16:00

    Charlie Watts died today at the age of 80. A sad event, but obviously a good innings.

    At first, I thought this probably had little relevance here. Then I thought of the many hours that I had coding with the Rolling Stones playing in the background: Charlie brought what they said was the engine to the Stones; he also brought that same engine to my coding.

    He was a great man and had a great life: R.I.P.

    — Ken

Organizational Culture (Part VI): Sociology
2 direct replies — Read more / Contribute
by eyepopslikeamosquito
on Aug 11, 2021 at 05:09

    An important scientific innovation rarely makes its way by gradually winning over and converting its opponents: it rarely happens that Saul becomes Paul. What does happen is that its opponents gradually die out, and that the growing generation is familiarized with the ideas from the beginning: another instance of the fact that the future lies with the youth.

    -- Planck's Principle

    A person who has not made his great contribution to science before the age of thirty will never do so

    -- Albert Einstein

    Remembering some of my physicist friends lightheartedly complaining about being over the hill at their thirtieth birthday party, I was intrigued to see if programming language designers were similarly youthful.

    Some Successful Programming Language Designers

    DesignerLanguageAge
    John BackusFORTRAN30
    Dennis RitchieC30
    Bjarne StroustrupC++30
    Yukihiro MatsumotoRuby30
    John McCarthyLISP32
    Brendan EichJavascript33
    Larry WallPerl33
    Guido van RossumPython34
    James GoslingJava39
    Anders HejlsbergC#39

    As you can see, many successful programming languages were designed by men in their thirties. Perhaps programming language designers tend to be older than physicists because designing a new programming language requires more wisdom, experience and social networking skills, rather than a blinding flash of inspiration (combined with exceptional calculation faculties) often associated with physics breakthroughs.

    Curiously, these successful programming language designers seemed to have precious little experience in actual programming language design! They just boldly created a language to meet what they perceived as an unmet need in their workplace -- typically without asking permission to do so.

    I had this idea that given how much time we had available for Amoeba, I could actually build a whole new language, design and implement it from scratch, and then use it to implement our suite of tools and still be ahead of the game compared to a situation where we would have just clunked on writing the things we wanted to write in C

    -- Guido van Rossum

    Larry similarly invented Perl as a way to make report processing easier at work and to more efficiently solve Unix problems that were being inefficiently solved in a motley mix of C, sed, awk and shell ... while Ritchie invented the C programming language to implement the Unix operating system after Bell Labs withdrew from the ill-fated Multics project. Later, one of Ritchie's workmates, Bjarne Stroustrup, added ideas from Simula to C, creating C++ as a language to make writing his simulations more enjoyable than C while running faster than Simula.

    Perhaps this lone hacker language designer trend was simply a natural reaction to the vacuum created by the failure of design by committee experienced by Algol-68, PL/I and Ada.

    This paper (based on a keynote address presented at the SIGACT/SIGPLAN Symposium on Principles of Programming Languages, Boston, October 1-3, 1973) presents the view that a programming language is a tool which should assist the programmer in the most difficult aspects of his art, namely program design, documentation, and debugging. It discusses the objective criteria for evaluating a language design, and illustrates them by application to language features of both high level languages and machine code programming.

    -- Hints on programming language design by Sir Charles Antony Richard Hoare (1973)

    Though Hoare's classic paper makes interesting reading, it's hopelessly outdated, sorely in need of a modern equivalent. I couldn't find one though (citations welcome), the nearest I could find being the Socio-PLT works of Leo Meyerovich (see Sociology References below).

    Programming Language Sociology and Evolution

    Another reason for the unpleasantly large size of modern language is the need for stability. I wrote C++ code 20 years ago that still runs today and I'm confident that it will still compile and run 20 years from now. People who build large infrastructure projects need such stability. However, to remain modern and to meet new challenges, a language must grow (either in language features or in foundation libraries), but if you remove anything, you break code. Thus, languages that are built with serious concern for their users (such as C++ and C) tend to accrete features over the decades, tend to become bloated. The alternative is beautiful languages for which you have to rewrite your code every five years.

    -- Bjarne Stroustrup

    Stroustrup hits the nail on the head: if you have serious concern for your users, as both Perl and C++ do, you must not break backwards compatibility.

    Related is Simon Peyton Jones’s unofficial motto for Haskell "avoid success at all costs" because becoming too successful harms the ability to modify Haskell ... an evolutionary dead end. I might add that Haskell seems to have been successful in its mission to avoid success. :)

    It seems to me that Perl and Raku have (finally, much too late) got it right: Perl does not break backwards compatibility (out of serious concern for its users), while Raku is free to innovate and improve without the shackles of backwards compatibility ... though as it becomes more widely adopted it will presumably need to pay more attention to backwards compatibility.

    Improving Programming Language Adoption

    Social factors seem to have a far greater impact on language adoption than technical features.

    From Sociology we can learn of many ideas that might be tried to improve language adoption:

    • Identify opinion leaders. Apparently, this approach proved effective in promoting safe sex to fight HIV/AIDS in the 1980s. Given how heavily Google invested in C++, Java and Python I was surprised by the growth of Golang, perhaps this can be partly explained by Rob Pike being an effective opinion leader at Google. Similarly, now that Guido moved from Google to Microsoft, will we see an increase in Python adoption at Microsoft and a decrease at Google?
    • Observability. This worked well to increase Coverity static code analysis adoption simply by running the customer's code base through Coverity and having it find hundreds of long-standing bugs the customer was not previously aware of.
    • Data mining and AI analysis (e.g. of github repos, SO, social media). Looking for niches to fill, for example.
    • More important than language features are the libraries and community supporting various domains - this explains why BioPerl is so popular and why Python is so popular in the AI/Machine Learning domain.
    • See the Sociology References and this response below for other ideas that might be tried to improve Programming Language Adoption.

    Other Articles in This Series

    Sociology References

    Other References

    Updated: corrections to history of B and C programming languages by Ritchie and Thompson. Expanded Other References section.

Organizational Culture (Part V): Behavior
1 direct reply — Read more / Contribute
by eyepopslikeamosquito
on Jul 16, 2021 at 06:34
Prefer Pure Perl Core Modules
9 direct replies — Read more / Contribute
by Leitz
on Jul 13, 2021 at 10:03

    "I prefer to use pure Perl core modules instead of depending on the CPAN."

    Saying that on IRC usually causes a flurry of negative comments. I agree that the CPAN is resource intense, and I've put stuff there. If you use lots of code from the CPAN, I'm not going to make fun of you. However, I would ask that you give me the same respect. Here's why I choose this path.

    1. Compatibility Pure Perl modules are portable, the target node doesn't need a set of compiler tools. While XS based modules might improve performance, not all nodes have compiler tools. Some nodes are precluded from having compilers based on resources or security mandates. Depending on a module that may not be installable creates a production risk.

    2. Idempotence Well engineered software can be installed and removed cleanly. When I worked for a telco, we would install, remove, and then reinstall software during QA. If it failed at any of those, it failed. Period. There is no "cpan" command to uninstall a module. The "cpanm" -U option for "uninstall" is marked "EXPERIMENTAL". The few times I've tried to use it to install modules that were installed via "cpan", it could not find the modules. Even when I told the command where the module was. If a module cannot be installed and removed cleanly then it does not belong on a production system.

    3. Upgradeability The "cpan" -u option (upgrade) comes with the warning: "Blindly doing this can really break things, so keep a backup." This conflicts with the concept of keeping software current to reduce security vulnerabilities and bugs. If a system's software cannot be cleanly upgraded it should not be in production.

    4. Security -- See Addendum below -- One of Perl's common object oriented modules is Moose. Installing Moose adds roughly 900 modules to the node. Who is security checking all those dependencies? Who wants to explain each and every one of those modules to a security auditor? In truth, how many of us could explain the risks and benefits of all nine hundred dependencies? And are we being paid to check someone else's code or are we paid to keep a production system running?

    5. Immiscibility Most Linux distributions require some version of Perl for operation. This sounds good for Perl, until you realize that the versions are often very out of date. If you want to use a semi-recent Perl you usually have to compile your own and install it somewhere. You also have to install any CPAN modules separately, which means your backups are now taking longer and you have more to sift through when trying to make space. And, of course, anyone who wants to use your code has to concoct the same environment.

    Solution

    My personal solution is to use pure Perl core modules or pure Perl CPAN modules that do not have a large dependency list. Large, in this sense, is "Am I willing to deal with these dependencies manually?" At some point in time I hope to be Perl-smart enough to help improve CPAN, but I'm not there yet.

    Addendum

    An earlier version of this page referenced YAML::Tiny in the security section. Investigating, based on hippo's comment (below) about "suggested_options" ('suggests_policy' in MyConfig.pm) removed YAML::Tiny as a culprit. YAML::Tiny has no non-core module dependencies.

    Chronicler: The Domici War (domiciwar.net)

    General Ne'er-do-well (github.com/LeamHall)

Cliques solution pertinent to my use case
3 direct replies — Read more / Contribute
by Sanjay
on Jul 07, 2021 at 11:57

    This refers to the hardness of finding cliques in a graph when the number of nodes becomes high. My use case is that I am trying to form cliques within a cluster where the sum of relationships is maximum.

    Background & definition (anthropomorphized for ease of understanding):

    Cluster: A number of persons where each person is related to at least one another person. The strength of the relationship is a number, i.e. Si-j is the strength of the relationship between person i & person j (Symmetric: Si-j = Sj-i). All combinations need not be present - if person x & person y have no relationship then Sx-y does not exist. No relationship with self - Si-i does not exist as well.

    Problem: Given a list of relationships Si-j: i from 1 to N, for each i some j from 1 to N, i!= j; (graph with the edges giving the strength of the relationship). Find those groups (cliques) where each member is connected to each other and the sum of the relationship is maximum, e.g. if the persons are as follows:

    Person-1 Person-2 Relationship-strength A B 92 A C 7 B C 2 C D 88

    Then the groups formed would be (A,B) & (C,D) for a sum of relationships of 180 (92 + 88).

    Any other combination would have a lesser sum, e.g. (A,B,C) & (D) would have a sum of relationships of 101 (92 + 7 + 2).

    How I did it: Convert the list of Si-j's into an integer linear programming problem and call a free solver lp_solve. Got good results - out of about 40,000 problem sets (clusters) only about 20 timed out. This on an ancient 3'rd gen i7 laptop with 4GB RAM. About another 10 were "very" large (more than 2,000 edges). Gives me confidence that use of a commercial solver and a larger timeout period may give even better results.

    Linear programming problem formulation left out here as it is more an algorithm issue rather than a Perl issue. We can discuss if interested. Off forum perhaps?

    Hope this is interesting. And, perchance, if it is useful for anyone then it would be the icing on the cake!

Organizational Culture (Part IV): Perl Culture
No replies — Read more | Post response
by eyepopslikeamosquito
on Jul 05, 2021 at 09:25
Perl OOP (POOP) - a use case for Util::H2O (hash to object)
1 direct reply — Read more / Contribute
by perlfan
on Jun 30, 2021 at 13:15
    Intuitively, I've been strongly drawn Util::H2O, and I finally have a solid example that has informed me explicitly what I have been feeling implicitly. My goal is to be laconic this time.

    Recently, I was writing a commandline tool in Perl, one of my favorite things to do. I needed Getopt::Long. I also was being nagged internally to use Util::H2O. Why? I simply wanted "real" accessors for the options hashref that I really like using with GetOptions. I ended up with something like this:

    use strict; use warnings; use Getopt::Long qw/GetOptions/; use Util::H2O qw/h2o/; # <~ here my $opts = {}; GetOptions( $opts, qw/ option1=s option2=i option3 option4! option5+ / ); h2o $opts; # <~ here #... exit; __END__ ## now we get the following, but only for the options actually passed +(updated) # $opts->option1 # $opts->option2 # $opts->option3 # $opts->option4 # $opts->option5
    Almost magically, with the addition of a single line (well, 2 if you count the use Util::H2O qw/h2o/; line,) I had accessors for $opts. I was pleased, to say the least. Note, I have not tried this with callbacks, but I also don't use them much. I suspect that works just fine. (update) Also note, in this case one gets accessors only for the options passed (or more precisely for only the keys that exist in $opts). I suspect this could be leveraged for more interesting and perlish ways of checking what options were actually passed.

    And now I can start to quantify where and when I'd like to use Util::H2O - not as an actual way to have a bunch of boiler plate at the top of my script to declare classes (which it does support), but perlishly (and iteratively) adding accessors to hash references - which my code tends to be absolutely full of. I've used it in some other places to clean up existing code and in new code, knowing full well I am going to be slinging hash references all over the place.

    In summary - check out Util::H2O. If you're like me, then you'll find little benefits to you that POOP offers can actually be satisfied in this natural way. It also occurs to me, this topic would be an ideal entry for the Perl Advent Calendar - so maybe I'll work out some more examples and make a submission later this year.

Benchmark: Storable, Sereal and JSON
3 direct replies — Read more / Contribute
by stevieb
on Jun 30, 2021 at 11:38

    After I get a distribution very stable, I go back and try to make it better, faster or more efficient. In IPC::Shareable, I've been benchmarking various serialization techniques to see which one works the fastest. I knew that Sereal was quite a bit faster than Storable, but there are some gotchas with it that I couldn't work around. This morning I tested with JSON, and to my surprise, it blew both out of the water!

    Results:

    Benchmark: timing 5000000 iterations of json, sereal, store... json: 17 wallclock secs (17.53 usr + 0.00 sys = 17.53 CPU) @ 28 +5225.33/s (n=5000000) sereal: 22 wallclock secs (21.78 usr + 0.00 sys = 21.78 CPU) @ 22 +9568.41/s (n=5000000) store: 49 wallclock secs (49.55 usr + 0.01 sys = 49.56 CPU) @ 10 +0887.81/s (n=5000000) Rate store sereal json store 102312/s -- -56% -64% sereal 233863/s 129% -- -18% json 286862/s 180% 23% --

    Benchmark code:

    use warnings; use strict; use Benchmark qw(:all) ; use JSON qw(-convert_blessed_universally); use Sereal qw(encode_sereal decode_sereal looks_like_sereal); use Storable qw(freeze thaw); if (@ARGV < 1){ print "\n Need test count argument...\n\n"; exit; } timethese($ARGV[0], { sereal => \&serial, store => \&storable, json => \&json, }, ); cmpthese($ARGV[0], { sereal => \&serial, store => \&storable, json => \&json, }, ); sub _data { my %h = ( a => 1, b => 2, c => [qw(1 2 3)], d => {z => 26, y => 25}, ); return \%h; } sub json { my $data = _data(); my $json = encode_json $data; my $perl = decode_json $json; } sub serial { my $data = _data(); my $enc = encode_sereal($data); my $dec = decode_sereal($enc); } sub storable { my $data = _data(); my $ice = freeze($data); my $water = thaw($ice); }

    I would never have expected that. Next up, a new serialization option for IPC::Shareable!

Perl tools for making code better
3 direct replies — Read more / Contribute
by Leitz
on Jun 29, 2021 at 10:02

    I have bodies of code to maintain and improve. Using tests is a given, and TDD is a wonderful thing. I am still learning Perl culture, and would appreciate guidance on tools and processes.

    My parameters are hopefully simple. I prefer clear and verbose code so I can understand it the next time I read it. When producing tools I prefer to stay as close to Core/StdLib Perl as possible. When working *with* tools, more options are open. For example, using a Perl::Critic plug-in while developing code is fine, shipping those add-ons is something I would avoid.

    My current plan is a cycle of:

    1. Write Tests
    2. Kwalitee
    3. Perl::Tidy
    4. Perl::Critic

    Repeating as necessary, or until I have to ship the code. Are there other tools to add to the mix? Better processes?

    Chronicler: The Domici War (domiciwar.net)

    General Ne'er-do-well (github.com/LeamHall)

Organizational Culture (Part III): Spaceflight and Aviation
2 direct replies — Read more / Contribute
by eyepopslikeamosquito
on Jun 28, 2021 at 02:37

    Let's continue this seemingly never-ending series by examining culture change in two domains near to my heart:

    Space Shuttle Challenger (1986)

    As soon as the button was pressed to mute NASA from our meeting, the managers said "we have to make a management decision", said Boisjoly.

    The general manager of Thiokol turned to his three senior managers and asked what they wanted to do. Two agreed to go to a launch decision, one refused. So he (the general manager) turns to him and said "take off your engineering hat and put on your management hat" -- and that’s exactly what happened, said Boisjoly. He changed his hat and changed his vote, just thirty minutes after he was the one to give the recommendation not to launch. I didn’t agree with one single statement made on the recommendations given by the managers.

    The teleconference resumed and NASA heard that Thiokol had changed their mind and gave a recommendation to launch. NASA did not ask why.

    I went home, opened the door and didn’t say a word to my wife, added Boisjoly. She asked me what was wrong and I told her "oh nothing hunny, it was a great day, we just had a meeting to go launch tomorrow and kill the astronauts, but outside of that it was a great day".

    -- from Remembering the mistakes of Challenger

    For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.

    -- Richard Feynman (in Rogers Commission Report on the Challenger disaster)

    Space Shuttle Columbia (2003)

    Although changes were made by NASA after the Challenger accident, many commentators have argued that the changes in its management structure and organizational culture were neither deep nor long-lasting.

    After the Space Shuttle Columbia disaster in 2003, attention once again focused on the attitude of NASA management towards safety issues. The Columbia Accident Investigation Board (CAIB) concluded that NASA had failed to learn many of the lessons of Challenger. In particular, the agency had not set up a truly independent office for safety oversight; the CAIB decided that in this area, "NASA's response to the Rogers Commission did not meet the Commission's intent". The CAIB believed that "the causes of the institutional failure responsible for Challenger have not been fixed", saying that the same "flawed decision making process" that had resulted in the Challenger accident was responsible for Columbia's destruction seventeen years later.

    -- from Space Shuttle Challenger disaster (wikipedia)

    Incredibly, NASA, with so many talented people, had somehow (twice!) failed to remedy a glaring Conflict of interest organizational structure/culture problem. It seems they failed to find a way to ensure that the technical team making launch safety decisions were allowed to do so without pressure or coercion from outside interests.

    (Which BTW reminds me of the poor old Release Manager in commercial software companies being relentlessly pressured by senior management to ship the next release ... less common perhaps in the open source world, though release management remains stressful).

    Perhaps the root cause of NASA's Space Shuttle woes was an unhealthy mix of commercial/financial and engineering/scientific culture. Certainly, NASA have achieved many breathtakingly brilliant successes in the purely engineering/scientific domain, such as the venerable Hubble Space Telescope and the recent stunning New Horizons mission to Pluto and beyond.

    It will be interesting to see in the coming decade if Elon Musk or Jeff Bezos or Sir Richard Branson can create a more effective culture for commercial space flight.

    National Cultures

    Social psychologist and former IBM researcher Geert Hofstede identified five dimensions of culture in his studies of national work related values:

    • Power Distance. How accepting are people of unequal power distribution?
    • Individualism. Is individual achievement or collective achievement emphasized?
    • Masculinity. How much does the culture accept gender roles?
    • Uncertainty Avoidance. How anxious are folks about uncertainty and risk?
    • Long-term Orientation. Do people take a long-term view of their work?
    Hofstede found that these five dimensions varied considerably between countries. The United States, for example, rated highest on Individualism, while Russia and India rated highest on Power Distance. Japan scored the highest on the Masculinity index. Russia and Japan have high Uncertainty Avoidance, while Denmark has a low score. Japan, South Korea and India have a much stronger Long-term Orientation than Canada, the UK and the USA.

    How this affects software teams was discussed previously at Nobody Expects the Agile Imposition (Part IV): Teamwork.

    Korean Airlines

    The captain made the decision to land despite the junior officer's disagreements, eventually bringing the plane down short of the runway, highlighting how a pilot can contribute to a disaster. In high power distance cultures, it is uncommon for subordinates to question their superiors. High power distance can be seen as the willingness to be in an unequal position, making it a challenge for an officer lower in the hierarchy to question the decisions of the one in power.

    -- Impact of culture on aviation safety (wikipedia)

    Korean Air had many fatal accidents between 1970 and 1999, during which time it wrote off 16 aircraft in serious incidents and accidents with the loss of 700 lives. The last fatal accident, Korean Air Cargo Flight 8509 in December 1999 led to a review of how Korean cultural attitudes had contributed to its poor crash history. Since then safety has improved.

    -- Korean Air incidents and accidents (wikipedia)

    In the cases of Korean Air flight 801 and Air Florida flight 90 one clear statement could have saved more than 300 lives. In each case, a series of leadership errors led to the deaths of hundreds of passengers. Cultural and contextual factors contributed to the leaders unknowingly creating unsafe conditions. A lack of cultural awareness and understanding of the environment was a culprit in both case studies, along with dozens of other airplane crashes. These two case studies serve as warnings for all leaders on the importance of including cultural dimensions in their leader(ship) development programs as well as adapting programs to the specific environmental factors. By taking a lesson from the aviation industry, other organizations can prevent similar mistakes, create better leaders and leader(ship) development programs, and possibly save lives.

    As a result of this and other accidents, Korean Air’s new CEO set about an ambitious organizational change. He made English the official language of the airline, redefined roles in the cockpit, and provided extensive communications training, among other initiatives to change the culture (Wee, 2013). Other airlines also made changes in response to the reoccurring communication challenges. One notable change made by many airlines was to adopt a program known as cockpit resource management (CRM). CRM requires the entire crew to work together and communicate (D’Amato, 2013). Training programs provide specific skills and procedures to maximize communication and effectiveness. CRM has proven to be very effective in reducing airplane crashes due to communication, regardless of cultural backgrounds.

    -- Copilot leadership (medium.com)

    It is gratifying (at long last!) to find a convincing example of where changing Organizational Culture actually worked!

    Other Articles in This Series

    References

Organizational Culture (Part II): Meta Process
3 direct replies — Read more / Contribute
by eyepopslikeamosquito
on Jun 17, 2021 at 06:03

    Googling Organizational Culture revealed many folks offering (often pricey) Organizational Culture workshops based on theories concocted by a pair of enterprising boffins, quietly contemplating at the University of Michigan, located in the picturesque village of Ann Arbor.

    Though the definitive reference on their work is available for purchase from amazon, you can also get a feel for their process by reading this early paper:

    A Process for Changing Organizational Culture by Kim Cameron, University of Michigan 2008
    Handbook of Organizational Development, 2008: 429-445 (cited by 345)

    Abstract: This chapter outlines a process for diagnosing and changing organizational culture. It uses the Competing Values Framework to describe a validated approach to helping an organization change from a current culture to a desired culture.

    ... and from the (mostly youtube) links in the References section below. As you might expect, I was too cheap to pay for advice on this topic, so instead watched some youtube videos and read Kim's original paper. Though not necessarily the best organizational change process (alternative citations welcome), at least this is a concrete thing that can be discussed and analysed, and thus serve as a starting point for discussing specific ways to improve Organizational Culture (and Perl organizational culture too).

    For those seeking a Perl Monks connection to this academic paper, notice that Ann Arbor Michigan is a mere two hour scenic drive from Hope College in Holland Michigan, the sacred birthplace of Perl Monks (if you drive via Portage Michigan, you can further pick up some COVID-19 vaccines from Pfizer’s huge manufacturing facility on the way).

    Definition of Organizational Culture

    Although many definitions of culture have been proposed, the two main disciplines are:

    • Sociological (organizations have cultures). Assumes you can identify differences among organizational cultures, can change cultures, and can empirically measure cultures.
    • Anthropological (organizations are cultures). Assumes that nothing exists in organizations except culture, and one encounters culture anytime one rubs up against any organizational phenomena.

    In her 2008 paper, Cameron gave a popular and practical definition of culture as:

    the taken-for-granted values, underlying assumptions, expectations, and definitions present which characterize organizations and their members
    which:
    • serves as the social glue binding an organization together; and
    • represents how things are around here, affects the way members think, feel, and behave.

    and further perceptively noticed that:

    With very few exceptions, virtually every leading firm has developed a distinctive culture that is clearly identifiable by its key stakeholders

    This distinctive culture is sometimes created by the initial founder: Walt Disney, Bill Gates, and Larry Wall, for example. All three of these legends developed something special, something more vital than corporate strategy, market presence, or technical advantages: the power that arises from a unique and spirited culture.

    Curiously, most people are unaware of culture until suddenly confronted with a different one: travelling to Vietnam, for example, finding yourself immersed in different noises and smells and unable to understand a word of the local lingo ... or asking a question on SO after years of posting at Perl Monks. :)

    The culture of most organizations is invisible, most members have a hard time describing it, let alone consciously changing it -- that is why you need tools, such as The Competing Values Framework and the Organizational Culture Assessment Instrument (OCAI), developed by scholars Cameron and Quinn.

    Notice that Organizational Climate is distinct from Organizational Culture: Climate is temporary, Culture enduring.

    The Competing Values Framework

    This framework, used to assess the dominant characteristics of organizations, differentiates on two vertical dimensions:

    1. Flexibility, Discretion, Dynamism (effective if: changing, adaptable, organic, e.g. Google, Nike)
    2. Stability, Order, Control (effective if: stable, predictable, mechanistic, e.g. Universities, Government agencies, Boeing)

    and two horizontal dimensions:

    1. Internal orientation, Integration, Unity (harmonious internal characteristics, e.g. IBM, the IBM way)
    2. External orientation, Differentiation, Rivalry (effective if: competing with others outside their boundary, e.g. Toyota, Honda)

    Together these two dimensions form four quadrants:

    1. Clan culture (Collaborate)
    2. Adhocracy culture (Create)
    3. Hierarchy culture (Control)
    4. Market culture (Compete)

    Hierarchy Culture:

    • Formalised and structured workplace. Procedures and controls govern what people do. Leaders: coordinators, organizers, monitors.
    • Maintaining a smoothly running organization is important.
    • Long term: stability, predictability, efficiency, formal rules and policies hold the organization together.
    • Success is defined in clear lines: authority, control, accountability, e.g. McDonalds, Govt agency on airport controls (strict guidelines for every small detail).

    Market Culture:

    • Competing environment, results-oriented workplace, external environment is hostile, consumers are selective, want value.
    • Leaders are hard-driving producers; competitors are tough and demanding.
    • Productivity, results and profits; glue is emphasis on winning.
    • Success is market share and penetration, e.g. Ikea, Walmart.

    Clan Culture:

    • Friendly workplace, people share a lot of themselves, leaders are mentors, facilitators, team builders, parent figures, held together by loyalty and tradition.
    • Commitment is high, long-term benefit of individual development, high cohesion and morale important.
    • Success is defined as internal climate and concern for people, premium on teamwork, participation and consensus, e.g. small family-owned companies, doctors without borders, wikipedia.

    Adhocracy Culture:

    • Dynamic, entrepreneurial culture, people take risks, leaders: visionary, innovative, risk-oriented.
    • Glue: Commitment to experimentation and innovation, to be at the leading edge, readiness for change is essential.
    • Long term emphasis is rapid growth.
    • Success is creating unique & original products and services, e.g. start-ups.

    Relationship between the four quadrants:

    • Each side represents opposites
    • Flexibility vs Stability
    • Internal vs External
    • Competing on the Diagonal: Clan (internal focus) v Market (external focus); Adhocracy (external organic) v Hierarchy (internal control)
    The competing on opposite sides of each quadrant give rise to the name of the model.

    Why Change Organizational Culture?

    The Competing Values Framework Introduction youtube (around the 12:15 mark) gives a fascinating case study of how Organizational Cultures change over time. Steve Jobs, a charismatic entrepreneurial leader, was a great cultural fit for the (Startup culture) Apple of the 1970s ... only to get fired in 1985 for his chaotic management style, when Apple needed more controls and standard procedures ... then finally re-hired in 1997 to heroically resurrect the company after it started having a hard time inventing new products! That is, Apple culture evolved from Adhocracy (Apple II era) to Adhocracy/Clan (Macintosh era) to Hierarchy (John Scully era) to Hierarchy/Market then balanced Hierarchy/Market/Adhocracy/Clan (on Jobs return).

    How does a language win? By being compelling enough to be used for new things. It's not solely a technical concern; it's a concern of the language community and ecosystem.

    -- Why Perl Didn't Win (essay from outspeaking.com)

    As indicated in the essay cited above, a good reason for Perl to change its culture may be to make it more attractive for new projects (compared to competing languages). Update: see this response for a present-day example of a domain in which Perl is less compelling than Python.

    Of course, if Perl was a commercial enterprise, one business strategy to cope with losing market share may be to seek a merger with Python ... thus allowing our new customers to write some truly astonishing code:

    # copy stdin to stdout, except for lines starting with # while left_angle_right_angle: if dollar_underscore[0] =eq= "#": continue_next; } print dollar_underscore; }

    Sorry, couldn't resist. :)

    Note that if you choose not to attempt to explicitly change your organization's culture, it will change anyway. Culture is evolving all the time.

    The Seven Steps To Culture Change

    1. Clarifying meaning.
    2. Identifying stories.
    3. Determining strategic initiatives.
    4. Identifying small wins.
    5. Crafting metrics, measures, and milestones.
    6. Communication and symbols.
    7. Leadership development.

    The Organizational Culture Assessment Instrument (OCAI)

    It might be fun to create a Perl Monks poll to see how people rate P5P culture or Perl Monks culture.

    1. DOMINANT CHARACTERISTICS
    The organization is:
    A. a very special place. It is like an extended family. People seem to share a lot of themselves.
    B. a very dynamic and entrepreneurial place. People are willing to stick their necks out and take risks.
    C. very production oriented. A major concern is with getting the job done. People are very competitive and achievement oriented.
    D. a very formalized and structured place. Bureaucratic procedures generally govern what people do.

    2. ORGANIZATIONAL LEADERS
    The leaders of the organization are generally considered to be:
    A. mentors, facilitators, or parent figures.
    B. entrepreneurs, innovators, or risk takers.
    C. hard-drivers, producers, or competitors.
    D. coordinators, organizers, or efficiency experts.

    3. MANAGEMENT OF EMPLOYEES
    The management style in the organization is characterized by:
    A. teamwork, consensus and participation.
    B. individual risk-taking, innovation, flexibility, and uniqueness.
    C. hard-driving competitiveness, goal directedness, and achievement.
    D. careful monitoring of performance, longevity in position, and predictability.

    4. ORGANIZATION GLUE
    The glue that holds the organization together is:
    A. loyalty and mutual trust. Commitment to this organization runs high.
    B. orientation toward innovation and development. There is an emphasis on being on the cutting edge.
    C. the emphasis on production and goal accomplishment. Marketplace aggressiveness is a common theme.
    D. formal rules and policies. Maintaining a smooth running organization is important.

    5. STRATEGIC EMPHASES
    The organization emphasizes:
    A. human development. High trust, openness and participation persist.
    B. acquiring new resources and meeting new challenges. Trying new things and prospecting for new opportunities are valued.
    C. competitive actions and achievement. Measurement targets and objectives are dominant.
    D. permanence and stability. Efficient, smooth operations are important.

    6. CRITERIA OF SUCCESS
    The organization defines success on the basis of:
    A. development of human resources, teamwork, and concern for people.
    B. having the most unique or the newest products. It is a product leader and innovator.
    C. market penetration and market share. Competitive market leadership is key.
    D. efficiency. Dependable delivery, smooth scheduling, and low cost production are critical.

    Other Articles in This Series

    References

    See Also

Organizational Culture (Part I): Introduction
4 direct replies — Read more / Contribute
by eyepopslikeamosquito
on Jun 11, 2021 at 07:58

    The recent turmoil in Perl's organizational culture, discussed recently here at Perl Monks in:

    left me feeling sad. Sadder after reading of the Australian Defence Force's long-drawn-out battles with cultural change: because it made me realise that organizational cultural problems are dauntingly difficult - and often prohibitively expensive - to put right.

    For therapy, I've decided to follow up my nine-part series on Agile processes in organizations with a new (ten-part :-) series of articles exploring the perplexing multi-disciplinary topic of Organizational Culture. There's certainly no shortage of material, with a mind-boggling number and variety of disciplines in play, such as:

    Please feel free to mention other disciplines I've overlooked or that you'd like to see discussed.

    A possible breakdown by topic is:

    • Broad Overview of Organizational Culture
    • Meta Process: that is, a process to build a process to effectively change Organizational Culture
    • Culture variation in different countries
    • Software Industry Culture
    • Open Source Software Culture
    • Perl Culture
    • Perl Monks Culture! :)
    • Space/Aircraft Industry Culture. As a space nut, I doubt I'll be able to restrain myself from scrutinizing the impact of organizational culture on the Space Shuttle Challenger and Columbia disasters.
    Again, feel free to suggest other topics you'd like to see discussed.

    Please note that I'm just an interested layman on all these topics, so please correct me when I veer off course. I also welcome your feedback and opinion, along with insights, anecdotes and any useful citations you may know of.

    To kick off this series, given that I've just finished reading Sapiens, I thought it'd be fun to speculate on how our early evolutionary history helps explain some of the peculiar and counter-productive behaviour we witness today.

    PreHistory

    Around 2.5 million years ago, an unremarkable new species appeared in Africa. Now, I doubt these early humans, enduring their daily battle for survival along with many other species, had any inkling back then that they would one day walk on the moon, split the atom, map the genome, calculate the age of the universe ... and compose Perl programs, obfus and poetry.

    How on earth did this unremarkable and physically weak new species win this brutal evolutionary war? Sapiens revealed the (surprising to me) secret:

    A green monkey can yell to its comrades, "Careful! A lion!". But a modern human can tell her friends that this morning, near the bend in the river, she saw a lion tracking a herd of bison. She can then describe the exact location, including the different paths leading to the area. With this information, the members of her band can put their heads together and discuss whether they should approach the river, chase away the lion, and hunt the bison.

    Homo sapiens is primarily a social animal. Social cooperation is our key for survival and reproduction. It is not enough for individual men and women to know the whereabouts of lion and bison. It's much more important for them to know who in their band hates whom, who is sleeping with whom, who is honest, and who is a cheat. ... The most important information that needed to be conveyed was about humans, not lions and bison. Our language evolved as a way of gossiping.

    Do you think that history professors chat about the reasons for the First World War when they meet for lunch, or that nuclear physicists spend their coffee breaks at scientific conferences talking about quarks? Sometimes. But more often, they gossip about the professor who caught her husband cheating, or the quarrel between the head of the department and the dean, or the rumours that a colleague used his research funds to buy a Lexus.

    Only Homo sapiens can speak about things that don't really exist. You could never convince a monkey to give you a banana by promising him limitless bananas after death in monkey heaven. But why is it so important? Fiction has enabled us not merely to imagine things, but to do so collectively. We can weave common myths. Such myths give Sapiens the unprecedented ability to cooperate flexibly in large numbers.

    Ants and bees can also work together in huge numbers, but they do so in a very rigid manner and only with close relatives. Sapiens can cooperate in extremely flexible ways with countless numbers of strangers. That's why Sapiens rule the world, whereas ants eat our leftovers and chimps are locked up in zoos. (Update: so far researchers have failed to locate lawyer bees; bees don't need lawyers because there is no danger that they might forget or violate the hive constitution).

    Myths and fictions accustomed people, nearly from the moment of birth, to think in certain ways, to behave in accordance with certain standards, to want certain things, and to observe certain rules. They thereby created artificial instincts that enabled millions of strangers to cooperate effectively. This network of artificial instincts is called culture.

    The theory of evolution tells us that instincts, drives and emotions evolve in the sole interest of survival and reproduction ... yet when some of these evolutionary pressures abruptly disappear (due to sudden changes in human culture), these hard-won instincts do not instantly disappear with them ... which explains why today young men drive recklessly and fight in pubs. They are simply following ancient genetic impulses that, though counter productive today, made perfect sense 70,000 years ago, where a young hunter who risked his life chasing a mammoth outshone all his competitors to win the affections of the local beauty. Sadly, those same successful macho genes today give us a mouthful on Perl mailing lists, IRC and Twitter.

    With their brain growing (to keep track of the thousands of connections, deceits and subterfuge required for superior gossiping) and their hips narrowing (to facilitate walking upright), evolution favoured early births ... leading to helpless babies ... requiring a tribe to raise them ... favouring those with strong social abilities. Being born immature and with a big brain further allowed humans to be taught more flexibly than any other species -- which explains why we witness today the appalling spectacle of impressionable youngsters being effortlessly led to love Python and hate Perl.

    Other Articles in This Series

    PM References

    References Added Later

    Updated: Make Homo sapiens stand out from surrounding text, and with the species name (sapiens) in all lower case, a convention used by biologists, such as erix. 13-June: added bee lawyer joke to quoted Sapiens passage.


Add your Meditation
Title:
Meditation:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (2)
As of 2021-09-27 10:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?