Hi, folks.
Recently I've found (here in PerlMonks) a link to one of Donald Knuth's great talks: Computer Programming as an Art. Knowing it was Knuth's sutff, I knew it would be worth my time.
One of the (often overlooked) benefits of computer programming is that it can be fun. But Knuth goes beyond that stating that it should be fun and that a program should be beautiful. Fun and beauty add a lot of value to the program and to programming.
Now, at the same time, observing the IT industry as whole, we can see how much the fun and beauty of programming are being overlooked. More and more "magic tools" are making way to sysadmin jobs, and they're focused on people who don't want to program. IT managers often see them as a means "not to depend too much on programmers". It's laughable, but I've seen it a lot.
One example of such tools are the configuration management tools, such as Ansible, Puppet, Chef and similar tools, which take a more declarative approach to systems management. I'm not saying these tools aren't useful (or even that they are necessarily bad), but I feel like they've taken a lot of the fun away from systems management.
This is not to mention GUI-only tools, which are mostly closed-source. Fortunately I've been far from them. But currently at my job, I help maintain some Ansible routines, and sometimes it's a daunting task. I can't remember how many times I had to grep a repository to find out when some variable was being set. If a declarative approach should be friendly, then I must say it has failed in my case, because the repository I work with has grown into a messy beast (I'm sure my co-workers agree).
Yet, I remember once I've put a perl one-line command in a playbook and was critized precisely for this action. I was told it would bring complexity and that other people would have difficulty maintaining perl code. Can you believe it? A one-line! I'd say it's a joke if someone else told me.
Of course, I wouldn't give up on Perl because of that. But it makes me wonder why must we always prove something that exists for decades is worthy a try, while something created a few years ago is promptly accepted. Of course, it's just my point of view, which not necessarily happens everywhere. Other points of view would be much appreciated. |
Re: "Magic tools" that take the fun away (Build/DevOps/Rex References)
by eyepopslikeamosquito (Archbishop) on Apr 10, 2021 at 09:29 UTC
|
I've had more time to reflect on your thought-provoking meditation.
Let me start with some rambling anecdotes from twenty years ago.
Around the year 2000, I was a keen C++ developer starting in a new company. Young (well, younger) and eager to make my mark.
Our C++ software had to build and run successfully across many different Unix and Windows flavours.
This was back when Cloud Computing was not talked about
and, like everyone else back then, we owned our own set of dedicated on-premise build machines.
We owned our own on-premise test lab too. :)
A recent hire to our (newly formed) build team had just received a gold star from management for "successfully"
implementing a brand new build system. On time and on budget. Only took a few months to develop.
On Windows, he had crafted a truly horrifying .BAT script. Thousands of lines. Opening it in Notepad made my blood run cold. Oh the horror! :)
This .BAT monstrosity drove our Unix builds from Windows via a telnet session (we even bought a commercial telnet product to support this approach!)
that ran many Unix commands to build, test, and package our software. Back then, our Unix builds took many hours to complete.
I would fire off his .BAT masterpiece from my PC every night ... only to arrive keen and early in the office the next morning (no working from home back then)
to discover that my build had failed, due to a broken telnet connection. Every single time.
Had to start a fresh build and manually nurse it through the day.
I couldn't get my work done!
Being young and reckless - and desperate to get my work done -
I implemented a brand new build system over the weekend (written in Perl) that used remote shell, not telnet.
No need to hold a telnet connection open for hours on our unreliable network.
Instead it created a source tarball locally, ftp'ed it to each Unix box, then fired off a Perl script via remote shell
to unpack the tarball, perform the build, run the tests and package our product.
No need for a long-lived connection, simply check your build status occasionally via a rsh command to tail the build log.
As you might expect, all developers switched to my system ... though I wasn't awarded a gold star from management.
Though that was fun for me, I doubt it was fun for our gold-star winning .BAT maestro build engineer.
Hardly an example of great teamwork.
BTW, we've improved our communication since then;
communicate early, often, and with trust.
Lots of eye-gouging to be sure, but it usually results in everyone pulling
in the same direction, as described at Conflict in Teams.
Around this time, I remember developing a strong dislike of Unix make, forming the opinion
that maintaining millions of lines of code built with the make build tool inevitably leads to misery.
You need a real programming language (not a toy DSL) to deal effectively with huge complex builds.
Sadly, I never did much about this, just suffered in silence, year after year.
My only consolation is, after googling today, I see I'm not alone in my suffering.
You see, way back in 1999,
a mysterious unsung Perl hero called Bob Sidebotham lovingly crafted an award-winning
Perl-based Cons build system.
From this tpj article:
Understandably, many Perl programmers don't like having to program in low-level languages like C and C++.
Even more irritating is writing the Makefiles necessary to compile and install those programs with the Unix make utility.
Worse still is being saddled with maintaining someone else's Makefiles.
Perhaps the most egregious aspect of make is the make language itself. (In fairness, make has been around for a long time, predating even Perl.
The standard for scripting in the Unix community at the time of make's emergence was the Bourne shell, and make represented a considerable
advance over the then-common practice of using shell scripts to build software.)
As a scripting vehicle, Makefiles are really just glorified collections of macros with no flow of control primitives,
no complex substitution rules, no variable scoping, and poor control over import/export of environment variables and command arguments.
If you want proof that make is inadequate, just look at the plethora of utilities that people use to avoid writing their own Makefiles,
like MakeMaker and imake. I could provide a long list of the shortcomings of the make language, but I'd be surprised if any Perl programmer
needs much persuasion.
Thanks Bob! I can definitely relate to that!
Sadly though, Cons seems to be in hibernation today.
Rubbing salt into the wounds, it appears to have been superseded by a Python-based version called
SCons:
SCons software history started with the Cons software construction utility created by Bob Sidebotham in 1999.
Cons was written in the Perl language.
It served as a base for the ScCons build tool, a design which won the Software Carpentry project SC Build competition in August 2000.
ScCons was the foundation for SCons.
The long list of "Make-incompatible" build tools at handwiki
suggests a lot of other people share my opinion on this.
Though I now regret not doing more about this twenty years ago (rather than suffering in silence)
it seems that reaching consensus on which "make-incompatible" build tool to use may prove problematic
because of resistance to "having" to learn Perl (or Python or Ruby) just to work on your build system.
Maybe that is why Cons lies dormant today.
Of course, the software landscape has changed dramatically since then.
Creating complex multi-machine test environments in the Cloud (using AWS in my case)
has become astonishingly easy compared to twenty years ago in our good old on-premise test lab ...
and BTW, astonishingly easy to run up huge AWS bills if you're careless and don't have proper cost controls in place. :)
Though I agree with you that using these new DevOps tools and Banal Configuration Languages is not as enjoyable as
crafting beautiful code in a real programming language, avoiding them nowadays seems problematic.
I did notice one glimmer of hope for Perl in the DevOps space when I googled today
and stumbled upon Rex,
an open source remote execution, configuration management and software deployment tool that combines
Perl and Secure Shell (SSH) for a portable, centralistic approach to its problem domain.
Super-searching reveals that at least one Perl monk, namely nysus, is actively using it today
(see Rex References below).
Update: GNU parallel is another SysAdmin tool written in Perl (see GNU Parallel References below).
Cons References
Rex References
GNU Parallel References
Other References
General References
DevOps References
- DevOps (wikipedia)
- Site reliability engineering (SRE) (wikipedia) (Site reliability engineering is closely related to DevOps, a set of practices that combine software development and IT operations, and SRE has also been described as a specific implementation of DevOps)
- Ansible - Tool often used in DevOps (wikipedia)
- Jenkins - Tool often used in DevOps, uses a DSL based on Groovy (wikipedia)
- Apache Groovy - Language for the Java Platform (wikipedia)
- Spring Framework - Application Framework for the Java platform (wikipedia)
- SolarWinds (wikipedia)
Build and Test Automation References
See Also
Updated: Dec-2021: Added GNU parallel references (mentioned here) - a Unix command line tool, written in Perl, that allows the user to execute shell scripts or commands in parallel. See also pexec and xargs and GNU Queue originally by Werner G Krebs (not heavily used anymore?).
Sep 2023: Added "Build and Test Automation References" section.
| [reply] |
|
|
Interesting story. Enlightening, actually. Reminds me of how much I have yet to learn, as I learned to program in 2009 and got involved with Perl ten years later, so I missed many cases like this one you tell.
Thanks for mentioning Rex, BTW. I didn't know it. It seems to have a saner approach to configuration management, though I could only guess until actually using it for real cases. Not that I expect to use it in my current job. As you mention in Conflict in Teams, we need to disagree and commit, and I missed the "disagree part", as Ansible was already in place when I got there, so I got the "commit part" only (actually, it's funny to say "commit", as it's git-versioned, so "commit" takes more than one sense here).
Again, Ansible is not bad, just not my first choice for the kind of automation my team performs. It works very well for "standard tasks", and I'm sure that's why it was chosen in the first place. But as business rules get complex, it begins to show its shortcomings.
So, I'll put Rex "in my pockets", just in case I start something new.
| [reply] |
Re: "Magic tools" that take the fun away
by eyepopslikeamosquito (Archbishop) on Apr 09, 2021 at 00:03 UTC
|
| [reply] |
|
|
| [reply] |
Re: "Magic tools" that take the fun away
by Discipulus (Canon) on Apr 09, 2021 at 10:34 UTC
|
Hello hrcerq and welcome to the monastery and to the wonderful world of perl!
> Recently I've found ..
Glad of someone profit of my messy bibliotheca!
You mention: art, beauty and fun. These are all terms related to the the human being as central point of view. I have the same idea and I consider programming, in the way we like it, as an humanistic art, not only an art. When the human is in the center of our perspective there is room for creativity and happiness.
Starting from Knuth's article you mentioned, I've written a mini serie (unfinished): Antiquitates - liber I - In memoriam Robert M. Pirsig and Antiquitates - liber II - De trivio atque quadrivio are available atm.
You can profit also the reading of a recent thread: Why Perl in 2020.
> IT managers often see them ("magic tools") as a means "not to depend too much on programmers". It's laughable...
I found this not laughable at all :)
While we can still have an humanistic approach to IT and programming, the business and therfore managers have a moneycentric approach and to not depend too much (and one day to not depend at all) on programmers is a key point in their strategy.
"Magic Tools" must be run by an idiot formed to accomplish boring tasks. If this worker is unpleasant or costs too much, they can replace them with another one with easy. In another corner of the world where salaries are risible? Even better!
In this way I also see the declining of Perl: is too humanistic. Not a suitable tool for robot-workers.
Higher the complexity of tools, the abstraction, harder to see the whole picture. If someone still has a vague idea of the whole picture they will be seen as wizards in some years.
My hope is that the above schema will collapse on itself oneday, all togheter. Then a good old sysadmin, perl outfitted, will be valuable as gold.
L*
There are no rules, there are no thumbs..
Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
| [reply]
[d/l] |
|
|
Thank you, Discipulus. I don't think your bibliotheca to be messy, but even if it is, it's a great resource, and as my spare time allows me, I'll dive deeper there.
Also, these other meditations you point out are very much related to the subject. Really appreciated.
> I found this not laughable at all :)
My point here is that even if you use a "no-code" solution, you depend on the programmers who maintain the magic tool behind that. So, in the of the day, you'd be just transferring your dependance to a programmer that is more distant and less accessible.
| [reply] |
|
|
"Magic tools" are great for streamlining repetitive, error-prone work. And there's no doubt that "configuration management" across a big stable of servers is near the top of that list. But, you'd better be sure that you know at all times what is really going on. It is very, very easy to make an "innocent" mistake with these power-tools that instantly has far-flung implications. For instance, it's absolutely necessary that you put the configuration files under git version-control, and that you very-carefully track them in your change management system.
| [reply] |
Re: "Magic tools" that take the fun away
by jmlynesjr (Deacon) on Apr 09, 2021 at 03:25 UTC
|
Vendors sell tools to IT managers, not programmers. Lost an ethernet vs. Token Ring battle in the 80s...No One Ever Got Fired by Choosing IBM...
James
There's never enough time to do it right, but always enough time to do it over...
| [reply] |
| A reply falls below the community's threshold of quality. You may see it by logging in. |
|
|
