Re^3: At what point do you rewrite that old shell script in Perl?

Here are a few rules of thumb I use. Your thumbs may be of a different size than mine (which would be quite probable if I was speaking literally).

I'd say any shell script that is subject to being updated as the result of a feature request is a strong candidate for rewriting in a more powerful language. Most shell scripts are part of an ad hoc implementation, a prototype of something, are fairly simple, are needed because more powerful languages aren't available (like at boot time for init scripts), or are used specifically because they can alter the environment of the login shell. If you're having to maintain changes to a shell script that's not bound to stay a shell script by circumstance then it's a good time to rewrite it.

Any time updating a program and rewriting (or refactoring) the program to be more maintainable are in the same ballpark in terms of time, opt for making it more maintainable later. This goes double if you're rewriting in a more suitable language precisely because the old language doesn't support the changes well. This goes for all languages and not just shell.

Any time a shell script is calling something written in Perl, take the time to figure out why you're calling that functionality from the command line instead of as a sub. If you have a good answer, consider not rewriting the shell script. If you don't, it's probably a good time to plan a rewrite.

If the startup times of your independent programs called by the shell script is slowing the script down too much (for some value of "too much") and you could easily duplicate most of the functionality of those utility programs in a few minutes of coding Perl, then go ahead and do most of the work in Perl and only shell out to the parts that are harder to duplicate.

If you foresee the need to run in disparate environments (like Windows and Linux) where you can choose fighting with some implementation of your shell for a nonstandard platform or just installing Perl (or using an existing Perl installation on the new target if it's there), then rewrite in Perl.

If you're having to write comments about program flow because your shell script is getting difficult to follow, then you need it to be in another, more powerful language.

If you're planning to handle more data than the command-line tools you're calling from the shell script handle well, Perl is an excellent solution.

Any situation in which your environment variables are suspect and the command-line tools you're calling change their behavior based on environment variables, you might want Perl just for the sake of things like easily checking/altering the whole $ENV hash and Taint.pm. Shell scripts that do CGI have always, for example, made me nervous.

Anything repetitive that is difficult to modularize and reuse is a sign you need a more flexible language. This is especially the case when it's because your language lacks a way to easily handle the type of conditional execution you need for the parts that aren't exactly the same.

This list isn't exhaustive, but it just might be exhausting. ;-) In any case, the best rule of thumb is probably that if you have to stop to consider rewriting a shell script in Perl, that you're going to be better off doing the rewrite.

Comment on Re^3: At what point do you rewrite that old shell script in Perl?

Replies are listed 'Best First'.
Re^4: At what point do you rewrite that old shell script in Perl? by rudder (Scribe) on Apr 13, 2008 at 16:00 UTC
Thanks for the excellent reply. I'll use that advice. One thing though; could you please clarify your sentence: "Any time updating a program and rewriting (or refactoring) the program to be more maintainable are in the same ballpark in terms of time, opt for making it more maintainable later."? Edit: That is, is a rewrite == making a shell script more maintainable?, and do you mean, "making something more maintainable later", as in "put off making it more maintainable -- do it later", or do you mean "making something more maintainable right now for the additional changes you'll probably have to make down the road"?	[reply]
Re^5: At what point do you rewrite that old shell script in Perl? by mr_mischief (Monsignor) on Apr 14, 2008 at 16:15 UTC
As for #1, anything that makes a program more maintainable later works. In most cases, rewriting from shell to Perl will help. There are lots of programs in Perl that could stand to be refactored or rewritten in Perl using clearer control flow and more idiomatic usage of the language, too. There are even programs, as flexible as Perl is, that would do well to go from Perl to other languages. There are probably equally as many (or more) IMO that would benefit from going from other languages into Perl. For #2, I meant that you should compare the time it would take to make your specific change to the amount of time it would take to improve the structure of the program as a whole so that future changes would be easier, quicker, and more manageable. If the time needed to refactor or rewrite the program is less than making the one current change, almost always make the systematic change then add your specific change to the new system. If the two are about the same, you'll probably want to rewrite or refactor and make your change to the new system. Even if it would take longer to make the systematic improvement than to make this single change, think seriously about making the system easier to change rather than just changing it. By "in the same ballpark", I mean whatever is a similar length of time in the terms of your development process. If we're talking about changes taking an hour, then "in the same balpark" might mean a day. If a change takes a week, then "in the same ballpark" probably doesn't mean 6 to 12 weeks even though that's the same ratio. Saying "I can get you a crufted together update by lunch, or I can give you a clean, high-quality change by tomorrow that will make these changes take mere minutes in the future" will probably go over well. Saying "I'm doing nothing new this quarter because this program that's running just fine needs to be overhauled" will probably not. Yet if you can make the code far more updateable in two weeks vs. making the one change in one week, there's probably a way to convince someone they can wait. Let's say you are likely to have 9 more changes in the near future. Say it would take x time to make your current changes. Let's say each change would take about the same amount of time, x. That's 10x. Let's say that completely overhauling the system would take 8 times as long as making this change, but it would make further changes take 10% as long because it's trivial to update your newer, cleaner system. That's 8x up front, and 0.9x in the near future. So for a total of 8.9x over ten changes, you've saved 1.1x time. The great thing is, not only have you saved enough time for another task and a long lunch, but the cleaner organization of your program will keep bearing fruit beyond these ten changes if you have further changes in the future. Yet if the one change is needed in production tomorrow due to a change in business process, then taking 8 days to get it done is unacceptable. If this change you're making is guaranteed to be the last change ever made to your program or suite of programs, then the time to refactor or rewrite it might not be worthwhile. The more changes there will be in the future, though, the higher dividend you get back from making your system easier to maintain. Always, always, when there are changes to be made, consider whether to improve maintainability rather than just doing the maintenance. When in doubt, give preference to the long-term gains of better maintainability. Any time a project that your need to maintain regularly starts to get difficult to maintain, pick a point at which the next change isn't needed immediately. Take as much time as you can budget between the request and the mandatory delivery time (without impacting other projects negatively) to clean up the system. Make your change against the new, more maintainable system. Ideally, you'd even document the estimate for the new feature in the old version of the system, the estimate against the new version, the actual time it took to code the edit for the new version, and time taken to make the system more maintainable. Use that as justification for more housekeeping work on projects in the future. If you can't make all your maintainability changes at once (which is likely, since you'll never foresee all the changes requested for an actively changing project anyway), then take baby steps. If the project's in shell, translate it straight into Perl the first time you get a chance. Then, modularize what you can. After that, take all the calls to cut, grep, sed, awk, wc, ls, and similar utilities and use the Perl idioms with the next change request. Then, use Perl's extra flexibility like overloading, hashes, code refs, and more to make the intent of the code and your control flow clearer with the next. The cost gets spread out this way, but the full benefits get realized later rather than sooner too. Consider architectural changes to the entire structure of the program, but don't get caught in the middle of writing a plugin framework, an event loop, or a scripting engine for your application when you're past due to deliver a simple feature addition.	[reply]


Perl: the Markov chain saw
	PerlMonks