http://qs321.pair.com?node_id=11100287

rje has asked for the wisdom of the Perl Monks concerning the following question:

Good morning fellow monks.

On purpose, I volunteered to update a deploy script. The current script is a Perl 4 stand-alone script that copies files into their proper deployment locations.

How you would do this?

I sold my boss on moving common code into a package, because it would (1) hide and reuse duplicated code, (2) shorten new typical deploy scripts dramatically; (3) potentially reduce accidental errors, (4) introduce such innovations as use strict; and use warnings; and (5) give the systems team a tiny bit of control over code maintainability, as opposed to 'no control' today.

CONSTRAINT: I'm limited to the standard distribution for v5.10.1. I can't install packages.

I'm thinking of a package that minimizes the "visual change" of the existing way, but adds a healthy dose of SAFER.

Here's a very typical example. Note the lack of safe practices.

THE WAY IT IS NOW

#!/usr/bin/perl #--------------------------------------------------------------------- +------------- # Filename: my_filename_goes_here.pl # # This script is executed by the deploy tool as part of the standard # deploy process. This script moves batch SE work flow batch scripts +to the correct # locations where <<we>> expect to find them within the local filesyst +em. # # Date Developer Comment # ---------- ---------------- ------------------------------- +------------- # 02-06-2019 R. Eaglestone Initial creation #--------------------------------------------------------------------- +------------- use File::Copy; # Define script-global variables. $outputDir = "/whatever/data/output"; $executionDir = "/whatever/jobs"; $installDir = "/www/apps/install/path/for/my/process/scripts"; %dirsToDeploy = ( "$outputDir/boo" => 0777, "$outputDir/boo/jum" => 0777, "$executionDir/boo" => 0777, "$executionDir/boo/jum" => 0777, ); %pathsToDeploy = ( "MY_LAUNCHER_11023200099.sh" => '/fancy/deploy/path', "fileSanity.pl" => '/fancy/deploy/path', "transferFile.sh" => '/fancy/deploy/path', ); %filesToDeploy = ( "MY_LAUNCHER_11023200099.sh" => 0775, "fileSanity.pl" => 0775, "transferFile.sh" => 0775, ); @filesToConvert = ( "MY_LAUNCHER_11023200099.sh", "fileSanity.pl", "transferFile.sh", ); # This subroutine ensures that all files are present. sub checkDeployValidity () { # First, check that the install directory exists. if ( !(-d $installDir) ) { return "install directory $installDir was not deployed."; } # Then, check that the base execution directory exists. if ( ! (-d $executionDir) ) { return "execution directory $executionDir does not exist."; } # Next, set permissions on the installation directory. if ( ! chmod(0775, $installDir) ) { return "Setting permissions on directory $installDir failed."; } # Now, check that all files were deployed. my @scriptFileNames = sort(keys(%filesToDeploy)); foreach $fileName (@scriptFileNames) { $currentFile = "$installDir/$fileName"; if ( !(-f $currentFile) ) { return "file $currentFile was not deployed."; } } # Finally, if all files are present, then return success. return ""; } # This subroutine builds any missing directories. sub setUpDirectoryStructure () { # Build any subdirectories, if necessary. my @directoryNames = sort(keys(%dirsToDeploy)); foreach $dirName (@directoryNames) { # Check whether the directory exists... if ( -d $dirName ) { # If so, then set its permissions appropriately. if ( ! chmod($dirsToDeploy{$dirName}, $dirName) ) { print "\tunable to do 'chmod $dirsToDeploy{$dirName} $dirName' +.\n"; } } elsif ( ! mkdir($dirName, $dirsToDeploy{$dirName}) ) { print "\tunable to create directory '$dirName'.\n"; } } # Finally, return success once all directories exist with proper per +missions. return ""; } # This subroutine removes carriage return characters from specific fil +es. sub dosToUnix () { foreach $fileName (@filesToConvert) { my $file = "$installDir/$fileName"; my $tempFile = "$file.tmp_dos2Unix"; open(DOS_FILE, "< $file") || return "cannot open $file for re +ad: $!"; open(UNIX_FILE, "> $tempFile") || return "cannot open $tempFile fo +r write: $!"; # Remove all carriage return ('\r') characters. while (<DOS_FILE>) { s/\r//g; print UNIX_FILE || return "cannot write to $tempFile: $!"; } close(UNIX_FILE) || return "cannot close $tempFile: $!"; close(DOS_FILE) || return "cannot close $file: $!"; rename($file, "$file.orig") || return "cannot rename $file to $fil +e.orig: $!"; rename($tempFile, $file) || return "cannot rename $tempFile to +$file: $!"; unlink("$file.orig") || return "cannot delete $file.orig: +$!"; } return ""; } # This subroutine copies files from the install dir to the execution d +ir. sub copyFilesToTarget () { my @fileNames = sort(keys(%filesToDeploy)); foreach $fileName (@fileNames) { my $source = "$installDir/$fileName"; my $path = $pathsToDeploy{$fileName}; my $target = "$path/$fileName"; if ( -f $target) { chmod(0777, $target) || return "chmod 777 for existing file '$ta +rget' failed."; } copy($source, $target) || return "file copy to '$target' failed."; chmod($filesToDeploy{$fileName}, $target) || return "chmod for '$t +arget' failed."; } return ""; } # This subroutine orchestrates the deploy process. sub main () { # Start by setting umask to 0 to simplify mkdir commands. my $oldUMask = umask(0000); # Next, check that all of the files exist. my $errorMessage = &checkDeployValidity(); if ( $errorMessage ) { umask($oldUMask); die("$0: ERROR - $errorMessage"); } # Then, set up the execution directory structure. $errorMessage = &setUpDirectoryStructure(); if ( $errorMessage ) { umask($oldUMask); die("$0: ERROR - $errorMessage"); } # Convert files from DOS format to Unix format. $errorMessage = &dosToUnix(); if ( $errorMessage ) { umask($oldUMask); die("$0: ERROR - $errorMessage"); } # Now, copy the files over. $errorMessage = &copyFilesToTarget(); if ( $errorMessage ) { umask($oldUMask); die("$0: ERROR - $errorMessage"); } # Finally, restore the old umask value (in case this process is reus +ed). umask($oldUMask); print "Copy succeeded. Job finished successfully.\n"; exit(0); } &main ();

THE WAY I'M THINKING IT COULD BE, my knee-jerk gut reaction:

#!/usr/bin/perl #--------------------------------------------------------------------- +-------------- # Filename: my_filename_goes_here.pl # # This script is executed by the deploy tool as part of the standard # deploy process. This script moves batch SE work flow batch scripts +to the correct # locations where <<we>> expect to find them within the local filesyst +em. # # Date Developer Comment # ---------- ---------------- ------------------------------- +-------------- # 02-06-2019 R. Eaglestone Initial creation #--------------------------------------------------------------------- +-------------- use DEPLOYER; use File::Copy; # Define script-global variables. DEPLOYER::outputDir("/whatever/data/output"); DEPLOYER::executionDir("/whatever/jobs"); DEPLOYER::installDir("/www/apps/install/path/for/my/process/scripts"); DEPLOYER::dirsToDeploy ( "$outputDir/boo" => 0777, "$outputDir/boo/jum" => 0777, "$executionDir/boo" => 0777, "$executionDir/boo/jum" => 0777, ); DEPLOYER::pathsToDeploy ( "MY_LAUNCHER_11023200099.sh" => '/fancy/deploy/path', "fileSanity.pl" => '/fancy/deploy/path', "transferFile.sh" => '/fancy/deploy/path', ); DEPLOYER::filesToDeploy ( "MY_LAUNCHER_11023200099.sh" => 0775, "fileSanity.pl" => 0775, "transferFile.sh" => 0775, ); DEPLOYER::filesToConvert ( "MY_LAUNCHER_11023200099.sh", "fileSanity.pl", "transferFile.sh", ); DEPLOYER::deploy();

Thus, DEPLOYER would use strict and warnings, have all the deployment guts in it, in a common module managed by the systems team, as opposed to the current chaos.

As I said, this is my gut reaction. Before I rush ahead with this, would you guys have any suggestions for me to consider?

Replies are listed 'Best First'.
Re: Modernizing a Deploy script to standard distro v5.10.1
by stevieb (Canon) on May 20, 2019 at 17:12 UTC
    "Before I rush ahead with this, would you guys have any suggestions for me to consider?"

    Yep!

    Note I don't have the time to go through the code at the moment (it's too nice out today to not be out in one of the boats!), but here's how I'd approach the overall project:

    • create a new distribution using the module-starter script installed by the Module::Starter distribution
    • inside of the newly-created distribution directory, create a "legacy" directory, and add your existing script to it
    • put all code in a Version Control System (Git, for example)
    • write unit tests to cover *all* of the existing legacy script functionality before doing anything else
    • write the same unit tests for the new script/module configuration, until you get at minimum the exact same functionality covered between legacy and the new setup (see Devel::Cover)
    • write proper documentation *as you write code*. Same goes for tests. You should write tests before, during and after each piece of functionality as its fresh on your mind
    • ask your systems teams for a development/test/qa system for every single platform type you'll be running your code on
    • ensure all tests (legacy and new) pass on all of these various platforms (use perlbrew or berrybrew to easily install an instance of each version of Perl you have in use across your infrastructure, so you know all tests pass on all of them)
    • impress boss and various teams when you confidently move to production, because all of the hard work was done in advance, and now your systems/infra teams don't need to scramble because of your unforseen circumstances :)
      • -stevieb

      THANK YOU. Yes, that's the best advice I've heard in a long time.
Re: Modernizing a Deploy script to standard distro v5.10.1
by holli (Abbot) on May 21, 2019 at 00:17 UTC
    Ah, this reminds me of the goold old times when I used to refactor CGI-scripts for a living. See, the script ain't that bad. Apparently it works, is battle tested and the worst thing is the "lack of control" what I interpret as changing the configuration of the script easily. Here is how you refactor something like that:

    First thing you do is adding use strict, run the script and cry a little. But then you boldy add the the missing my statements at the broadest possible scope (usually file global) until the script compiles again.
    Now your script is strict-safe, but you still have a lots of globals. You want to limit their scope. So pick one global and add a underscore to the name. Run the script.
    In all subroutines where now are errors, add the missing variable (as a reference) to the subroutines argument list. Within the subroutine change all occurances of the global variable to new reference, change code where if neccessary (e.g. $formerGlobal[0] to $nowReference->[0] ).
    Now find all callers of the subroutine and change them so they pass in the global variable to the subroutine.
    Rinse and repeat with all globals and all subroutines they appear in.

    While you're at it change the calling style of subroutines from Perl4 &foo too foo() at several places.
    After that you have subroutines that are independent of any global variables which now appear only in the main function. The subs are now "real functions". Black boxes with no side effects that are functionally separated, have a clean interface and can be unit tested.

    And that is now the ideal point to get rid of the globalfest alltogether by factoring them out into a configuration file. This gives you the control you want, if you make it so the user can pass in a config file name to use.

    Wow, that's a lot of text, my nodes usually are way more concise but I guess this node made a litte nostalgic. So nostalgic in fact I coded it right away for fun =)
Re: Modernizing a Deploy script to standard distro v5.10.1
by Fletch (Bishop) on May 20, 2019 at 15:16 UTC

    Not strictly related but 5.10 has been end of life'd for a while now; prior to 5.22 is unsupported at the moment. See perlpolicy.

    Edit: Eew; know how that goes. I once had to write perl5 code that would create perl4 code to ship because that was what Tivoli was still using. *shudder*

    The cake is a lie.
    The cake is a lie.
    The cake is a lie.

      Yeah. I have no control over that. :(
Re: Modernizing a Deploy script to standard distro v5.10.1
by eyepopslikeamosquito (Archbishop) on May 21, 2019 at 11:03 UTC
Re: Modernizing a Deploy script to standard distro v5.10.1
by RonW (Parson) on May 24, 2019 at 18:22 UTC

    (Sorry for the delayed reply. I started then got interrupted by an emergency.)

    You might also want to consider having DEPLOYER do a little more of the "grunt work".

    You have 3 lists of the same files: pathsToDep­loy, filesToDep­loy and filesToCon­vert. You only need one list/table, then derive the individual lists.

    In DEPLOYER, do

    my %pathsToDep­loy; my %filesToDep­loy; my @filesToCon­vert; sub Deploy { my %ToDep­loy = @_; %pathsToDep­loy = map { $_ => $ToDep­loy{$_}->[0] } keys %ToDep­l +oy; %filesToDep­loy = map { $_ => $ToDep­loy{$_}->[1] } keys %ToDep­l +oy; @filesToCon­vert = keys %ToDep­loy ); # other code }

    Then in your deploy script, you only need:

    DEPLOYER::Dep­loy ( "MY_LAUNCHER_1102320­0099.sh" => [ '/fancy/deploy/path', 0775 ]­, "fileSanity.pl" => [ '/fancy/deploy/path'­, 0775 ], "transferFile.sh" => [ '/fancy/deploy/path'­, 0775 ], );

    This will make the deploy scripts easier to configure because each file is mentioned only once, instead of the 3 times the legacy deploy scripts works and your current new proposal.

    Also, as long as DEPLOY.pm has use File::Copy; the deploy scripts only need use DEPLOY;