Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Yet another config file editing programme : Tell me how to make it better !

by dazz (Beadle)
on Sep 02, 2021 at 07:40 UTC ( [id://11136353]=perlquestion: print w/replies, xml ) Need Help??

dazz has asked for the wisdom of the Perl Monks concerning the following question:

Hi I have written yet another Linux config setting program. It works but I am seeking advice in improving it. This one reads network settings from an input file. The file is generated by an application that accepts user entries to a GUI. The entries are fully checked to ensure they are within valid ip ranges before they are saved to the file. This programme will rarely run as a Systemd service only when the input file is changed by the user. Readability and maintainability are more important to me that speed or brevity. The input file looks like this:
#IP Configuration #Fri Aug 27 16:07:40 NZST 2021 routers=192.130.1.1 interface=eth0 domain_name_servers=8.8.8. 8.8.1.1 ip_address=192.130.1.10/24
The target configuration file is dhcpcd.conf. The cut down version of the file looks like this:
#static domain_name_servers=192.168.1.1 # fallback to static profile on eth0 #interface eth0 #fallback static_eth0 ### Static LAN IP profile profile static_eth0 static ip_address=192.130.1.10/24 static routers=192.130.1.1 static domain_name_servers=8.8.8. 8.8.1.1 # fallback to static profile on eth0 interface eth0 fallback static_eth0 # Access Point Static interface mimirbox wlan0 interface wlan0 static ip_address=192.130.3.10/24 nohook wpa_supplicant
The target section is the static IP LAN profile, and nothing else. I have written a programme that basically is divided into 2 sections:
- reads the input file into a hash.
- does a find and replace on the dhcpcd config file.
The programme works but it is a lot longer that it probably should be.
The programme is as follows:

#!/usr/bin/perl use strict; use warnings; use Tie::File; use feature "switch"; no warnings 'experimental'; # Declare constants use constant false => 0; use constant true => 1; # Declare variables my $ip_filename = '/home/mimir/mb_ip.cfg'; # The ip info entered by t +he user into the GUI my $dhcpcd_filename = '/home/mimir/d.conf'; # The dhcp configuration f +ile to be written to #my $dhcpcd_filename = '/etc/dhcpcd.conf'; # A hash of config parameters to be updated in order # of how they should be saved. # Initiate with reasonable start values. my %ip_params = ( interface => "usb0", ip_address => "1.1.1.0", routers => "127.0.0.0", domain_servers => "1.1.1.1"); my $ip_line; # whole parameter line my @ip_fields; # parameter fields in a line # The list of saved user param variables tie my @ip_array, 'Tie::File', $ip_filename or die "Cannot tie file '$ip_filename': $!"; ########### Get param values from the GUI input file mb_ip.cfg ### Read the values from mb_ip.cfg and save to a hash for $ip_line (@ip_array) { @ip_fields = split /=/, $ip_line; # get the parameter name and val +ue given ($ip_fields[0]){ when ( /interface/ ) { $ip_params{'interface'} = $ip_fields[1]; } when ( /ip_address/ ) { $ip_params{'ip_address'} = $ip_fields[1]; } when ( /routers/ ) { $ip_params{'routers'} = $ip_fields[1]; } when ( /domain_name_servers/ ) { $ip_params{'domain_servers'} = $ip_fields[1]; } default { print "no match to params\n\n" } } } print "Network parameters loaded into the array \n" +; for(keys %ip_params){ print(" $_ is $ip_params{$_}\n"); } untie @ip_array; ########### Find the lines in the config file to be replaced. # Find the required interface section # Initiate the boolean variables that only change one parameter # in the config file section. # When all are found and changed, jump out. my $isFoundInterface = false; my $isFoundIP = false; my $isFoundRouters = false; my $isFoundDNS = false; # The config file to search and replace variables tie my @dhcpcd_array, 'Tie::File', $dhcpcd_filename or die "Cannot tie file '$dhcpcd_filename': $!"; print " \n\n\n"; # Find and replace the config lines for $ip_line (@dhcpcd_array){ print "$ip_line\n"; # look for 'profile static_eth0' that marks start of the section if ( $ip_line =~ /^.*profile\s*static_$ip_params{'interface'}\s*/ ) + { $isFoundInterface = true; # Found the first line of the sectio +n. print "############## Found profile line\n" ; next; # Jump to the next line in the file. } # When the profile interface section is found, look for the parameters + to change if ( $isFoundInterface == true ) { given ($ip_line){ when ( /^.*static\s*ip_address=/ ) { $ip_line = " static ip_address=$ip_params{'ip_address'} +"; $isFoundIP = true; } when ( /^.*static\s*routers=/ ) { $ip_line = " static routers=$ip_params{'routers'}"; $isFoundRouters = true; } when ( /^.*static\s*domain_name_servers=/ ) { $ip_line = " static domain_name_servers=$ip_params{'dom +ain_servers'}"; $isFoundDNS = true; } default { if ($isFoundInterface and $isFoundIP and $isFoundRouters and $isFoundDNS ) { last; } #stop searching the co +nfig file when changes all done. } } } print "isIP : $isFoundIP\n"; print "isRouters : $isFoundRouters\n"; print "isDNS : $isFoundDNS\n\n"; print "looking for the ip config params\n"; } untie @dhcpcd_array;

Note: The program still includes debug print statements. None of the ip addresses are actually used by me. I looked through the CPAN library and could not find anything that precisely suits my needs. I don't do much perl programming so there is a lot I don't know.
I am seeking constructive feedback on my programme.

Dazz

Replies are listed 'Best First'.
Re: Yet another config file editing programme : Tell me how to make it better !
by eyepopslikeamosquito (Archbishop) on Sep 02, 2021 at 13:37 UTC

    What slapped me in the face the instant I saw your script is:

    • No subroutines!
    • Poor scoping of variables: $ip_line, for example, is a glaring example of an evil global variable.

    While this style of programming may work for small throw away scripts, it does not scale over time. As your program suite grows over time, you'll want to put utility subroutines into modules, along with unit tests, so the code can be understood in isolation, tested in isolation, and reused by multiple scripts. For that to work, you must avoid global variables. Instead, each subroutine must have well-defined inputs and outputs.

    To illustrate, here's a sample subroutine to read your input file along with an illustrative trivial main program that calls it:

    use strict; use warnings; use Data::Dumper; # Read an input file # Return a reference to a hash of properties sub get_properties { my $fname = shift; open my $fh, '<', $fname or die "open '$fname': $!"; my %hash_ret; my $line; while ( defined( $line = <$fh> ) ) { chomp $line; $line =~ s/^\s+//; # remove leading $line =~ s/\s+$//; # and trailing whitespace next unless length $line; # ignore empty lines next if $line =~ /^#/; # ignore comment lines my ($key, $val) = split /\s*=\s*/, $line, 2; $hash_ret{$key} = $val; } close $fh; return \%hash_ret; } @ARGV or die "usage: $0 property-file\n"; my $prop_file = shift; print "prop_file='$prop_file'\n"; my $properties = get_properties($prop_file); print Dumper( $properties );

    Note that this subroutine does not use any global variables, just reads the specified input file and returns a reference to a hash of properties ... so could be easily put into a module and reused by many different main programs in your suite.

    A couple of related points from On Coding Standards and Code Reviews:

    • Minimize the scope of variables, pragmas, etc.
    • Think about how to test your code from the beginning. This improves code quality because: writing a test first forces you to focus on interface (from the point of view of the user); hard to test code is often hard to use; simpler interfaces are easier to test; functions that are encapsulated and easy to test are easy to reuse; components that are easy to mock are usually more flexible/extensible; testing components in isolation ensures they can be understood in isolation and promotes low coupling/high cohesion. See also: Effective Automated Testing and Test::More.

Re: Yet another config file editing programme : Tell me how to make it better !
by kcott (Archbishop) on Sep 02, 2021 at 10:31 UTC

    G'day Dazz,

    "Readability and maintainability are more important to me that speed or brevity."

    That's great. Here's some suggestions:

    • Avoid experimental features: they are subject to change. Some weeks ago I wrote "Re: Modern exposition of "case" or Switch in perl 5.34?". There is discussion about switch with several examples of alternatives.
    • I question the use of Tie::File. Is there a specific reason you chose that? I would have thought a simple open would have been clearer: it's more readable and commonly understood (I suspect many would need to consult the documentation to understand the Tie::File syntax).
    • Consider the autodie pragma. Hand-crafting I/O exception messages is tedious and error-prone. Write less code and let Perl do the work for you.
    • On comments in your code:
      • In general, only use comments to clarify; not to reiterate or state the obvious.
      • Many are completely superfluous; for instance, "# Declare constants" immediately before "use constant ..." statements.
      • Some may become incorrect as the code matures; for instance, you have "mb_ip.cfg" in at least two comments, if those comments were necessary, which I question, use "$ip_filename" (if the filename changes, you won't have to search for and edit every occurrence of the hard-coded filename).
      • Some are just wrong to start with; for example, "# A hash ... in order ..." — hashes are unordered.
    • On constants:
      • It is customary to use uppercase: e.g. TRUE and FALSE; not true and false. Keeping to expected conventions will improve readability.
      • Group your constants into one or more blocks; for example,
        use constant { TRUE => 1, FALSE => 0, };
      • Note that my ($x, $y, $z); declares three variables all of which will evaluate to FALSE; there is no need to individually assign FALSE to each.
    • A final else (or default or whatever you use) should not contain a condition; that would be elsif.
    • When specifying whitespace in a regex, do so explicitly. Instead of "/^.*static\s*ip_address=/", it would better to write /^\s+static\sip_address=/; that matches leading whitespace (tabs and spaces) and won't match "whateverstaticip_address=":
      $ perl -E 'my $x = "whateverstaticip_address=X"; say $x =~ /^.*static\ +s*ip_address=/ ? "YES" : "NO"' YES $ perl -E 'my $x = "whateverstaticip_address=X"; say $x =~ /^\s+static +\sip_address=/ ? "YES" : "NO"' NO
    • Use consistent indentation. That will greatly improve readability and facilitate easier and less error-prone maintenance.

    There's quite a few points. There may be others that I didn't pick up on my first pass through your code.

    — Ken

Re: Yet another config file editing programme : Tell me how to make it better !
by tybalt89 (Monsignor) on Sep 02, 2021 at 18:09 UTC

    Readability more important than brevity :( Well, I can try...

    #!/usr/bin/perl use strict; https://perlmonks.org/?node_id=11136353 use warnings; use Path::Tiny; my $dhcpcdfile = 'fake.353'; # FIXME filename -w $dhcpcdfile or die "cannot write $dhcpcdfile"; my %ip_params = ( interface => "usb0", ip_address => "1.1.1.0", routers => "127.0.0.0", domain_servers => "1.1.1.1" ); %ip_params = ( %ip_params, # add new data to defaults path('inputfile.353')->slurp =~ /^(\w+)=(.*?)\s*$/gm ); # FIXME file +name use Data::Dump 'dd'; dd 'ip_params', \%ip_params; { # block scope for locals local @ARGV = $dhcpcdfile; local $^I = '.bak'; # make backup, do inplace edit my $foundinterface = 0; while( <> ) { if( /^\s*profile\s+static_$ip_params{interface}\b.*\n/m ) # alter th +is section { $foundinterface = 1; warn "found section for $ip_params{interface} at line $.\n"; } elsif( $foundinterface and /static/ ) { s/^\s*static\s+(\w+)=\K.*/$ip_params{$1}/m or warn "failed to chan +ge $1"; warn "set $1 to $ip_params{$1}\n"; } elsif( $foundinterface ) { $foundinterface = 0; warn "ending changes at line $.\n"; } print; } }

    The warns and Data::Dump can be removed, of course, making it even more readable :)

      Hi
      I have had a go a re-crafting the code by @tybalt89 and other comments. It works but I don't know why. It also spits out errors.

      The errors indicate $1 for the params hash is not initialised on the line:
       s/^\s*static\s+(\w+)=\K.*/$ip_params{$1}/m or warn "failed to change $1";
      I can't see where it is, or should be, initialised.

      If you think I should be able to work this out for myself, you are probably right but my excuse is that I am suffering the effects of concussion from a bicycle crash that nearly killed me in February. Writing some software is exercise for my brain in an attempt to restore something like normal service. As part of my mental gymnastics, I have also been refurbishing a vintage RF signal generator. The diary writeup is here: https://www.eevblog.com/forum/repair/wavetek-2025a-0-2-2-200mghz-rf-sig-gen-repair/
      As with software, I haven't professionally worked on hardware for decades.

      The current code looks like this:
      #!/usr/bin/perl ##### FUNCTION # To read ip settings from a machine generated file and # write those ip settings to the dhcpcd.conf file. # This script is designed to be started by a systemd service.path # The service monitors the mb_ip.cfg file for change. When the file +is changed # the service calls this script. # Updated ip settings are applied after reboot. ##### INPUT # Takes user updates of IP settings saved in to the file /home/mimir/ +mb_ip.cfg # The user setting are range checked by the app before being saved to + file. # The file mb_ip.cfg will only have ip settings that are valid (but n +ot necessarily correct). ##### OUTPUT # Writes the ip settings to the /etc/dhcpcd.conf file. # # Reference: https://www.perlmonks.org/?node_id=11136353 # # Dazz # ver 1.3 # 8 Sept 2021 use strict; use warnings; use Path::Tiny; use Data::Dump; ###### Input file with ip settings my $ip_filename = '/home/mimir/mb_ip.cfg'; # The ip info entered by t +he user into the GUI ###### Output dhcp configuration file # my $dhcpcdfile = '/etc/dhcpcd.conf'; my $dhcpcdfile = 'd.conf'; # TEST ###################################################################### +###################### use constant { FALSE => 0; TRUE => 1; }; # Load the input parameters ip_params from the input file my %ip_params; %ip_params = ( %ip_params, # add new data to defaults, split with "=" path($ip_filename)->slurp =~ /^(\w+)=(.*?)\s*$/gm ); # Print the contents of ip_params. #TEST use Data::Dump 'dd'; dd 'ip_params', \%ip_params; #TEST # Convert 2 lines DNS to 1 line DNS. $ip_params{domain_name_servers} = $ip_params{domain_name_server_1}." " +.$ip_params{domain_name_server_2}; delete($ip_params{domain_name_server_1}); delete($ip_params{domain_name_server_2}); # Print to check the contents ip_params with combined DNS line. + #TEST use Data::Dump 'dd'; dd 'ip_params', \%ip_params; + #TEST { # block scope for locals local @ARGV = $dhcpcdfile; local $^I = '.bak'; # make backup, do inplace edit my $foundinterface = FALSE; while( <> ) { if( /^\s*profile\s+static_$ip_params{interface}\b.*\n/m ) # look for + profile with matching interface name { # format m +atches 'static profile_eth0' $foundinterface = TRUE; warn "found section for $ip_params{interface} at line $.\n"; } elsif( $foundinterface and /static/ ) { warn "ip param key : $ip_params{$0}\n"; #TEST warn "ip param val : $ip_params{$1}\n"; #TEST ***ERROR: $1 not initia +ted. Lines 72 and 73*** s/^\s*static\s+(\w+)=\K.*/$ip_params{$1}/m or warn "failed to chan +ge $1"; # match string before \K Return string after \K # ?? What increments $ip_params through the hash ?? # ?? What stops over-writing of other later sections ?? eg. wlan0 warn "set $1 to $ip_params{$1}\n"; } elsif( $foundinterface ) { $foundinterface = FALSE; warn "ending changes at line $.\n"; } # Output line to dhcpcdfile print; } }
      The input file is this: (slightly different to the previous version)
      #IP Configuration #Mon Sep 06 14:29:34 NZST 2021 routers=192.168.9.91 interface=eth0 domain_name_server_2=8.8.4.9 domain_name_server_1=8.8.8.9 ip_address=192.168.10.9/24
      The abbreviated output file to be updated is this:
      #static domain_name_servers=192.168.1.1 # fallback to static profile on eth0 #interface eth0 #fallback static_eth0 ### Static LAN IP profile for Mimirbox profile static_eth0 static ip_address=1.1.1.1/24 static routers=1.1.1.1 static domain_name_servers=1.1.1.1 2.2.2.2 # Access Point Static interface mimirbox wlan0 interface wlan0 static ip_address=192.130.2.20/24
      If these 3 files are placed in the same directory, the code should run.

      Dazz

        domain_name_servers does not exist in your input file. There is domain_name_server_2 and domain_name_server_1 instead.
        Just a name mismatch.

        What do you think the value of $0 is in the line ?

        warn "ip param key : $ip_params{$0}\n"; #TEST
Re: Yet another config file editing programme : Tell me how to make it better !
by bliako (Monsignor) on Sep 02, 2021 at 07:56 UTC

    After a quick look at your code: how about removing comments before processing?

    I think that's a must because you do allow comments (in your examples), but then you do when ( /interface/ ) { which will match a commented-out interface line or anything really that mentions the word "interface".

    What I usually do for removing comment lines is $line =~ s/#.*$//; next if $line =~ /^\s*$/;

    Secondly, if your grammar will grow then you are better off using other solutions, which other Monks I am sure will mention.

    bw, bliako

      Hi
      Thanks for your feedback.

      The input file is machine generated so the content and format is well defined. The regex is loose.
      The output config file could be hand edited and so the regex is a lot more restrictive. Comments are ignored.

      I am hoping to be shown a better way of doing this.

      Dazz
Re: Yet another config file editing programme : Tell me how to make it better !
by dazz (Beadle) on Sep 03, 2021 at 00:55 UTC
    Hi OK some good feedback there. I appreciate the efforts.
    Just some feedback on some of the points.
    I usually write the comments first then write the code to do what the comments say. When I go back to my code long after I have forgotten what I wrote, I read my comments to figure out what is doing.
    I started using Switch, then I read in CPAN "do not use if you can use given/when". Then when I used given/when, I ended up with warnings plus advice not to use given/when. Hmmmmmm. As an occasional perl user, that is a little frustrating rewriting the same section of code 3x to do exactly the same thing. I just need something that has case type functionality.
    My program is running on a Raspberry Pi. There is a very small, but finite risk of the Pi being turned off in the middle of a file write. Tie::File looked like it would minimise, but not eliminate, the risk of file corruption due to power loss. I have no idea which CPAN modules are commonly, or infrequently used. How could I know??
    I was actually expecting to be told that I should have used CPAN XXX module, written specifically to do the sort of thing I am doing. Editing config files is a common requirement. I know there are modules for ini files, but apparently not for the dhcpcd.conf type of file that has sections, but not really marked as such.
    The major flaw with my code is the risk that one of the parameters is deleted from the config file before my code is run, causing the script to keep looking in the following sections for the missing parameter. The easiest workaround is to place the section last in the file so the search will hit EOF as the section end marker.
    I accept that my code writing is far from perfect, and I will be going through my code to make improvements and corrections based on the advice received. If I haven't mentioned your feedback above, it is not because I have ignored it.
    Thanks.

    Dazz

      There is a very small, but finite risk of the Pi being turned off in the middle of a file write
      Ha ha, I have long experience contemplating this annoying and tricky problem! The straightforward solution I concocted back in 2003 (and am still happy with) is to simply write a new file on the same file system ... and then use (atomic) rename to clobber the original file - but only after the new file has been successfully written. This is described in detail at:

        See ->spew (and ->edit and ->edit_lines) in Path::Tiny

      Solution using Path::Tiny which writes the new file alongside the existing file, then does an atomic rename. See Re^2: Yet another config file editing programme : Tell me how to make it better ! and Re^3: Yet another config file editing programme : Tell me how to make it better !

      #!/usr/bin/perl use strict; # https://perlmonks.org/?node_id=11136353 use warnings; use Path::Tiny; my $dhcpcdfile = 'fake.353'; # FIXME filename -w $dhcpcdfile or die "cannot write $dhcpcdfile"; my %ip_params = ( interface => "usb0", ip_address => "1.1.1.0", routers => "127.0.0.0", domain_servers => "1.1.1.1" ); %ip_params = ( %ip_params, # add new data to defaults path('inputfile.353')->slurp =~ /^(\w+)=(.*?)\s*$/gm ); # FIXME file +name my $foundinterface = 0; path( $dhcpcdfile )->edit_lines( sub { if( /^\s*profile\s+static_$ip_params{interface}\b.*\n/m ) # alter th +is section { $foundinterface = 1; } elsif( $foundinterface and /^\s*static/m ) { s/^\s*static\s+(\w+)=\K.*/$ip_params{$1}/m or die "failed to chang +e $1"; } elsif( $foundinterface ) { $foundinterface = 0; } } );
      I started using Switch, then I read in CPAN "do not use if you can use given/when". Then when I used given/when, I ended up with warnings plus advice not to use given/when. Hmmmmmm. As an occasional perl user, that is a little frustrating rewriting the same section of code 3x to do exactly the same thing.

      I completely agree and understand. The (mis-)management of the switch/case equivalent in Perl over the years has become something of a lesson in how not to do it. given/when never struck me as a good idea so I just avoided it but plenty succumbed. Similarly, I never went anywhere near smartmatch but only a minority were enticed into that one. FWIW the current guidance is here. It will probably change again.

      Sometimes being behind the curve is a good thing. These days I usually write Perl which is something like 5 years behind the current release in terms of features. This gives enough lead-in to be able to ascertain which "new" features have turned out to be turkeys and should be avoided. Such an approach is not for everyone, of course, but it can have its advantages (not least of which is back-compatibility).


      🦛

      Readability and brevity - together again !

      #!/usr/bin/perl use strict; # https://perlmonks.org/?node_id=11136353 use warnings; use Path::Tiny; my $dhcpcdfile = 'fake.353'; # FIXME filename my %ip_params = ( # defaults interface => "usb0", ip_address => "1.1.1.0", routers => "127.0.0.0", domain_servers => "1.1.1.1" ); %ip_params = ( %ip_params, # add new data to defaults path('inputfile.353')->slurp =~ /^(\w+)=(.*?)\s*$/gm ); # FIXME file +name { # block for local local $/ = ''; # paragraph mode path( $dhcpcdfile )->edit_lines( sub { if( /^\s*profile\s+static_$ip_params{interface}\b.*\n/m ) # alter th +is section { s/^\s*static\s+(\w+)=\K.*/$ip_params{$1}/gm; } } ); }
        { # block for local local $/ = ''; # paragraph mode path( $dhcpcdfile )->edit_lines( sub { if( /^\s*profile\s+static_$ip_params{interface}\b.*\n/m ) # alter th +is section { s/^\s*static\s+(\w+)=\K.*/$ip_params{$1}/gm; } } ); }
        To an experienced perl programmer with deep knowledge of the language, this might look readable but for an occasional unprofessional perl user like myself, it would probably take at least half an hour to figure out what it is doing.
        If I came back to this sort of code in a couple of years to alter/reuse it, I'd be back to square one.
        If I had a future application that required a different input, this sort of all-in-one read/write approach would be difficult for me to repurpose.
        It seems that brevity and obfuscation in perl code are inseparable.

        For me, readability and brevity would be discovering a module Unix::ConfigFile::DHCPCD, that included a method "UpdateInterfaceIP". I think that a valid metric for a modern language is the code I don't have to write.

        Please note I am not criticizing the skill or helpfulness those that give up their own time to write replies to people like me. I don't want to sound ungrateful. This is my goto place to find expert advice on perl, but claiming code is "readable" on a website that has a section devoted to "Obfuscation" is not a good look.


        Dazz

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11136353]
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chanting in the Monastery: (8)
As of 2024-04-19 12:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found