Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Single file oriented for different usage

by arbingersys (Pilgrim)
on Feb 26, 2008 at 16:55 UTC ( #670323=perlquestion: print w/replies, xml ) Need Help??

arbingersys has asked for the wisdom of the Perl Monks concerning the following question:

I have a Perl/MySQL project called Sylbi, and it uses a configuration file, Config.pm, that contains such things as DBI connection strings, session settings, etc. I've noticed this before in other projects, but I'm noticing it acutely on this one: That I'm basically having to maintain several copies of Config.pm. One that's being actively developed and might have experimental changes in it, one that has sensitive authentication information for my online demo, and one with default, stable settings for inclusion in the distro.

So I think the problems are pretty obvious that come about with this kind of setup. (1) Redundant edits. If I add, remove, change anything significant in the dev Config.pm, I also have to make this change in the other files. (2) Memory faults. It's easy to forget to make the redundant changes, or even to copy the wrong file to the wrong place (i.e. my demo), causing failures all around. Also in this vein is the mere headache of remembering, after some time has elapsed since working on the project, where I stored the alternate Config.pm files, whether they each contain the appropriate settings, etc.

I can't imagine this is a new problem. How would/do the Monks handle this type of thing? Are there any tools out there that would provide a better, more intuitive way to manage a single file with different orientations?

Update: I should probably try and make myself a little more clear. I was thinking that it would great if within a single file (i.e. Config.pm), you could have sections defined that could then be omitted under certain build conditions. This would be especially helpful, I think, while in development. You would only have to edit a single file, and, as in the cases I outline above, some settings flagged experimental would be removed when building the distribution, or another section with some sensitive data would be included only if you were assembling the 'demo'. Are there any tools/techniques that lend themselves to this idea?

A blog among millions.
  • Comment on Single file oriented for different usage

Replies are listed 'Best First'.
Re: Single file oriented for different usage
by Ineffectual (Scribe) on Feb 26, 2008 at 18:23 UTC
    We use a package called Config::Inifiles for all of our configuration files. You can have sections in a file that have headings and load information from each of the headings in your code. You should be able to make a [GLOBAL] setting and a [DEV] section. You should probably include a second ini file for the authentication information though and have your package expect there to be two files (one with GLOBAL/DEV settings and one with AUTH information).

      Thanks, that could work. I'll definitely check it out.

      A blog among millions.
      Yes, I was going to suggest something similar: Config::ApacheFormat lets you use Apache-style configuration files for other purposes, so you can, for example, override the global definitions based on path.

      I have used a similar approach (though I've never used Config::Inifiles...somehow I missed it...I'm going to check it out) and it works pretty well for us. In our case we develop on Win32 systems and then run the applications on Unix systems...so we use different Config files to handle the differences between Win32 and Unix.

      The authentication is interesting to me as we've recently been looking at requiring login + authentication on our applications to ensure that only folks authorized to be commanding our systems are allowed to do so (we already have a login layer just to get on the command & control computers and that's been good enough so far...but we're in the midst of some changes that make that vulnerable). So I've been considering having a development login and an operations login. So this thread is especially interesting to me.

      ack Albuquerque, NM
Re: Single file oriented for different usage
by dragonchild (Archbishop) on Feb 26, 2008 at 17:04 UTC
    If using a single file is causing you headaches, then don't use a single file.

    In my projects, we solve that problem by having two config files. One is the main one, one has $USER-specific overrides. So, when you run the app, the main config file is loaded, then, if one exists, the $USER-specific one is loaded and overrides anything it needs to override.


    My criteria for good software:
    1. Does it work?
    2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

      Thanks. That would certainly work in the case where the parameters are established, and they won't change much. (Like if I ever near stable, non-beta releases.) But while it's under active development and I still might be experimenting, I can't guarantee that parameters, parameter names, etc, won't shift around, be removed, or altered in a way that would break a user config file. Any suggestions in this case?

      A blog among millions.
        You couldn't guarantee that anyways, even in prod. Just look at how often parameter names change in Oracle, let alone MySQL or Apache. Just get it done. :-)

        My criteria for good software:
        1. Does it work?
        2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
Re: Single file oriented for different usage
by apl (Monsignor) on Feb 26, 2008 at 17:12 UTC
    How about a run-string parameter with one of three possible values ( 'EXP', 'DEMO', 'LIVE'). Your single configuration file could then have EXP_DB_HOST, DEMO_DB_HOST and LIVE_DB_HOST as opposed to DB_HOST as a field.

    Common fields would remain consistant, but you'd have to add a little intellegence when getting a detail that can vary.

      (It begins to look a little like #ifdef/#include, doesn't it? :)

      My mind sort of wandered in this direction, and it would work for some of the problem, definitely. But I wouldn't want the username/passwords for the online demo in the source, obviously.

      I've realized that my initial query was a bit unclear, and what I'm really asking is whether a single file could be diced up while 'Make'-ing a target to include the appropriate differences, i.e. EXP, LIVE, DEMO. Sorry about that.

      A blog among millions.
Re: Single file oriented for different usage
by snoopy (Curate) on Feb 27, 2008 at 00:20 UTC
    Hey arbingersys, tell me about it. I'm maintaining a number of loosely coupled, but interelated web site instances. In addition there are multiple development and test sites. A lot of the information is shared, but to give each it's own copy, that's a cut and paste nightmare.

    Since you ask, here's how I've approached it:

  • I maintain a meta config file. This has knowledge about all the websites in the cluster.
  • At run time, when I load the meta-config, it conditionally extract just the config information required by this application, running on this host, in this working directory:

    A snippet from a typical (YAML) config file and Perl extraction code follow. The "about" clauses represent the conditional parts:

Re: Single file oriented for different usage
by ruzam (Curate) on Feb 27, 2008 at 02:57 UTC
    My current "design in progress" experimental web application environment makes heavy use of file paths to control what gets used where. Pretty much every file path from the config files, to the modules, to images, javascript source and CSS are searched and loaded as needed at run time according to first found in the path order. My production application is at the root of the search path in the "main" path. I run my test version with an extra "dev" path inserted before "main". Any file (including config files) not found in the "dev" path automatically fall back to the "main" path. When I start hacking up a new version, I copy each file I edit into the "dev" path as I need to touch it. In dev I can change anything I want, create new stuff and generally yank the application around while still using the existing prod code/data directly for anything not changing. At any time I can create a new path for testing/bugfixing/development and re-order paths on top of paths so I can test fixes on fixes.

    It's actually alot slicker than my feeble description :)

    More to your question, I routinely keep several different paths (main, test, dev) each with their own variation of config file (different databases, different login, different file paths). Even config file format changes between paths are not an issue.
Re: Single file oriented for different usage
by absolut.todd (Monk) on Feb 27, 2008 at 02:58 UTC
    I have a way of doing it which I have found useful. Unfortunatly I cant take the credit for creating it.

    This is the minimal form of it

    package Foo::Config; use ... BEGIN { use Exporter; @EXPORT = qw/ &config /; # the following is the config data %config = ( dev => { A => {foo => 'dev_a'}, B => {this_is_not_foo => 'not_dev_b'}, _default => { foo => 'default_foo', database_env => 'dev_db', }, }, prod => { A => {foo => 'prod_a'}, B => {this_is_not_prod_foo => 'prod_b'}, _default => { foo => 'default_prod_foo', database_env => 'prod_db', }, }, _default => { A => { default_across_all_a_environments => 'def', }, _default => { the_same_everywhere => 1 }, }, # other stuff you want to run in the BEGIN block, i.e. set up the defa +ult environment. One thing i do is get a $environment variable based + on the hostname. } #END BEGIN sub config($) { my $field = shift; if (exists $config{$environment}->{$sub_environment}{$field} ) { return $config{$environment}->{$sub_environment}{$field}; }; if (exists $config{'_default'}->{$sub_environment}{$field} ) { return $config{'_default'}->{$sub_environment}{$field}; }; if (exists $config{$environment}->{'_default'}{$field} ) { return $config{$environment}->{'_default'}{$field}; }; if (exists $config{'_default'}->{'_default'}{$field} ) { return $config{'_default'}->{'_default'}{$field}; }; return undef; }
    Then in my code i do the following:
    use Foo::Config; my $config_var = config('foo');
    The way it works is as follows. The %config defines all the configuration across all environment. In the case where we use it we have a production and development box, as well as a sub environment based on different countries (here A B and C). When a configuration variable is needed the config method is called. This then walks through the %config to pull out the correct variable.

    There are several layers to it. From the most specific to the more general case:

  • $environment->$sub_env - This is details specific to that environment. I.e. the database schema may be specific to dev and production, as well as the sub-environment.
  • $environment->_default - this is for all elements within an environment which may be the same across an environment. For example development and production databases may be on different machines, but the same machine serves all sub-environments.
  • _default->$sub_env - this is for all elements which are consistent across all environments, but may be different for each sub-environment, i.e. a reply-to address for the email may be different for each country.
  • _default->_default - this is for all elements which are consistent across all environment. i.e. the body of an email you want to send out.

    The nice thing about this is that the more specific case will overwrite the more general ones. So its easy to change specific cases, i.e. a sub environment's database is moved to a new machine, so you only need to add a new config variable for that.

    Actually, re-reading your original post this doesn't really help with that problem. But i thought i might still post it as a nice solution ive found to multiple configuration problems.

      Hmmm. Most interesting. I just posted a reply to one of the other responses regarding a current issue that I, too, have regarding flexibility in processing Config files. This is actually a little closer to my specific problem and looks promising. Thanks for the code snippet. It'll give some good things to mull over.

      ack Albuquerque, NM
Re: Single file oriented for different usage
by clinton (Priest) on Feb 27, 2008 at 12:27 UTC

    Take a look at Config::Merge (my module) which loads configuration from a directory tree of files in any of several formats (XML, YAML, ini, Perl, Config::General), merging them together into a single hash.

    It has a second phase, that of merging in local overrides - you can override single hash keys, single elements in arrays, or whole sub-trees within the config. The default setup merges in a file called local.$file_type at the point where it finds it in the directory tree. However, in the examples, I show how you could customise this process to load this data based on (eg) host name, or whatever else tickles your fancy

    hth

    Clint

      Nice module, Clint. And it is documented in an exemplary way.
Re: Single file oriented for different usage
by arbingersys (Pilgrim) on Feb 27, 2008 at 15:40 UTC

    Thanks, everyone who has posted so far. You've given me a lot to mull over.

    A blog among millions.
Re: Single file oriented for different usage
by scorpio17 (Canon) on Feb 28, 2008 at 17:40 UTC
    Another (simple) trick is to have a "production_config.ini" and a "development_config.ini", then create a symbolic link called "config.ini". On the production machine, the link points to the production config file and on the development machine it points to the development config file. So, regardless of which machine you're on, you always edit "config.ini", and your app always reads from that file, but it's contents vary depending on which machine you're on. If you use source-code-control (like svn) then you're able to back up both versions (and you can just ignore the symbolic link).

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://670323]
Approved by ikegami
Front-paged by ikegami
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (5)
As of 2022-06-30 19:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My most frequent journeys are powered by:









    Results (98 votes). Check out past polls.

    Notices?