http://qs321.pair.com?node_id=1177346


in reply to Dynamic Config

Well, there is one claim from the documentation that probably serves as a primary motivation for the creation of the project:

Being able to change your configuration without bringing the service down is extremely important for web apps.

I don't find that to actually be true. Because a well-designed service sits behind some type of load balancer (even just a reverse proxy) and is designed such that you can restart every instance of the service without impacting any users of the full service.

But there can certainly be cases where it would be convenient to push configuration changes without requiring a restart even if the down-side of a restart is only that performing it is inconvenient for your staff for some reason.

I've worked on a lot of services that supported several different types of dynamic configuration. None of those were particularly difficult to implement, except for the most elaborate one which provided a telnet-like interface to each running service, which only made sense because these services were using a lot of threads. I'm not sure if that interface was ever used in Production even once. It had the potential to be a powerful tool, but such complex tools rarely find the confluence of a problem that can be solved by them along with a person working on the problem who sees how to solve the problem in that complex way "better" than more mundane techniques.

How were all of the simple dynamic configuration facilities implemented? Well, first you need to consider the traps in dynamic configuration.

If a configuration setting changes, then you risk having two parts of the system working with different values for that setting. Such can actually break parts of the service. So, if you are very careful, then you can write your code such that the value for a setting is fetched once at the start of the section of code where the value for that setting should remain constant. And then also be careful to not cache that setting too long such that a configuration change never fully takes effect (or can just take a very long time to fully take effect).

Requiring your programmers to be very careful is usually a good way to introduce a lot of bugs.

I find that many configuration changes just naturally only impact things when initialization is done. For example, if you have a configuration for the duration of the idle time-out when talking to some related service, then that time-out might be specified when you construct your connection to that service. So, your dynamic configuration system would have to somehow trigger the tear-down and re-establishment of such connections when a configuration change gets pushed (or trigger different code that can adjust the time-out of existing connections).

That is why the dynamic configuration systems that I've worked with usually either only deal with a tiny subset of configuration settings (none of which have such traps associated with them) or achieve dynamic configuration by having the worker process get recreated when a configuration change needs to take effect.

For the second case, the effect is only subtly different from the "restart the service to update the configuration model". Instead of the push process telling the instance of the service to restart, each worker notices (only at an appropriate point) that the configuration has been updated and restarts itself.

And "notice a configuration change" was almost always just noting that stat indicates that some file has been updated.

Even if you put your configuration in a git repo or in some database, there certainly are some advantages to pushing that configuration information to a particular server as a simple flat file that is trivial to parse (JSON being the best answer for that). Your database or git server needing maintenance should not lead to your service having to enter a maintenance window as well. Such maintenance should only impact your ability to automatically push changes to your configuration.

Some of you may be concerned that this only prevents mis-matched configuration within a process. But, if you have something that two communicating processes must agree upon, you shouldn't be getting that value separately from configuration in each process. You should have one process telling the other process what value to use.

- tye