Greetings Monks!

I'm a sysadmin who runs a few redundant web servers. I've created a lot of tools and systems to help me manage my beasties, and I'm trying to create an overall manager of them.

The key is, I need it to be super-uper-duper reliable, so I need to keep track of what I'm doing, what I've done and what I plan to do. I need this information to be written atomically and I want to be able to look back at previous state. The idea being, the machine could crash at any time in any sequence and I want to be able to piece things togeather. That will actually be easier than it sounds, but right now I'm planning this atomic history config thingy...

Here's an example: Say a server is in ACTIVE mode and I want to transition to RESERVE mode. I'd want to write down somewhere something like:

STATE=ACTIVE
CHANGING_TO=RESERVE
CHANGE_STEP=TELL_OTHER_SERVER
[download]

...as you can see, if the server crashed in the middle of this, I'd need this file to be accurate enough to piece things togeather and finish the job.

Okay, here's the actual question: Has this or a large piece of this already been done? I don't want to re-invent the wheel.

I was thinking I'd write a JSON/YAML blob to a file as the "current" state and keep a log of all previous states as a file containing a whole bunch of these JSON blobs. But if someone's already written something that does what I want, I'd rather go with that.

I was also thinking this sounds a lot like a transactional database with transaction history BUT I really don't want to do that because this needs to be able to work even if major components, like a database server, are down.

Okay, hopefully that makes sense. I'm looking for something to atomically store arbitrary data and also have a history of all previous revisions (hey, maybe a versioning system should come into this...). I just want to avoid re-inventing the wheel.

Thanks!
Pileofrogs