Re^2: Atomic Config Updater with History?

in reply to Re: Atomic Config Updater with History?
in thread Atomic Config Updater with History?

All of what BrowserUk says is true, although most journaled filesystems will limit your liability when using regular files. Couple this with (at least on Linux) sync-ed writes (which you *don't* want to do a lot of, as they are dreadfully slow), and you might get by. A transactional DB is better, but you do have to remember that the two-phase commit is designed to ensure that multiple operations on the DB itself are either all done, or not done (ie rolled back). When part of what you are trying to 'commit' has nothing to do with the database (ie, transition a server to a new state), then you are still not atomic. In BrowserUk's example, if you

INSERT the command msg
perform the command
commit the INSERT

but the system crashes before step 3 completes, the DB will rollback the INSERT, but the DB has no knowledge of the command you performed. You would have to take the additional step of looking at the DB's transaction log (which many DBs allow you to record in a readable format). Upon crash recovery, if you see a 'command rollback', you would want to check the state of the execution of that command, and try to 'roll that back' too...

fnord

Comment on Re^2: Atomic Config Updater with History?

Replies are listed 'Best First'.
Re^3: Atomic Config Updater with History? by BrowserUk (Patriarch) on Mar 26, 2011 at 17:54 UTC
Sync'd writes (and write-thru) can be slow due to the absence of caching, but so are most journalled file-systems. When part of what you are trying to 'commit' has nothing to do with the database (ie, transition a server to a new state), then you are still not atomic. In BrowserUk's example, Agreed. That example only works if repeating the performed, but unlogged command over is effectively a noop. Mind you, breaking processing up into steps such that any given step can be repeated 2 or more times without affecting the overall result is something of a black art in itself. The basic steps are: a) don't discard source data for a given step, until the output data for that step has been successfully processed by the next step. b) discard any source data for this step that is 'incomplete'. Sentinel values are useful for this c) Once the input data for this step--ie. the output of the previous step--has been successfully processed, delete the associated input data to the previous step. Of course, in critical systems, 'delete' is probably spelt 'move to archive'. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]

Replies are listed 'Best First'.

Re^3: Atomic Config Updater with History?
by BrowserUk (Patriarch) on Mar 26, 2011 at 17:54 UTC

Sync'd writes (and write-thru) can be slow due to the absence of caching, but so are most journalled file-systems.

When part of what you are trying to 'commit' has nothing to do with the database (ie, transition a server to a new state), then you are still not atomic. In BrowserUk's example,

Agreed. That example only works if repeating the performed, but unlogged command over is effectively a noop.

Mind you, breaking processing up into steps such that any given step can be repeated 2 or more times without affecting the overall result is something of a black art in itself. The basic steps are: a) don't discard source data for a given step, until the output data for that step has been successfully processed by the next step. b) discard any source data for this step that is 'incomplete'. Sentinel values are useful for this c) Once the input data for this step--ie. the output of the previous step--has been successfully processed, delete the associated input data to the previous step. Of course, in critical systems, 'delete' is probably spelt 'move to archive'.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

[reply]

In Section Seekers of Perl Wisdom