If you have enough memory, you could write a daemon process that would hold the entire dataset in memory. Other processes could communicate with the daemon process to ask questions. Have the daemon process periodically re-read the data from disk to bring in the changes. That way you could get even higher performance than the custom flat-file approaches by avoiding all disk IO.

L