How about making sure you create a unique key for each point, then serializing the data points (Freeze() is one way, Conway's OOP book lists several), and shoving them in a tied DBM structure?

The caveat here is making sure you know WHICH DBM your product will be using: "Bucket-size" limitations vary greatly, and with 1000 objects, if you are using (for example) the "standard" DBM (nDBM, IIRC) on Solaris 2.6, you're going to either have 65-byte objects or run out of space; other DBM implementations have much larger bucket-size limitations, and at least one is much smaller.

The big advantages of DBM are that it's easy to implement/maintain (there's even the explicit-but-deprecated dbmopen() command, if tie() is too heady for your future maintainers), data manipulation is easy (just regular hashes), and it's fairly quick (YMMV, again depending on which version of DBM you're using).

The big drawback (apart from the bucket size problem described above) is that there is no internal locking mechanism in the DBM specification; if you need locks (will multiple scripts be rewriting the same object simultaneously?), you have to implement them yourself.

That said, there are some other games you can play to push the performance even higher. One is that if you have a ramdisk, or a system that supports a shared swap//tmp filesystem (many commercial versions of Unix will overlay swap space with the /tmp filesystem -- if the machine is not swapping, /tmp is memory-mapped; if it is, swap data starts getting written into /tmp), you can put the DBM in the memory-mapped space (good idea, though, to copy the file somewhere more static every so often as a backup), so that reads/writes with the tied DBM never actually touch disk.

As far as optimizing the CGI (and serializing the DBM connections), one approach might be to have a separate daemon which babysits the DBM, and communicates back and forth with the CGI instances via named pipes. This way, you wouldn't have to worry about the overhead of the CGI actually opening the DBM each time.

Of course, you never specified what platform you were implementing this on. All of my advice with this goes for naught if you aren't on a Unix or Linux (or OS X??) system.

Spud Zeppelin * spud@spudzeppelin.com