After a good example of efficient record handling...

NeilF has asked for the wisdom of the Perl Monks concerning the following question:

My current forum uses flat text files and reads/writes the entire file when adding a new message.

I'd like to look into using fixed length files for indexes etc for faster and more efficient data handling.

I've searched, googled and even downloaded example forums, but cannot see examples.

Can anyone point me to some example of good record handling - eg using seek etc to only update the records required. If its code specifically for a forum all the better!!!

Edited 1 Jan 2006, by footpad: Added basic formatting tags.

Comment on After a good example of efficient record handling...

Replies are listed 'Best First'.
Re: After a good example of efficient record handling... by chromatic (Archbishop) on Jan 01, 2006 at 21:49 UTC
Are you sure you want to use fixed-length record sizes for forum posts? Do you have a small maximum length for the body of the post? If not, all of your records will be the size of the largest possible record -- and if that's not small, you'll have to do a lot of IO to read each message from the disk, pull it through the cache, and get it where `perl` can handle it. If you really think that will improve your speed (and I don't, even without benchmarking it), you need some sort of unique ID per message which you can use as an offset into the file. Multiply that by the size of a record and use `seek` and `sysread` on a filehandle to extract just the record you want. If that doesn't seem fun to you (and it's not), you could use a unique record separator between records and let Tie::File take care of reading and writing. If this were my project, I would use DBD::SQLite and avoid all of the tedious mucking about with O(n) access and let indexed columns get that down to O(log n) or better, while not having to process all of the data myself. Please consider that instead.	[reply] [d/l] [select]
Re: After a good example of efficient record handling... by holli (Abbot) on Jan 01, 2006 at 23:37 UTC
Taking your performance/io problems in Perl always reads in 4K chunks and writes in 1K chunks... Loads of IO! into account, i think you could use a flat file for storing post titles and a reference to a file that contains the actual post. holli, /regexed monk/	[reply]
Re^2: After a good example of efficient record handling... by NeilF (Sexton) on Jan 01, 2006 at 23:47 UTC
I agree, but I can see lots of problems/unkowns with this and would like to see and article or even better a working example...	[reply]
Re: After a good example of efficient record handling... by TedPride (Priest) on Jan 02, 2006 at 09:23 UTC
See pack and unpack for record handling; sysopen, sysseek, sysread, syswrite for retrieving or writing records; and flock for keeping the files from being modified during writes. As mentioned above, post contents should be in a separate file, with the post record containing just start and end offsets. This keeps your records neat and fixed-length. If your users are allowed to do edits, you should allocate a few extra characters for post content so small increases don't require moving it to a new spot in the file. After the time limit for edits is reached, your clean-up utility can rewrite the content to exact size. It would, however, be simpler to just do this with mySQL. Certain things like indexes can get quite complicated to program.	[reply]
A reply falls below the community's threshold of quality. You may see it by logging in.


Think about Loose Coupling
	PerlMonks