hi everyone,

i'm working on a search engine project and i am using my file system & text files as part of the search engine's database.

i have been using Tie::File to access these DB text files directly (if i understand correctly, using TIE does not pull the whole text file into memory, thus saving time & memory when working with really large files??)

i have also been using Compress::Zlib to compress large chunks of data to inside the DB text files ... each text file in our DB is stored as an array and looks like this:

numeric id | plain text data | zipped big chunk of data \n
numeric id | plain text data | zipped big chunk of data \n

i'm thinking this is useful because i can TIE to the file, pull any array element, then uncompress the zipped data to use.

now my question is two fold ...

1) is this a good srategy to manage large volumes of data or am i missing something?

2) i would also like to compress each DB text file (in addition to the already zipped chunks of data in each file), but i can't figure out how to TIE into the zipped file without pulling the whole text file into memory.

hope this makes sense ... needless to say, i'm a big fat newbie when it comes to PERL.

Shane