I've worked on spiders that have used the file system, and spiders that have used databases. It's certainly cheaper to use the file system. But thinking about the size of the dataset, we can easily afford to put these records into a database. There are only c. 600,000 records, and they're small -- not even full web pages. I like the idea of using SQLite.