The reason I think that SQLite would be useful is that if we want to separate the spider from indexer, finding the articles to update in the index is as simple as

SELECT * FROM ARTICLES WHERE LAST_UPDATED > $LAST_TIME_I_RAN
[download]

instead of searching the filesystem. Stored on the filesystem, we will need code to

• search,
• store, and
• update

the documents. SQLite provides all of that for free. Want to move to a different machine? -- The database is a single file. Plus, who knows what other useful things SQLite's flexibility will allow us to do?