http://qs321.pair.com?node_id=1217234


in reply to Re: Best way to store/access large dataset?
in thread Best way to store/access large dataset?

SQLite is the last thing that should come to mind when dealing with large datasets. I can't imagine why you would even think about using SQLite over Postgres, or depending on your needs, loading it up in to S3 and querying it using Athena, or Dynamodb. There's a wealth of technology out there made specifically for processing vast amounts of data and running calculations on it, for relatively cheap. SQLite is not one of them.

Three thousand years of beautiful tradition, from Moses to Sandy Koufax, you're god damn right I'm living in the fucking past