ETL? And there are a core group of files that will be repeatedly analyzed. But the overall sets change. So files can be added or removed from the calculations as needed.
Each record has 3 "columns" of data with a million rows per column. There are a couple other static values that are a single value. I believe that is one to many? And the samples can be grouped by another singular static value stored with the record. (The shape identifier.)
I'm pretty lost when it comes to the database stuff, so I'm going to point my colleagues here and see what they say honestly!