Re^3: Best way to store/access large dataset?

in reply to Re^2: Best way to store/access large dataset?
in thread Best way to store/access large dataset?

The ultimate goal of this will to be pulling data from a database and creating the binaries on the fly through a series of calculations, and then using this script to determine the next series of data points to pull from the database. (This serves as a filter.)

I have to ask:

"ultimate goal is pulling data from a database"? Then why were you talking about these .txt files in the OP?

"creating the binaries"? What are "binaries"?

Why pull data from a database to do "calculations" (apparently external from the db) when a database can do efficient calculations for you?

Comment on Re^3: Best way to store/access large dataset?

Replies are listed 'Best First'.
Re^4: Best way to store/access large dataset? by Speed_Freak (Sexton) on Jun 28, 2018 at 19:09 UTC
I missed this response, but I think I've answered the questions throughout the post. But if not I'll give it a shot now. The database doesn't exist yet, and I need to do the work as a proof of concept. So once the database exists, the script will be changed to point there instead of the files. Binaries are just a presence/absence representation of an attribute. They are calculated from a series of raw values by evaluating the relationships of those values a few different ways. I'm all for the database doing the calculations if it can. I'm in completely unfamiliar territory here, so recommendations are appreciated.	[reply]

Replies are listed 'Best First'.

Re^4: Best way to store/access large dataset?
by Speed_Freak (Sexton) on Jun 28, 2018 at 19:09 UTC

I missed this response, but I think I've answered the questions throughout the post. But if not I'll give it a shot now.

The database doesn't exist yet, and I need to do the work as a proof of concept. So once the database exists, the script will be changed to point there instead of the files.
Binaries are just a presence/absence representation of an attribute. They are calculated from a series of raw values by evaluating the relationships of those values a few different ways.
I'm all for the database doing the calculations if it can. I'm in completely unfamiliar territory here, so recommendations are appreciated.

[reply]

In Section Seekers of Perl Wisdom