Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

Re^3: Best way to store/access large dataset?

by erix (Parson)
on Jun 22, 2018 at 19:53 UTC ( #1217254=note: print w/replies, xml ) Need Help??

in reply to Re^2: Best way to store/access large dataset?
in thread Best way to store/access large dataset?

The ultimate goal of this will to be pulling data from a database and creating the binaries on the fly through a series of calculations, and then using this script to determine the next series of data points to pull from the database. (This serves as a filter.)

I have to ask:

"ultimate goal is pulling data from a database"? Then why were you talking about these .txt files in the OP?

"creating the binaries"? What are "binaries"?

Why pull data from a database to do "calculations" (apparently external from the db) when a database can do efficient calculations for you?

  • Comment on Re^3: Best way to store/access large dataset?

Replies are listed 'Best First'.
Re^4: Best way to store/access large dataset?
by Speed_Freak (Sexton) on Jun 28, 2018 at 19:09 UTC

    I missed this response, but I think I've answered the questions throughout the post. But if not I'll give it a shot now.

    The database doesn't exist yet, and I need to do the work as a proof of concept. So once the database exists, the script will be changed to point there instead of the files.
    Binaries are just a presence/absence representation of an attribute. They are calculated from a series of raw values by evaluating the relationships of those values a few different ways.
    I'm all for the database doing the calculations if it can. I'm in completely unfamiliar territory here, so recommendations are appreciated.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1217254]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (6)
As of 2020-07-06 12:57 GMT
Find Nodes?
    Voting Booth?

    No recent polls found