Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

Background

I am working in a small research project with limited resource, where a lot of data (measured and calculated values) need to be handled. Data comes from scientific experiments and are usually first stored in Excel spreadsheets. One spreadsheet contains data from a few experiments. For each experiment there is typically some basic information (50 data items) and a number of time series of measured values (10 series, 100 points in time and 30 measured values for each time). There exists about 100 old spreadsheets and some hundred new will be created. The old spreadsheets are similar but not standardised.

Representation of a quantity

A quantity is a property that is measured. Example: mass, length, time. A unit is a standard quantity against which a quantity is measured. Example: gram, metre, second; which are units of the above quantities.

This can also be described by:

A = {A} * [A]
A is the symbol for the quantity, {A} symbolizes the numerical value of A, and [A] represents the corresponding unit. (e.g., A = 300 * m = 0.3 * km). {A} is often called the measured value.

Example of thing need for a quantity are:
  • Value
  • Unit
  • Property measured/calculated (temperature of air)
  • Measuring/calculation method
Representation of unit

The Unit must be represented in a consistent way. For Units from The International System of Units (SI) can the exponents of the base units be used. SI has 6 base units (meter [m], kilogram [kg], second [s], … ). The derived units can be expressed using exponents of the base units (area [m2], speed [ms-1]).

Goals and restrictions

Primary goals are:
  • In a consistent way store the data from the experiments
  • Make it possible for Perl programs to use the stored data
  • Make it possible for Perl programs to create new sets of stored data
  • Make the data available to JavaScript programs

MS Window systems are used.

Design ideas

  • Use one text file to represent data from one experiment. The text files are stored in a directory structure in a normal file system.
  • For each spreadsheet there is one file associated, containing the additional information needed to create “standardised” representation from a spreadsheet. A Perl scripts is used to create the “standardised” representation.
  • A temporary database is created for each purpose. (I do not think it is possible to have one on-line database with everything.) A Perl script load the temporary databases from a number of “standardised” files.
  • A database implementation which can provide accessed to Perl and JavaScript program is selected.
  • Make it possible from Perl to create new or augment “standardised” files. (Write to the database and then create “standardised” files with the new and updated data.)

Questions

  • Are there any similar Perl-based system already implemented?
  • Is there a better design strategy? What should be changed?
  • What type of database can be used? Is MongoDB possible and suitable?
  • Suitable format for the “standardised” files.
  • Perl modules for handling SI units and conversion between such unit.

In reply to Strategy for simple data management by bojinlund

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (6)
As of 2022-05-24 12:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you prefer to work remotely?



    Results (82 votes). Check out past polls.

    Notices?