Background
I am working in a small research project with limited resource, where a lot of data (measured and calculated values) need to be handled. Data comes from scientific experiments and are usually first stored in Excel spreadsheets. One spreadsheet contains data from a few experiments. For each experiment there is typically some basic information (50 data items) and a number of time series of measured values (10 series, 100 points in time and 30 measured values for each time). There exists about 100 old spreadsheets and some hundred new will be created. The old spreadsheets are similar but not standardised.
Representation of a quantity
A quantity is a property that is measured. Example: mass, length, time.
A unit is a standard quantity against which a quantity is measured. Example: gram, metre, second; which are units of the above quantities.
This can also be described by:
A = {A} * [A]
A is the symbol for the quantity, {A} symbolizes the numerical value of A, and [A] represents the corresponding unit. (e.g., A = 300 * m = 0.3 * km).
{A} is often called the measured value.
Example of thing need for a quantity are:
-
Value
-
Unit
-
Property measured/calculated (temperature of air)
-
Measuring/calculation method
Representation of unit
The Unit must be represented in a consistent way. For Units from The International System of Units (SI) can the exponents of the base units be used. SI has 6 base units (meter [m], kilogram [kg], second [s], … ). The derived units can be expressed using exponents of the base units (area [m2], speed [ms-1]).
Goals and restrictions
Primary goals are:
-
In a consistent way store the data from the experiments
-
Make it possible for Perl programs to use the stored data
-
Make it possible for Perl programs to create new sets of stored data
-
Make the data available to JavaScript programs
MS Window systems are used.
Design ideas
-
Use one text file to represent data from one experiment. The text files are stored in a directory structure in a normal file system.
-
For each spreadsheet there is one file associated, containing the additional information needed to create “standardised” representation from a spreadsheet.
A Perl scripts is used to create the “standardised” representation.
-
A temporary database is created for each purpose.
(I do not think it is possible to have one on-line database with everything.)
A Perl script load the temporary databases from a number of “standardised” files.
-
A database implementation which can provide accessed to Perl and JavaScript program is selected.
-
Make it possible from Perl to create new or augment “standardised” files.
(Write to the database and then create “standardised” files with the new and updated data.)
Questions
-
Are there any similar Perl-based system already implemented?
-
Is there a better design strategy? What should be changed?
-
What type of database can be used?
Is MongoDB possible and suitable?
-
Suitable format for the “standardised” files.
-
Perl modules for handling SI units and conversion between such unit.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|
