http://qs321.pair.com?node_id=471887

tomazos has asked for the wisdom of the Perl Monks concerning the following question:

I think a Perl Typeless Autovivifying Database would be cool.

The idea is to try and optimize for ease of use, by trading off performance and reliability. (Update: Reliability in the sense that: the extent to which a developer or user error is caught quickly before it causes more trouble in the system. Is there a better word for this?) Let me put this out there and see what you think.

The relevant points about it as compared to an SQL database: (1) It's typeless. Any scalar, of any length, can be stored in any column. (2) Tables and Columns are automatically created as needed on the fly when they are first used (like perl hash keys). ie No writing CREATE TABLE definitions or ALTER TABLEs. Other than that it is equivilant to DBI::(MySQL|Oracle|*SQL).

Maybe it could be built on top of DBI and MySQL somehow?

use PerlTypelessAutovivifyingDB; my $dbh = PerlTypelessAutovivifyingDB->opendb() or die $!; # open mast +er database file, location set on PerlDB module install my $db = $dbh->db('animals'); # use database 'animals', create it if i +t doesn't exist my $table = $db->table('cats'); # use table 'cats', create it if it do +esn't exist $table->key('name'); # alter table to make column 'name' a unique key, + permanently my @cats = ( { 'name' => 'lucy', 'age' => 2, 'species' => 'lion' }, { 'name' => 'mary', 'age' => 5, 'species' => 'lion' }, { 'name' => 'cindy', 'age' => 10, 'species' => 'tiger' } ) foreach my $cat (@cats) { $table->set(%$cat); # insert or update entry represented by hash i +nto table. # any non-existant columns are autovivified in + the table like in hashes # IF hash assigns a unique key column an exist +ing value # update that entry # ELSE # insert new entry, set any key columns to +a new unique value (eg first available integer) } my $sth = $table->select { $a{'species'} eq 'lion' } ('name', 'age'); +# prepare and execute select name and age where species is lion; foreach (my ($name, $age) = $sth->fetchrow_array()) { print "$name ($age years), "; # print 'lucy (2 years), mary (5 ye +ars)' } $sth->finish(); my $joinedtable = $db->table { $a{'cats.name'} eq $b{'kittens.mothers +name'} } ('cats','kittens'); # join two tables on cats.name == kitten +s.name my $sth2 = $table->select ('cats.name', 'kittens.name'); # default to +leftmost table name in column name conflict foreach (my ($name, $kitten) = $sth2->fetchrow_array()) { print "$name has a kitten $kitten. "; } $sth2->finish(); # database automatically closes itself on block exit

Andrew Tomazos  |  andrew@tomazos.com  |  www.tomazos.com

Replies are listed 'Best First'.
Re: Perl Typeless Database
by ikegami (Patriarch) on Jul 02, 2005 at 07:19 UTC
Re: Perl Typeless Database
by davidrw (Prior) on Jul 02, 2005 at 13:37 UTC
    It seems like DBD::AnyData's "Working with in-memory tables" section will pretty much give a typeless database (though no idea performance-wise), and has the huge advantage of working under DBI and Class::DBI. Does that satsify your requirements?
Re: Perl Typeless Database
by zentara (Archbishop) on Jul 02, 2005 at 12:52 UTC
      That link is for version 2. From the FAQ:
      • datatypes in SQLite version 2.8
        SQLite is "typeless".
      • version 3.0
        Version 2 of SQLite stores all column values as ASCII text. Version 3 enhances this by providing the ability to store integer and real numbers in a more compact format and the capability to store BLOB data.
        Version 2 of SQLite stores all column values as ASCII text. Version 3 enhances this by providing the ability to store integer and real numbers in a more compact format and the capability to store BLOB data.

        Yes but, unless you enable the strict affinity mode, any column (apart from integer primary keys) can be used to store any type - just like V2. So unless you want typing-by-column you don't have to have it.

        It's just like Perl and has value-based typing rather than container based typing (unless you ask for it), V3 just has more effective representations at the back end.

Re: Perl Typeless Database
by CountZero (Bishop) on Jul 02, 2005 at 09:07 UTC
    I can understand that you want to trade off speed for ease of use, but giving away reliability? If the database is not reliable, what use can it be?

    CountZero

    "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

      Nothing is 100% reliable. Things can still be useful though. When a baseball player goes into bat, does he expect to hit a home run every time? Of course not.

      Don't like that example? How about UDP. When you send a UDP packet across the network it is not guaranteed to arrive. The protocol has still been used for many things for a long time.

      -Andrew.


      Andrew Tomazos  |  andrew@tomazos.com  |  www.tomazos.com

        I would not want any of my database programs to have a data collection average of even a Hall of Fame baseball player. A Ted Williams program, for example, would get its data less than 40% of the time.

        UDP is, in of itself, unreliable. But programs using it are not necessarily so. Would you use unreliable NFS? I doubt it, but NFS uses UDP. But it also uses RPC, which has a request/reply protocol that allows for retransmissions should a reply not arrive within a specified window. Other UDP applications (e.g., video or audio streaming) may allow a certain level of loss because they know that some percentage of dropped packets will be undetectable by the human using the application. The key is that they have methods they can use within, or layered above, the UDP protocol (e.g., sequence numbers) that allow the receiver to monitor and detect the (un)reliability of the underlying UDP protocol.

        Reliability/speed tradeoffs are acceptable for services but the (un)reliability has to be detectable.

        True, all to true, but when I put something in a database, I expect to find it back. It is bad enough as it is that probably no complex piece of software is without bugs, but deliberately adding unreliability is too much if you ask me.

        CountZero

        "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

Re: Perl Typeless Autovivifying Database
by redhotpenguin (Deacon) on Jul 03, 2005 at 06:15 UTC
    Your requirements sound like you might want to look at using a native xml database to store your information in xml. Bear in mind though that just as there are issues with strongly typed databases there are also issues with storing weakly typed data.
Re: Perl Typeless Autovivifying Database
by mattr (Curate) on Jul 04, 2005 at 05:17 UTC
    I have not used it but I believe Firebird/Interbase is typeless (also see page of its main developer IBPhoenix). There is also apparently an OO thing called IBPerl, no longer maintained but GPL. FYI.