Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

DBI execute_array

by plendid (Sexton)
on Feb 11, 2009 at 14:39 UTC ( [id://743082]=perlquestion: print w/replies, xml ) Need Help??

plendid has asked for the wisdom of the Perl Monks concerning the following question:

Hello,

I'm attempting to insert thousands of rows into a database table. Each row consists of one field. I believe that the fastest way to do this is with the DBI/execute_array function. But try as i might i can't seem to comprehend what it is that the DBI module expects.

I've attempted numerous coding permutations, googling perl execute_array, reading through the CPAN DBI documentation, Safari and ASPN. No luck. Can someone explain to me how to format the information in the following code to yield the desired result?

my @books = qw(1 2 3 4 5); my $sql = qq(insert into temp_selectedTitles values (?)); my $sh = $pics->prepare($sql); $sh->bind_param_array(1, [@books]); #this returns no rows my $tuples = $sh->execute_array({}, \@books); #this returns the error 'DBD::Oracle::st execute_array failed: 0 bind +values supplied but 1 expected...' my $tuples = $sh->execute_array({\ArrayTupleFetch=>$sh});

The desired result is five new database records,

table temp_selectedTitles book =========== 1 2 3 4 5

Thanks.

Replies are listed 'Best First'.
Re: DBI execute_array
by ikegami (Patriarch) on Feb 11, 2009 at 14:52 UTC

    Example from the docs:

    my $sth = $dbh->prepare("INSERT INTO staff (first_name, last_name) VAL +UES (?, ?)"); my $tuples = $sth->execute_array( { ArrayTupleStatus => \my @tuple_status }, \@first_names, \@last_names, ); if ($tuples) { print "Successfully inserted $tuples records\n"; } else { for my $tuple (0..@last_names-1) { my $status = $tuple_status[$tuple]; $status = [0, "Skipped"] unless defined $status; next unless ref $status; printf "Failed to insert (%s, %s): %s\n", $first_names[$tuple], $last_names[$tuple], $status->[1]; } }

    It inserts from @first_names and @last_names. You insert from @books.

    my $sth = $dbh->prepare("INSERT INTO temp_selectedTitles VALUES (?)"); my $tuples = $sth->execute_array( { ArrayTupleStatus => \my @tuple_status }, \@books, ); if ($tuples) { print "Successfully inserted $tuples records\n"; } else { for my $tuple (0..$#books) { my $status = $tuple_status[$tuple]; $status = [0, "Skipped"] unless defined $status; next unless ref $status; printf "Failed to insert (%s): %s\n", $books[$tuple], $status->[1]; } }

    Update: There's something buggy with the error reporting. The next will never execute. Does anyone know what they were trying to accomplish?

    Update: No, it could still be executed, but it still looks very odd.

Re: DBI execute_array
by runrig (Abbot) on Feb 11, 2009 at 15:21 UTC
    I believe that the fastest way to do this is with the DBI/execute_array function.
    The fastest way, since you're using Oracle, would be with sqlldr. It's more of a hassle than DBI, but a lot quicker.

    Update: Well, execute_array on Oracle is pretty darn fast. What took several minutes with single row inserts now takes a few seconds. Color me impressed. My only issue is with the return value, which doesn't seem to be correct.

    Update 2: With new library DBIx::BulkUtil even sqlldr is not a hassle anymore.

Re: DBI execute_array
by samtregar (Abbot) on Feb 11, 2009 at 16:53 UTC
    What database are you using? If it's MySQL I'm pretty sure a multi-value insert will be faster. For example, you could insert in blocks of 100:

    my $sql = "INSERT INTO temp_selectedTitles values ". join(", ", "(?) +" x 100); my $sth = $dbh->prepare($sql); for (my $i = 0; $i <= $#books; $i+=100) { $sth->execute(@books[$i..$i+99]); }

    Of course if you don't have an exact number of blocks then you have to clean up what's left with another statement, either one at a time or custom fitted. I use this technique all the time and it always beats going row by row.

    I think execute_array() is mostly abandoned at this point - I can't remember the last time I saw it used outside the DBI docs.

    -sam

      Good trick! Except there should be an extra set of parenthesis in your join statement in order for the 100 question elements to be passed to the join function as a list :

       my $sql = "INSERT INTO temp_selectedTitles values ". join(", ", ("(?)") x 100);

      Also, because the example used only one element in each row, it was not quite clear to me that the above example did not just create a single row with 100 columns (it does not). It was also unclear from the example how you might apply this concept when inserting rows that have multiple columns. After playing around with it a little, I was able to get something like the following to work:

      $dbh->do("CREATE TABLE IF NOT EXISTS temp_selectedTitles (title VARCHA +R(128), author VARCHAR(128), location VARCAR(128))"); my $number_of_rows_waiting_to_commit = 0; my @chunk_of_values_for_database = (); foreach my $book (@books) { push (@chunk_of_values_for_database, $book); push (@chunk_of_values_for_database, $authors{$book}); push (@chunk_of_values_for_database, $locations{$book}); $number_of_rows_waiting_to_commit++; if ($number_of_rows_waiting_to_commit >= 100){ my $sql = "INSERT IGNORE INTO $DT_table (title, author, locati +on) VALUES ".join(", ", ("(?, ?, ?)") x $number_of_rows_waiting_to_co +mmit); my $store_1 = $dbh->prepare($sql); $store_1->execute(@chunk_of_values_for_database); $number_of_rows_waiting_to_commit = 0; @chunk_of_values_for_database = (); } } my $sql = "INSERT IGNORE INTO $DT_table (title, author, location) VALU +ES ".join(", ", ("(?, ?, ?)") x $number_of_rows_waiting_to_commit); my $store_2 = $dbh->prepare($sql); $store_2->execute(@chunk_of_values_for_database);

      Note that I actually tested this on a table with 6 columns and inserted 10,000 rows at a time instead of 100. It probably will work for more than that, but there may be limits to the size of the $sql string that you pass to the DBI and that in turn gets passed to whatever database you are using. Also, too large of a string could potentially cause a penalty in performance. But from what I've found, this trick does indeed speed up the process tremendously.

Re: DBI execute_array
by tweetiepooh (Hermit) on Feb 11, 2009 at 17:37 UTC
    The following code segment works on my Oracle 9.2 setup
    # table books - title char(10) my $dbh; connectdb(); # sets $dbh use your connection here, I have AutoCommit= +>0 my $sth = $dbh->prepare("insert into books values (?)"); my @val = qw(one two three four five); $sth->bind_param_array(1,\@val); my $tuples = $sth->execute_array({ArrayTupleStatus=>\my @tuple_status} +); print "Tuples = $tuples\n"; $dbh->commit(); $dbh->disconnect(); sub connectdb { $dbh = DBI->connect(); }
    Running this gives
    Tuples = 5 sqlplus user/pass SQL> select * from books TITLE ----------- one two three four five
    I think I have typed the code OK, different machine to terminal attached to Unix/Oracle.
Re: DBI execute_array
by roboticus (Chancellor) on Feb 11, 2009 at 20:39 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://743082]
Approved by eff_i_g
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others contemplating the Monastery: (1)
As of 2024-04-19 00:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found