Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

dbi: moving big data among databases (prepared statements)

by leostereo (Beadle)
on Jun 16, 2018 at 17:43 UTC ( [id://1216786]=perlquestion: print w/replies, xml ) Need Help??

leostereo has asked for the wisdom of the Perl Monks concerning the following question:

Hi guys , I need to move big databa among databases. I have been reading about improving performance with mysql prepared statements. So far I dont understand why it is better neither how to use it.
Lets suppose I have the following "bring and insert data" procedure:

#!/usr/bin/perl use strict; use warnings; require DBI; ### #variables here my @big_data_array; ### foreach $id (@big_data_array){ $query1 = "SELECT * from database1.table WHERE id = '$ +id' my $sth1 = $dbh1->prepare($query1); $sth2->execute(); while (my @row = $sth1->fetchrow_array ) { $val1 = $row[0]; $val2=$row[1]; } $query2 = "insert into database2.table ('value1','valu +e2')". " VALUES('$val1','$val2')"; my $sth2 = $dbh2->prepare($query2); $sth2->execute(); }

Question is how to achieve same result using prepared statements ?
And why is it better ?
I will try to find more info about it. Regards, Leo.

Replies are listed 'Best First'.
Re: dbi: moving big data among databases (prepared statements)
by atcroft (Abbot) on Jun 16, 2018 at 18:25 UTC

    It is my understanding that the purpose of preparing the statement is to reduce the work you are done each iteration. As written, you are preparing the statement each time, which will add a little time to each pass. Perhaps a better way to do this would be:

    # Recommend not using '*' but specifying the field names, # to future-proof code. my $query1 = "SELECT * FROM database1.table WHERE ( id = ? );"; my $query2 = "INSERT INTO database2.table ( value1, value2 ) VALUES ( ?, ? ) ;"; my $sth1 = $dbh1->prepare( $query1 ) or die $dbh1->errstr; my $sth2 = $dbh2->prepare( $query2 ) or die $dbh2->errstr; foreach my $id ( @big_data_arrray ) { # Original code had an error # $sth2->execute was called in both cases $sth1->execute( $id ) or die $dbh1->errstr; while ( my @row = $sth1->fetchrow_array ) { $sth2->execute( @row[0 .. 1] ) or die $dbh2->errstr; } $sth1->finish; }

    Instead of performing 2*N (N=number of items in @big_data_array) prepare calls, only 2 prepare calls are used.

    Hope that helps.

      Following up: Every time your 'select *' gets compiled, the DB has to query the metadata to expand the '*'. A smart database will cache this information, but the work still has to be done. If it's a lot of data and a slow link, a colleague and I found that using prepared statements (prepare_cached(), actually) reduced a database query that took more than a day (to move 24 hours' data) to one that took an hour or so.

      There is another reason to use prepared queries: they are resistant to SQL injection attacks. I would post a link, but Perl Monks censored it. Google "Bobby Tables" for what I mean.

Re: dbi: moving big data among databases (prepared statements)
by AnomalousMonk (Archbishop) on Jun 16, 2018 at 18:20 UTC

    In the statement
        $query1 = "SELECT * from database1.table WHERE id = '$id'
    the quoted string is not terminated and the statement is not complete. How did this compile in the first place? Perhaps see Short, Self-Contained, Correct Example for another approach to preparing code for discussion.


    Give a man a fish:  <%-{-{-{-<

Re: dbi: moving big data among databases (prepared statements)
by nikosv (Deacon) on Jun 16, 2018 at 22:22 UTC
    is bulk offloading from database A and bulk uploading to database B not feasible?
Re: dbi: moving big data among databases (prepared statements)
by clueless newbie (Curate) on Jun 17, 2018 at 12:51 UTC
    deleted as it is essentially duplicates a previous post.
A reply falls below the community's threshold of quality. You may see it by logging in.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1216786]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others taking refuge in the Monastery: (6)
As of 2024-04-19 09:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found