Re^2: How to improve MYSQL search performance of perl?

Hi,

As I need to read line by line and search them in the database, I used a subroutine to handle all database work. Below is my code.

sub search{
#take search parameter from html <form/>
my $q = $_[0]; #taking param;
my $found = 0; #initialize category found count;
#open database
my $ref = {RaiseError => 1, AutoCommit => 0};
my $dbh = DBI->connect('DBI:mysql:diet', $ref) || die "Failed to connect: $DBI::errstr";
#prepare SQL search statement
my $sql = qq{select topic FROM table1 WHERE uri LIKE '$q'};
my $sth = $dbh->prepare($sql);
$sth->execute();
while(my $record = $sth->fetchrow_hashref) {
no warnings;
print &topic($record->{topic}), "
\n";
$found++;
}
if ($found == 0){ #if no category was found, output no found
print "

$q is not found in the database!

\n";
}
$sth->finish();
$dbh->disconnect(); #disconnect from database;
}

thanks again,

Nan

Comment on Re^2: How to improve MYSQL search performance of perl?

Replies are listed 'Best First'.
Re^3: How to improve MYSQL search performance of perl? by davidrw (Prior) on Aug 19, 2005 at 17:00 UTC
I would take trammell's suggestion a step further and also not recreate the statement handle every time and actually take advantage of the statement handle (and placeholders)-- i think this will have a decent improvement in the performance (amount of gain is probably db-dependent): my $dbh = DBI->connect('DBI:mysql:diet', {RaiseError => 1, AutoCommit +=> 0} ) \|\| die "Failed to connect: $DBI::errstr"; my $sth = $dbh->prepare( qq{select topic FROM table1 WHERE uri LIKE ?} + ); search($sth, 'foo'); search($sth, 'bar'); $sth->finish(); $dbh->disconnect(); #disconnect from database; sub search{ my $sth = shift; # require statement handle (this could probably be + a global var instead if desired) my $q = shift; # take search parameter from html <form/> my $found = 0; #initialize category found count; $sth->execute($q); my $rows = $sth->fetchall_arrayref( {} ); printf "%d rows found for '%s'.\n", scalar(@$rows), $q; foreach my $row (@$rows){ printf " Topic: %s\n", &topic($row->{topic}); } } [download]	[reply] [d/l]
Re^4: How to improve MYSQL search performance of perl? by nan (Novice) on Aug 24, 2005 at 17:06 UTC
Hi, Many thanks for the code as I learned a lot from you. But even opened database once, the search is still slow (running 200 queries one by one) and I always recieve Internal Server 500 Error after a long waiting and can't see any results back to screen (before modifying, I could see a few results back as the database open and close everytime). I'm not sure it is due to the configuration of apache server or MySQL database so I'm rebuilding the database now to see if it helps. thanks again, Nan	[reply]
Re^5: How to improve MYSQL search performance of perl? by davidrw (Prior) on Aug 24, 2005 at 17:17 UTC
Can you post your revised code? Are you re-using the statement handle? What's the error in the error_log (or if you run on command line)? Also, try adding `$\|=1;` in your script to make sure it's not buffereing so yuo'll see output as it's happening. If the loop to execute the queries is good now (resuing handles, etc), then this probably just goes back to the issue mentioned elsewhere in this thread about SQL performance .. the speed will depend entirely on your SQL, table schema, and indexes (can you provide any of those?).	[reply] [d/l]
Re^6: How to improve MYSQL search performance of perl? by nan (Novice) on Aug 25, 2005 at 16:01 UTC
Re^7: How to improve MYSQL search performance of perl? by davidrw (Prior) on Aug 25, 2005 at 17:44 UTC
Re^3: How to improve MYSQL search performance of perl? by trammell (Priest) on Aug 19, 2005 at 15:55 UTC
One improvement you can make is to only open your database handle once at the beginning of the script, and reuse that handle instead of recreating it for each query.	[reply]


Perl: the Markov chain saw
	PerlMonks