Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re: Parsing a lot of data. Very slow. Need suggestions

by mephit (Scribe)
on Aug 15, 2002 at 21:06 UTC ( [id://190524]=note: print w/replies, xml ) Need Help??


in reply to Parsing a lot of data. Very slow. Need suggestions

Hmm, I haven't gone over this with a fine-tooth comb, but do you really need to have those `report pattern...` lines inside a for loop? The only variable inside those two lines is $date, which appears to be a function of other variables that are set outside the for loop. In other words, are you executing the exact same system call in those backticks each time through the loop? (Maybe the debugger or profiler would be able to tell you? I don't really know.) If that's the case, you might want to move those lines outside of the loop? Or maybe you have a reason for calling the same exact command several times. Or maybe I'm just missing the point completely. *shrug*

You also mentioned "SQL queries" and updating a database. There's no SQL in this script at all. Perhaps it's in one of the external programs? Regardless, if you're not using placeholders when updating your database, you may want to consider doing so. That'll save time when interacting with the database.

--

There are 10 kinds of people -- those that understand binary, and those that don't.

Replies are listed 'Best First'.
Re: Re: Parsing a lot of data. Very slow. Need suggestions
by RMGir (Prior) on Aug 15, 2002 at 23:10 UTC
    There's no SQL in this script at all.

    I wonder what

    printf("update msnNewsletter set open='%d', click='%d' where dateCreat +ed like '%d-%02d-% +02d%%';\n", $opens, $transfers,$year,$month,$day);
    is, then? :)
    --
    Mike
Re: Re: Parsing a lot of data. Very slow. Need suggestions
by vxp (Pilgrim) on Aug 16, 2002 at 12:43 UTC
    its not the same command, because the $date variable gets changed each time the process is called. in this case here, its 120 dates. 120 different commands. and then 120 more, for another similar command. they need to be in a loop. :)
      use place holders. There is a lot of overhead in your rdbms reparsing an sql statement...

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://190524]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others romping around the Monastery: (2)
As of 2024-04-24 17:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found