Re: Super Duper Inefficient

here's a clue:

foreach $server (@servers) {
  chomp $server;
  open TMP, ">/tmp/$server.tmp";
  $ENV{"FILENAME"} = "/tmp/$server\.tmp";
  foreach(@DATA) {
    ($hostid, $time, $get, $post, $currconn) = split(/\t/);
    next if ($hostid ne $SERVER{$server});
    ## more here...
  }
  close TMP;
}
[download]

this is the inefficiency. invert these loops to make this something like:

## iterate through record
foreach(@DATA) {
  ($hostid, $time, $get, $post, $currconn) = split(/\t/);
  ## match against the server list and process accordingly
  foreach $server (@servers) {
    chomp $server;
    next unless $hostid eq $SERVER{$server};

    open TMP, ">>/tmp/$server.tmp";
    $ENV{"FILENAME"} = "/tmp/$server\.tmp";

    ## more here...

    close TMP;

    ## done with this record, move on
    last;
  }
}
[download]

then you'll iterate over the data once, writing to multiple files depending on the $hostid variable. after each data record is processed, it moves on to the next line without evaluating any other servers in the list.

if you haven't already, you might consider using warnings or -w. you might also consider using strict. it would take some work with this code, but it makes perl debugging so much easier.

~Particle *accelerates*

Comment on Re: Super Duper Inefficient Select or Download Code

Replies are listed 'Best First'.
Re: Re: Super Duper Inefficient by Earindil (Beadle) on Jun 20, 2003 at 13:19 UTC
Would opening and closing files thousands of times be more of a hit than just 80 or so times? If generating 5 day graphs @DATA would be approx 288580 lines. Would it perhaps be better to open all the files first before the loop and then close them all after?	[reply]
Re^3: Super Duper Inefficient by particle (Vicar) on Jun 20, 2003 at 13:26 UTC
sure, you could open all files first, and store the filehandles in a hash, keyed by servername. depending on the size of the records, you could also process the data in one loop, store it in memory, and write it all out in a second loop. there are many ways to do it. you might try benchmarking a few to see what works best for you. ~Particle accelerates	[reply] [d/l]


more useful options
	PerlMonks