here's a clue:

foreach $server (@servers) {
  chomp $server;
  open TMP, ">/tmp/$server.tmp";
  $ENV{"FILENAME"} = "/tmp/$server\.tmp";
  foreach(@DATA) {
    ($hostid, $time, $get, $post, $currconn) = split(/\t/);
    next if ($hostid ne $SERVER{$server});
    ## more here...
  }
  close TMP;
}
[download]

this is the inefficiency. invert these loops to make this something like:

## iterate through record
foreach(@DATA) {
  ($hostid, $time, $get, $post, $currconn) = split(/\t/);
  ## match against the server list and process accordingly
  foreach $server (@servers) {
    chomp $server;
    next unless $hostid eq $SERVER{$server};

    open TMP, ">>/tmp/$server.tmp";
    $ENV{"FILENAME"} = "/tmp/$server\.tmp";

    ## more here...

    close TMP;

    ## done with this record, move on
    last;
  }
}
[download]

then you'll iterate over the data once, writing to multiple files depending on the $hostid variable. after each data record is processed, it moves on to the next line without evaluating any other servers in the list.

if you haven't already, you might consider using warnings or -w. you might also consider using strict. it would take some work with this code, but it makes perl debugging so much easier.

~Particle *accelerates*

Would opening and closing files thousands of times be more of a hit than just 80 or so times?
If generating 5 day graphs @DATA would be approx 288*5*80 lines.
Would it perhaps be better to open all the files first before the loop and then close them all after?

sure, you could open all files first, and store the filehandles in a hash, keyed by servername. depending on the size of the records, you could also process the data in one loop, store it in memory, and write it all out in a second loop. there are many ways to do it. you might try benchmarking a few to see what works best for you.

~Particle *accelerates*

next if ($hostid ne $SERVER{$server});
[download]

my %new_data = ();
foreach ( @DATA ) {
  my ($hostid, $rest) = split /\t/, $_, 2;
  push @{$new_data{$hostid}}, "$hostid\t$rest";
}
[download]

--tidiness is the memory loss of environmental mnemonics

I was formulating this same response myself, but I'll provide a minor variation, which does pretty much the same, but uses automatic variables instead of temporary placeholder variables:

my %new_data = ();
foreach ( @DATA ) {
  /^(.*?)\t.*$/;
  push @{$new_data{$1}}, $_;
}
[download]

Either case is equally valid in my eyes - I do not know which is faster, the split() or //, so I'll leave that to better souls than mine to answer on.

The important thing I'd like to point out is that the above method - ran as a step before the server loop - effectively reduces your redundancy to two reads - once to populate the hash, and then once for the server-specific loop.

use POSIX 'str2time';

my $time = strftime '%m/%d/%Y %H:%M', localtime($time);
[download]

      
--------------------------------
SV* sv_bless(SV* sv, HV* stash);

That's great. Thanks. I was already using POSIX earlier in the script for mktime to get my time in seconds but didn't know about strftime. Looks much better and cuts out several lines. Thanks to everyone else to. I'm trying out each of your suggestions.

foreach (@DATA) {
   ($hostid, $time, $get, $post, $conn) = split(/\t/);
   push @{$munged_data{$hostid}},
      {time => $time,get => $get,post=>$post,conn=>$conn};
}
[download]

foreach my $server (@servers) {
   foreach(@{$munged_data{$server}}) {
     #do stuff with $_->{time} etc.
   }
}
[download]

Cheers, Ben.