Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re: storing data in a hash

by shmem (Chancellor)
on Jun 22, 2006 at 16:04 UTC ( #556930=note: print w/replies, xml ) Need Help??


in reply to storing data in a hash

From what you write it is not very clear to me what you want to do.

Does the log file span a day change?

If so, is it, that given queries reflected in the logfile as for the following graph

day 1 | day 2 ---- | -------- | -------| ----|-- -----|-- ----|--- --|---- |------ | -------
you want to divide the data amount between the days and sum them up foreach date?

Or is the the logfile always either one day or the other?

In either case, use Time::Local. When does the script run? I assume the script runs roughly at any time between 20:30:00 and 03:00:00. Make a UNIX timestamp of the day change with

my @now = localtime; my $hour = $now[2]; $now[$_] = 0 for 0..2; # set hour, min, sec to 0; $daychange = timelocal(@now); # add one days seconds if running before midnight $daychange += 86400 if $hour > 2;
Then you can compare the timestamp in the second field of your rows to that timestamp and sum it to the right day sum:
$sum{ $daystamp <= $stoptime ? 'first' : 'second' } += $linesum;
Voila, you have the sum of one day in $sum{'first'} and of the next day in $sum{'second'}, now you can insert into your database whichever of the two contains data. If the log file spans days, you could split the amount of bytes according to the second before and after a days shift foreach row.

cheers,
--shmem

_($_=" "x(1<<5)."?\n".q/)Oo.  G\        /
                              /\_/(q    /
----------------------------  \__(m.====.(_("always off the crowd"))."
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

Replies are listed 'Best First'.
Re^2: storing data in a hash
by sunadmn (Curate) on Jun 22, 2006 at 19:57 UTC
    shmen
    That doesn't seem to work at all as I have made a test file that has different dates in it and I still only get one or the other when testing to see if the $sum{'first'} or $sum{'second'} have content. Let me try to exaplain the problem again.

    OK so the data file has data that can be either from today or tomorrow in it. Data reads top down line for line, so lines 0-10 would be for today and then let's say lines 11-21 are for tomorrow. Now we want to take all the lines that belong to today and sum them up and insert the date (today's date from the UNIX time stamp in the file) and sum into our DB then insert the same for tomorrow. The time stamp fields represent a "RUN" period for the given line, meaning the time the query started and the time it stoped. We want to look at the stop time and convert the UNIX time stamp to scalar context so we have a human readible date. Now that we have that we want to take all Jun 22 data sum it all together and insert it into the DB and if we see Jun 23 we do the same and insert that data in a new row. Does this make sense?

    SUNADMN
    USE PERL
      Thinking it hard, no. How can the file have dates of tomorrow? Isn't it a log? Isn't it rather yesterday and today?

      --shmem

      _($_=" "x(1<<5)."?\n".q/)Oo.  G\        /
                                    /\_/(q    /
      ----------------------------  \__(m.====.(_("always off the crowd"))."
      ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
        Yeah sorry my brain is fried today.
        SUNADMN
        USE PERL
      Oops, serveral typos in the code I gave you. Always use -w ! :-)
      #!/usr/bin/perl -w use Time::Local; my @now = localtime; my $hour = $now[2]; $now[$_] = 0 for 0..2; # set hour, min, sec to 0; $daychange = timelocal(@now); my @dataSUM = <DATA>; my $sum =0; my $firststart; my $laststop; #my $unixtime; foreach my $line(@dataSUM) { my @lineData = split(/[ \t]+/, $line); my $startime = $lineData[0] if(&is_numeric($lineData[0])) || + die "$lineData[0] not numeric\n"; my $stoptime = $lineData[1] if(&is_numeric($lineData[1])) || + die "$lineData[1] not numeric\n"; # $unixtime = scalar localtime $lineData[1] if(&is_numeric( +$lineData[1])) || die "$lineData[1] not numeric\n"; my $xover_bytes = $lineData[7] if(&is_numeric($lineData[7])) || + die "$lineData[7] not numeric\n"; my $art_bytes = $lineData[9] if(&is_numeric($lineData[9])) || + die "$lineData[9] not numeric\n"; my $list_bytes = $lineData[11] if(&is_numeric($lineData[11])) || + die "$lineData[11] not numeric\n"; my $newnews_bytes = $lineData[13] if(&is_numeric($lineData[13])) || + die "$lineData[13] not numeric\n"; my $lineSum = $xover_bytes + $art_bytes + $list_bytes + $newn +ews_bytes; # $sum += $lineSum; $sum{ ($daychange >= $stoptime ? 'first' : 'second') } += $lineSum; $firststart = ($startime < $firststart) || $firststart ==0 ? $star +time : $firststart; $laststop = ($stoptime > $laststop ) ? $stoptime : $laststop; } map { print "$_ => $sum{$_}\n" } keys %sum; sub is_numeric { my $data = shift; return($data =~ m/^\d+$/); } __DATA__ 1149621887 1150922348 stormchaser 69.0.0.0 69.0.0.0 NG004193 +@net.NET 3 820 156 1757 0 0 0 0 + 0 0 176 1149621900 1150922348 stormchaser 69.0.0.0 69.0.0.0 NG004193 +@net.NET 1 648 80 00 0 0 0 0 + 0 93 1149622376 1150922348 stormchaser 69.0.0.0 69.0.0.0 NG004193 +@net.NET 3 793 243 852 0 0 0 0 + 0 0 268 1149625677 1150922348 stormchaser 213.0.0.0 213.0.0.0 Sinkbad@ +net.net 1 493 0 00 0 0 0 0 + 0 1 1149621887 1151008726 stormchaser 69.0.0.0 69.0.0.0 NG004193 +@net.NET 3 820 156 17570249 0 0 0 0 + 0 0 176 1149621900 1151008726 stormchaser 69.0.0.0 69.0.0.0 NG004193 +@net.NET 1 648 80 00 0 0 0 0 + 0 93 1149622376 1151008726 stormchaser 69.0.0.0 69.0.0.0 NG004193 +@net.NET 3 7932259 243 85227944 0 0 0 0 + 0 0 268
      Yields:
      first => 5363 second => 110731920

      cheers,
      --shmem

      _($_=" "x(1<<5)."?\n".q/)Oo.  G\        /
                                    /\_/(q    /
      ----------------------------  \__(m.====.(_("always off the crowd"))."
      ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://556930]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (3)
As of 2020-07-08 04:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?