Questions on Optimisation (DateTime::SpanSet and RRDs.pm)

smullis has asked for the wisdom of the Perl Monks concerning the following question:

Hello All...

I have a simple script that calculates the average values from a number of RRD files. I would now like to discard all data returned by RRDs::fetch that does not fall within working hours.

So, after some reading - and some trial and error - the following works:

use strict; 
use warnings;
use DateTime::Event::Recurrence;
use DateTime::SpanSet;
use RRDs;
#
# Set up paths, config data etc...
#

foreach $file ( @array_of_RRD_filenames )
{
    my ( $metric, $interval, $begin, $end ) =
      ( 'AVERAGE', '1800', '-800h', 'now' );
    my $start_span =
      DateTime::Event::Recurrence->weekly( days => [ 1 .. 5 ], hours =
+> 8 );
    my $end_span =
      DateTime::Event::Recurrence->weekly( days => [ 1 .. 5 ], hours =
+> 18 );
    my $span_set = DateTime::SpanSet->from_sets( start_set => $start_s
+pan,
                                                 end_set   => $end_spa
+n );
    push @opts, $file;
    push @opts, $metric;
    push @opts, "-r $interval";
    push @opts, "-s $begin";
    push @opts, "-e $end";
    my ( $start, $step, $names, $data ) = RRDs::fetch @opts;
[download]

For those unfamiliar with RRDs.pm, $data is an AoA ref.

    my ( $intotal, $outtotal, $i );
    foreach my $value ( @{ $data } )
    {
        $start += $step;
        my ( $minute, $hour, $day, $month, $year ) =
          ( localtime( $start ) )[ 1, 2, 3, 4, 5 ];
        $month += 1;    # Why does localtime give month -1?
        my $dt = DateTime->new(
                                year   => $year,
                                month  => $month,
                                day    => $day,
                                hour   => $hour,
                                minute => $minute
        );
        next unless $span_set->contains( $dt );
        $i++;
        $in_total  += $value->[ 0 ];
        $out_total += $value->[ 1 ];
    }
    # Now go off and work out averages / percentages etc...
    # Build a HoH and exit the for loop.
}
# Output some purty HTML.
[download]

The problem is, this is incredibly slow! I've qualitatively isolated the major slowdown to the "next unless $span_set->contains ( $dt )" -- this slows things down by a factor of 5.

dprofpp just segfaults when I try to profile the script.

Anyone have any suggestions as to how to optimise this? Is there another way rather than using Date::Time?

Many thanks in advance for any pointers / help / critisicism / suggestions.

Cheers

Comment on Questions on Optimisation (DateTime::SpanSet and RRDs.pm) Select or Download Code

Replies are listed 'Best First'.
Re: Questions on Optimisation (DateTime::SpanSet and RRDs.pm) by edoc (Chaplain) on Jan 26, 2005 at 13:48 UTC
Does this do what you need? If so, I'm guessing it should do a bit less work than DateTime would.. `my ( $minute, $hour, $day, $month, $year, $wday ) = ( localtime( $start ) )[ 1, 2, 3, 4, 5, 6 ]; next unless (($wday >= 1) && ($wday <= 5) && ($hour >= 8) && ($hour < 18));` [download] cheers, J	[reply] [d/l]
Re^2: Questions on Optimisation (DateTime::SpanSet and RRDs.pm) by smullis (Pilgrim) on Jan 26, 2005 at 14:25 UTC
Thankyou! This is exactly the kind of answer I was looking for... Cheers SM	[reply]
Re: Questions on Optimisation (DateTime::SpanSet and RRDs.pm) by fglock (Vicar) on Jan 27, 2005 at 11:49 UTC
`$span_set->contains( $dt );` Actually this method was not optimized for operating on simple datetimes. The sentence is executed as: `$span_set->contains( DateTime::Set->from_datetimes( dates => [ $dt ] ) );` I'm working in the optimization. update: DateTime::Event::Recurrence was uploaded to CPAN as version 0.19_01. If you benchmark it, please let me know.	[reply] [d/l] [select]


Perl Monk, Perl Meditation
	PerlMonks