Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Data Normalization with PDL

by lin0 (Curate)
on Jul 07, 2007 at 15:59 UTC ( #625423=snippet: print w/replies, xml ) Need Help??

A common practice in machine learning is to preprocess the data before building a model. One popular preprocessing technique is data normalization. Normalization puts the variables in a restricted range (with a zero mean and 1 standard deviation). This is important to achieve efficient and precise numerical computation.

In this snippet, I present how to do data normalization using the Perl Data Language. The input is a piddle (see comment below for a definition) in which each column represents a variable and each row represents a pattern. The output is a piddle (in which each variable is normalized to have a 0 mean and 1 standard deviation), and the mean and standard deviation of the input piddle.

What are Piddles?

They are a new data structure defined in the Perl Data Language. As indicated in RFC: Getting Started with PDL (the Perl Data Language):

Piddles are numerical arrays stored in column major order (meaning that the fastest varying dimension represent the columns following computational convention rather than the rows as mathematicians prefer). Even though, piddles look like Perl arrays, they are not. Unlike Perl arrays, piddles are stored in consecutive memory locations facilitating the passing of piddles to the C and FORTRAN code that handles the element by element arithmetic. One more thing to note about piddles is that they are referenced with a leading $



use warnings;
use strict;

use PDL;
use PDL::NiceSlice;

# ================================
# normalize
# ( $output_data, $mean_of_input, $stdev_of_input) = 
#   normalize( $input_data )
# processess $input_data so that $output_data
# has 0 mean and 1 stdev
# $output_data = ( $input_data - $mean_of_input ) / $stdev_of_input
# ================================
sub normalize {
    my ( $input_data ) = @_;
    my ( $mean, $stdev, $median, $min, $max, $adev )
        = $input_data->xchg(0,1)->statsover();
    my $idx = which( $stdev == 0 );
    $stdev( $idx ) .= 1e-10;
    my ( $number_of_dimensions, $number_of_patterns )
        = $input_data->dims();
    my $output_data
        = ( $input_data - $mean->dummy(1, $number_of_patterns) ) 
        / $stdev->dummy(1, $number_of_patterns);
    return ( $output_data, $mean, $stdev );
Replies are listed 'Best First'.
Re: Data Normalization with PDL
by lin0 (Curate) on Jul 07, 2007 at 16:22 UTC

    One typical practice is to use the code above to normalize the training data and to use the resulting mean and standard deviation to normalize the testing data as illustrated in the snippet below:

    #!/usr/bin/perl use warnings; use strict; use PDL; use PDL::NiceSlice; # ================================ # normalize_testing_patterns # $output_data = # $normalized_testing_patterns( $input_data, # $mean_of_input, # $stdev_of_input) ) # # processess $input_data so that $output_data # has 0 mean and 1 stdev # # $output_data = ( $input_data - $mean_of_input ) / $stdev_of_input # ================================ sub normalize_testing_patterns { my ( $input_data, $mean, $stdev ) = @_; my ( $number_of_dimensions, $number_of_patterns ) = $input_data->dims(); my $output_data = ( $input_data - $mean->dummy(1, $number_of_patterns) ) / $stdev->dummy(1, $number_of_patterns); return $output_data; }



Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: snippet [id://625423]
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others studying the Monastery: (4)
As of 2023-11-30 04:32 GMT
Find Nodes?
    Voting Booth?

    No recent polls found