Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Histogramming -- Floating point numbers as hash keys

by dokkeldepper (Friar)
on Dec 17, 2005 at 21:27 UTC ( [id://517520]=perlquestion: print w/replies, xml ) Need Help??

dokkeldepper has asked for the wisdom of the Perl Monks concerning the following question:

Dear comonks,

for the purpose of building histograms for data sets containing some millions of numbers, that is counting for several millions of pairs ($key,$value) the number of $values for every value.

A standard approach is to build a hash %values and increment $values{$value}++ while iterating over the data. However, because the $value (s) are floats this may be misleading because of possibly varying internal representation of the same real number.

Or, as the Third Coming of The Camel said 'floating-point numbers are in native machine format only'.

Is there some way (module, idiom) to represent real numbers such that the same real number gets the same floating point representation, so that I can use them as hash keys for the approach above? Or, is there a way to rethink that problem, to get the same result, that is, a histogram?

May $values{pack("d",$value)} be a usefuld approach?

Finally, it would be very useful to have the $values preserved as numbers, because I will do computations with the histograms (WARPing, smoothing, estimation of spectra).

Thank you , deckel the doppel 8-)

-dokkeldepper

Remarks: In response to some replies: I ask for a consistent representation of floats not for precision.

A histogram has a priori nothing to do with graphics. For example, most advanced databases compute histograms of the data to guess the optimality of query execution plans. The graphical representation of a histogram is not my concern.

  • Comment on Histogramming -- Floating point numbers as hash keys

Replies are listed 'Best First'.
Re: Histogramming -- Floating point numbers as hash keys
by BrowserUk (Patriarch) on Dec 17, 2005 at 23:19 UTC

    If you use the pack 'd', $value; approach you will likely end up with every key in your hash having a value of 1. Even a single bit difference in the lowest significant bit will prevent 2.0 & 2.000000000000000001 being considered the same.

    Your best bet might be to use sprintf to truncate and stringify the values into a standardised format to whatever precision fits with your needs. Eg.

    $hash{ sprintf '%.6e', $number };

    This will rationalise the continuous range of inputs into a set of discrete bands to whichever precision you need. This shows the effect of using 6 and 11 digits of precision on a range of pairs of values that have a small difference between them as input (and would be different keys using pack 'd'):

    printf "%20s becomes %.6e\n", $_, $_ for map{ ( $_, ( $_ + 1e-10 ) ) } map{ "2e$_" } -300, -30, -3, 0, 3, 30, 300; 2e-300 becomes 2.000000e-300 1e-010 becomes 1.000000e-010 2e-30 becomes 2.000000e-030 1e-010 becomes 1.000000e-010 2e-3 becomes 2.000000e-003 0.0020000001 becomes 2.000000e-003 2e0 becomes 2.000000e+000 2.0000000001 becomes 2.000000e+000 2e3 becomes 2.000000e+003 2000.0000000001 becomes 2.000000e+003 2e30 becomes 2.000000e+030 2e+030 becomes 2.000000e+030 2e300 becomes 2.000000e+300 2e+300 becomes 2.000000e+300 printf "%20s becomes %.11e\n", $_, $_ for map{ ( $_, ( $_ + 1e-10 ) ) } map{ "2e$_" } -300, -30, -3, 0, 3, 30, 30 2e-300 becomes 2.00000000000e-300 1e-010 becomes 1.00000000000e-010 2e-30 becomes 2.00000000000e-030 1e-010 becomes 1.00000000000e-010 2e-3 becomes 2.00000000000e-003 0.0020000001 becomes 2.00000010000e-003 2e0 becomes 2.00000000000e+000 2.0000000001 becomes 2.00000000010e+000 2e3 becomes 2.00000000000e+003 2000.0000000001 becomes 2.00000000000e+003 2e30 becomes 2.00000000000e+030 2e+030 becomes 2.00000000000e+030 2e300 becomes 2.00000000000e+300 2e+300 becomes 2.00000000000e+300

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Histogramming -- Floating point numbers as hash keys
by jhourcle (Prior) on Dec 17, 2005 at 22:30 UTC

    Typically, when I build histograms for floating point numbers, I have some level of binning required (ie, determining what the size of each catchment area is).

    For instance, although the values might have 6 significant figures past 0, I don't want that the graph to deal with 10^6 datapoints -- I might only want 10^4, so rather than hashing on $value, I'll hash on &bin($value), where 'bin' is a function to strip the value to the precision I want. (which may not be a linear function)

    If you really need the precision -- It's possible that PDL might be what you're looking for (at least, for the estimation of spectra, as I know there are folks that do physics number crunching with it, although I have no idea what it actually entails, not being a physicist myself).

    Update I don't know why we got accused of graphing -- I guess you could graph in PDL, but unless there's a terminology difference, and you're using histogram differently than I'm used to, it is impossible to build a histogram of a continuously variable value -- you have to build a series of intervals -- and it's true for database indexing, you just don't get to see the implementation (it's actually computing what the appropriate sizes of the intervals are)

Re: Histogramming -- Floating point numbers as hash keys
by QM (Parson) on Dec 18, 2005 at 03:20 UTC
    Is there some way (module, idiom) to represent real numbers such that the same real number gets the same floating point representation, so that I can use them as hash keys for the approach above?
    Are you concerned about it being portable, or just consistent on the same hardware? You've got some answers that should help with consistency.

    If you want it portable, you may need something like Math::BigFloat, which, IIRC, should be portable.

    -QM
    --
    Quantum Mechanics: The dreams stuff is made of

Re: Histogramming -- Floating point numbers as hash keys
by BrowserUk (Patriarch) on Dec 18, 2005 at 10:14 UTC
    The graphical representation of a histogram is not my concern.

    And my reply had nothing to do with graphical representation.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://517520]
Approved by ikegami
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others meditating upon the Monastery: (6)
As of 2024-04-19 16:08 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found