more useful options | |
PerlMonks |
intelligent re-sampling of databy spurperl (Priest) |
on Jun 27, 2005 at 10:34 UTC ( [id://470201]=perlquestion: print w/replies, xml ) | Need Help?? |
spurperl has asked for the wisdom of the Perl Monks concerning the following question:
In this node I complained about Excel's ancient limitations that make charting large data sets with it impossible. Various solutions were proposed, but I was looking for something quick and as non-drastic as possible. Moving away from Excel is not an option, at the moment, for various reasons. So, I decided to resample the data. Having N data points when N > MAX_LIMIT (about 32k for Excel charts) I have to resample the data in such a way as to display it in less than MAX_LIMIT points. Now, naive sampling doesn't work here because this way it's easy to lose interesting information (assume I have all the data 0 and a single 1 somewhere. I want to see it, and using a trivial resampling algorithm the chance for seeing it is small). The solution I've currently set on is a "maximum filter". I divide the data to "windows" and take one "representable" value from the window - the maximal data value. This works very nicely since we don't have negative data, but I wonder what are the more general algorithms for arbitrary data ? After all, Excel and other plotters use it. Even if I plot only 32000 points, there still are only ~1000 pixels in my screen so Excel does some resampling of its own.
Back to
Seekers of Perl Wisdom
|
|