Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: intelligent re-sampling of data

by mattr (Curate)
on Jun 28, 2005 at 07:00 UTC ( [id://470515]=note: print w/replies, xml ) Need Help??


in reply to intelligent re-sampling of data

If you're asking about perl this and above post on GD are relevant. Get the data out of Excel and reprocess it yourself. Then either make a new excel sheet with condensed data, or draw your own chart in GD.

I have had good results with parsing data from up to 1MB size spreadsheets (the parsing takes a little while at that size). I am plotting just a certain window in time though as bar charts with plain vanilla GD.

You could do lots of processing, but your eye is really good at picking things out too. So instead of figuring out how to reduce the amount of data I would like to suggest that you simply plot all of the data, perhaps with an appropriate transformation (stretch) to emphasize what you are looking for.

If it is just 1 and 0 well that's not so bad. But if it is fine differences then try and expand the spectrum of color you use too.. Once you have a finely drawn chart, explore it with a program that lets you change gamut and zoom in if you like, there may be a lot of detail you can't see easily but will become uncovered by changing the palette. For example xephem may show you what kinds of things are possible (I have the comet on my mind). It can find dim stars, and invert or add tails to make them easier to see. Or the gimp, though I have in an image processing program called NIH Image (or NIH ImageJ for windows/linux/osx). It's not just for biotech, and has an astronomy file as a sample which may be applicable to your problem. It has some interesting features although I don't think it lets you drag and adjust the lookup table (GLUT) in realtime anymore like photoshop does. Something that lets you do that and maybe zoom or view from different angles (if you have a 3d box for example) can help you find patterns extremely quickly.

Possibly an opengl model of it could be interesting too (I'm thinking of Wx::GLCanvas though search.cpan.org searching for "opengl" gave some likely suspects.) There's undoubtedly a lot of other stats programs out there that could be used, like R statistics language with the IDPmisc libary package for display of large datasets (adds effects to the display). And there is IBM's OpenDX (Open Visualization Data Explorer, which is open source). The Colormap Editor in the screenshot on that page is what I was talking about anyway it is a very cool system, only a 40 megabyte download away!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://470515]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others romping around the Monastery: (3)
As of 2024-04-25 02:22 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found