BrowserUk has asked for the wisdom of the Perl Monks concerning the following question:
The problem I'm dealing with is the removal of test equipment artifacts from experimentally derived datasets. In layman's terms, the sensing circuit has a tendency to oscillate a little occasionally; which introduces inflections into the data which must be removed before the data can be used for the next part of the processing which requires monotonic data.
This is a case where a picture is worth a thousand words: http://oi63.tinypic.com/2z72xbr.jpg. The blue is (a small subset of) the raw data and the red is the desired output.
This code performs the cleanup. (It runs as a pipe filter, and I've adapted it to use __DATA__ for the small subset shown in the graphic for posting purposes.):
The basic mechanism is to compare the points as successive pairs and look for a negative slope. When a negative slope is found, *both* points are removed from the dataset.
The complication is that pairwise comparisons have to continue with the next point against the *previous* unremoved point.
To (I hope) clarify, in the following:
H G F E D C B A
- Point A is passed through
- Points A & B are compared, positive slope: pass B through.
- Points B & C are compared, positive slope: pass C through.
- Points C & D are compared, negative slope: block D and remove C.
- Points B & E are compared, positive slope: pass E through.
- Points E & F are compared, negative slope: block F and remove E.
- Points B & G are compared, positive slope: pass G through.
- Points G & H are compared, positive slope: pass H throught.
Resultant dataset: A, B, G, H.
The above code works; but is there a better way of coding it?
I'm not going to define "better"; I'd simply like to see a range of alternative codings.
(Please note: this is not about filtering or smoothing or interpolating the data; just removing spurious values.)
The output from the sample code: