in reply to Finding local maxima/minima in noisy, pediodic data

This seems to work pretty well on any dataset produced by your generator, along with a few variations on that, that I've tried:

#! perl -slw use strict; use Data::Dump qw[ pp ]; use GD::Graph::lines; use GD::Graph::colour qw(:colours :lists :files :convert); my $graph = GD::Graph::lines->new( 6000, 768 ); die 'No data file specified' unless @ARGV and -e $ARGV[ 0 ]; open my $fh, '<', $ARGV[ 0 ] or die "Couldn't open $ARGV[ 0 ]"; my @data = ([],[]); push( @{ $data[ 0 ] }, $1 ), push( @{ $data[ 1 ] }, $2 ) while <$fh> =~ m[(\S+)\s+(\S+)]; close $fh; #pp \@data; <>; $graph->set( title => 'Y over X', 'bgclr' => 'white', 'transparent' => 0, x_label => 'X', x_max_value => 6000, x_tick_number => 6, y_label => 'Y', y_tick_number => 8, ) or die $graph->error; my @hilos; my $crossover = $data[1][0]; my $dir = $data[1][1] > $crossover; my( $max, $min ) = ( -1e308, 1e308 ); my $count = 0; for my $y ( @{ $data[ 1 ] } ) { ++$count; $max = $y if $y > $max; $min = $y if $y < $min; if( $dir and $y < $crossover ) { push @hilos, ( $max ) x $count; $count = 0; $dir = 0; $max = $crossover; } elsif( !$dir and $y > $crossover ) { push @hilos, ( $min ) x $count; $count = 0; $dir = 1; $min = $crossover; } } push @data, \@hilos; my $gd = $graph->plot( \@data ) or die $graph->error; open(IMG, '>800691.png') or die $!; binmode IMG; print IMG $gd->png; close IMG; system '800691.png';

It should be reasonably efficient as there's no sorting involved, just a single pass through the data. Just supply the script with the name of the data file as its only argument. The maximas and minimas (do they have a collective name? extremas?), will be plotted as a square wave in green overlaying the data in red.

It's easy to see datasets that it would screw up on, but it just depends how good a model your generator is?


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP PCW It is as I've been saying!(Audio until 20090817)

Replies are listed 'Best First'.
Re^2: Finding local maxima/minima in noisy, pediodic data
by kikuchiyo (Hermit) on Oct 12, 2009 at 14:57 UTC
    Thanks.

    I couldn't run your program because I don't have the GD::Graph module (I couldn't install it and don't have time right now to fix it), but I took the core loop of your program and adapted it. Well, it is not satisfactory.

    It incorrectly labels some extrema as true ones but also misses some of them.

    It turns out that my test data generator was too simple. Here is a real example.

      In the interim, I plotted the second and third columns against the index on separate graphs using my first algorithm and posted the results here & here respectively.

      Perhaps you could load one or both of these images into a simple graphics editor and annotate the misses and false hits for us. I can see for example that the last 5 periods on the first graph above have false hits,

      Update: but I didn't notice any misses? Now I have. The 12th period on the first graph is a miss.

      But maybe I'm plotting the wrong values? Maybe you are working with the midpoints of the 2nd & 3rd columns?


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        The data file posted on pastebin contains data from two channels recorded during the same measurement. 2nd col. vs. 1st col. is one dataset and 3rd vs. 1st. is the other.

        So you are plotting the right values.

        You are also correct about misses and false hits.
        Anyway, here is your first graph with missed peaks marked with ugly blue arrows and false peaks marked with smaller but equally ugly gray arrows.

      Could you explain how the 3 columns in this sample relate to the two column data in your ealier post?


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.