Re^7: Any idea for predicting the peak points in the graph by perl

The graph you provided is upside down

That's irrelevant. It does not affect the veracity of the plot.
But as it seems to bother you; I've posted the same graph flipped vertically (with a couple of additions) below.
Correct Graph

Your link leads to a page that reads: This image or video is currently unavailable.
Please provide your spreadsheet that you used to generate it,

I didn't use a spreadsheet. This is a Perl site. I used Perl :)
I am very confused how you could even make that happen. You'd have to have multiplied all y values with a -1 to cause such a thing.

Or; plot the data on a medium that has the origin top left with Y running top to bottom. As is the convention with most computer graphics.
The 3 curves presented here are not to scale and lack axes. As such they misrepresent the meaningfulness of the first derivative in peak detection.

Since all the Y values on the three curves are related numerically, their absolute values are irrelevant. Hence scales are superfluous.
The X values are the same, drawn to the same scale and offset for all three plots.
Your conclusion that upside down derivatives are not useful in peak detection is in fact correct. I assure you that a correctly oriented first derivative is useful for this type of problem.

Sorry, but that makes no sense at all. Since the important points are where the curves transition across the Y=0 line. Whether that transition occurs going from above to below or below to above doesn't change anything one iota.
If you redraw it to scale with axes you will notice that any time the first derivative crosses the x axis (aka it is zero) there will be a peak.

If that were the case, I would not have posted. Take another, closer look at the graph.
The additional black horizontal line is the x-axis (Y=0) of both the 1st (green) and 2nd (blue) derivative plots. The additional black vertical lines mark the peaks and troughs in the data plot (red).
Note how the 1st derivative plot (green) doesn't transition 0 at all for the first two turning points, and is substantially inaccurate for turning points four and six; slightly inaccurate for the fifth; leaving just 2: the third and seventh that it hits accurately.
You may now be wondering how the black vertical lines were drawn.
If you subtract consecutive data points and compare the results to 0:
```
my @deltas = map{ ( 0 <=> $y[$_-1] - $y[$_] ) } 1 .. $#y;
[download]
```
You get a dataset like this:
```
-1 1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 1 1 1 -1 -1 -1 -1 1 1 
+1 1 1 1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 1 1
[download]
```
You then plot a vertical line at the X value everwhere the sign changes:
```
$deltas[$_-1] != $deltas[$_] and $im->line( $x[$_], 0, $x[$_], 800, 0 
+) for 1 .. $#x-1;
[download]
```
And there you have your maxima and minima simply, directly, and accurately.

With a very small perl program (less than 200 lines) I have a peak detector that works pretty well for this type of psuedo-sinusoidal data.

So verbose!? :)

Here is my Perl code, a whole 40 lines, that plots the above linked graph:

#! perl -slw
use strict;
use Data::Dump qw[ pp ];
use GD;
use constant {
    WHITE => unpack( 'N', pack 'CCCC', 0, 255, 255, 255 ),
    RED   => unpack( 'N', pack 'CCCC', 0, 255,   0,   0 ),
    GREEN => unpack( 'N', pack 'CCCC', 0,   0, 255,   0 ),
    BLUE  => unpack( 'N', pack 'CCCC', 0,   0,   0, 255 ),
};

my( @x, @y, @yd1, @yd2 );
( $x[@x], $y[@y], $yd1[@yd1], $yd2[@yd2] ) = map{ $_ //= 0 } split whi
+le <DATA>;
chomp @yd2;

$_ = ( $_ -4 ) * 1000 for @x;
$_ /= 6   for @y;
$_ = $_ / 320 + 400 for @yd1;
$_ = $_ / 8000 + 400 for @yd2;


my $im = GD::Image->new( 1000, 800, 1 );
$im->filledRectangle( 0, 0, 1000, 800, WHITE );
$im->line( 0, 400, 1000, 400, 0 );
$im->line( $x[$_-1], $y[$_-1], $x[$_], $y[$_], RED ) for 1 .. $#x;
$im->line( $x[$_-1], $yd1[$_-1], $x[$_], $yd1[$_], GREEN ) for 2 .. $#
+x-1;
$im->line( $x[$_-1], $yd2[$_-1], $x[$_], $yd2[$_], BLUE ) for 3 .. $#x
+-2;

my @deltas = map{ ( 0 <=> $y[$_-1] - $y[$_] ) } 1 .. $#y;
$deltas[$_-1] != $deltas[$_] and $im->line( $x[$_], 0, $x[$_], 800, 0 
+) for 1 .. $#x-1;

$im->flipVertical;

open PNG, '>:raw', "$0.png" or die $!;
print PNG $im->png;
close PNG;

system 1, "$0.png";
[download]

And the dataset taken directly from salva's post above:

__DATA__
4.1330 4580.87
4.1470 4522.75     -1328.56
4.1600 4545.00     -7791.65     -935707.98
4.1730 4320.17    -25656.96    -1544908.57
4.1870 3852.26    -49504.19    -1396364.62
4.2000 2983.56    -63358.81       59518.52
4.2130 2204.93     -47956.7     1368481.22
4.2270 1688.73    -26409.81     1302338.03
4.2400 1491.87    -12793.58      687964.39
4.2530 1356.09     -8522.74      246232.62
4.2670 1261.75      -6145.3       120543.1
4.2800 1190.17     -5268.08       99532.76
4.2930 1124.78     -3557.44      180758.68
4.3070 1094.12      -387.59      357928.14
4.3200 1114.32      6106.62      533914.53
4.3330 1252.89     13494.19      473482.01
4.3470 1478.66     18890.63      334228.18
4.3600 1762.94     22518.35       61078.35
4.3730 2064.14     20478.67     -278669.88
4.3870 2315.86     14994.26     -533957.74
4.4000 2468.98      6061.81     -715688.03
4.4130 2473.47     -3613.63     -574250.77
4.4270 2371.42     -9442.96     -274812.86
4.4400 2218.51    -11033.58       93502.85
4.4530 2084.54     -7011.89      376399.97
4.4670 2029.19      -870.78       468240.9
4.4800 2061.03      5630.62      455004.27
4.4930 2175.58     10959.33      278503.96
4.5070 2356.93     13150.22       13708.45
4.5200 2530.64     11329.46      -316569.8
4.5330 2651.50      4919.41     -526292.81
4.5470 2663.46     -2880.44     -563985.17
4.5600 2573.73    -10308.19     -476770.66
4.5730 2395.45    -15276.48     -346338.56
4.5870 2161.26    -19659.33      -286611.8
4.6000 1864.65       -23015      -18659.54
4.6130 1562.87    -20144.48      218052.13
4.6270 1320.75    -17127.59      195150.31
4.6400 1100.43    -14875.42      287534.19
4.6530  933.99      -9651.7      387333.91
4.6670  839.83     -4417.41      369091.59
4.6800  814.72       313.77      341683.76
4.6930  847.99      4466.37      298057.93
4.7070  935.31      8361.33      382671.47
4.7200 1073.75      14798.5      519444.44
4.7330 1320.07     21866.89      519764.75
4.7470 1664.15     28832.15      506969.93
4.7600 2098.54     35555.08
4.7730 2588.58
[download]

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

Comment on Re^7: Any idea for predicting the peak points in the graph by perl Select or Download Code

Replies are listed 'Best First'.
Re^8: Any idea for predicting the peak points in the graph by perl by gaimrox (Initiate) on Jul 16, 2012 at 03:10 UTC
Strawman? Our discussion here is in reply to the parent who made a claim that the second derivative was useful in finding extrema. I have posted so that future readers will not believe this, as I believe it to be untrue. In reply to my claim you have posted a number interesting things - but none seemingly address the claims of the parent, or myself. The OP claims that the second derivative is useful for determining extrema. His claim suggests that looking at toggling signs on the resulting series indicates extrema. I claim this as false. I also claim that it is possible to find extrema using the first derivative. Any time the derivative crosses the X axis there will be a maxima or minima present within the source series. I do not claim that this will find all extrema - I do claim that it will also require additional processing to capture all extrema of interest. Are you claiming this is false? Your above program is quite a bit shorter than mine. I have a different type of data, but I am pretty sure that looking at deltas like this is a more precise and efficient way of solving the problem than using first derivatives - I disagree that it's more accurate, but that's nitpicking. Overall it is better than doing solely a first derivative and picking from that. Finally, I strongly disagree that using inverted graphs (even if that's what's used in computer graphics) is a good way to prove your point about much of anything. That said, had you labelled your graph and explained what it was I we both could have saved some time	[reply]
Re^9: Any idea for predicting the peak points in the graph by perl by BrowserUk (Patriarch) on Jul 16, 2012 at 09:36 UTC
Strawman? Hardly. You said 2nd derivative was no good and that 1st derivative was the way to go. I demonstrated that neither is a good mechanism for the data in question. You countered: "your graph is upside down" -- a strawman if ever. I demonstrated that of the 7 inversion points in the data; first derivative only sees 5 of them. And of those 5; three are substantially inaccurate. Indeed, if you draw a few vertical lines on your own linked graph -- which is now at least accessible -- you can see both problems: missed turning points and inaccurately discovered ones. The age old problem with first (and second) deritives -- along with many other numerical methods -- is that they have a tendency to discover values that don't exist in the dataset. Ie. calculated values that fall between the discrete values that are actually in the dataset. Whilst this is fine for theoretical discussion -- rounded to some number of sig.fig. -- it leaves real-world applications with the need to fall back upon heuristics -- ie. guesses -- in order to "correct" calculated values and align them with the actual data. Imagine the dataset represents clock-speed (or power drain) from a deep-bin sort of newly minted cpus. -- ie. when cpus are manufactured, there is some considerable variability in their electrical performance; and manufacturers can sort the parts by their actual performance, and charge premium prices for the better ones. In the many all-too-real scenarios like this that crop up in manufacturing every day, having calculated maxima and minima that fall at theoretical points on the curve; between the actual values that are there, isn't very useful for the selection processes that are the reason for performing the calculation in the first place. So no, not a strawman. A legitimate and relevant discussion in context. I accept that my original graph was, as presented, difficult to interpret. But I anticipated that anyone interested would produce their own graph to verify the accuracy of my quick plots and conclusions. Which you did. Inspect their own plots carefully and notice the obvious discrepancies that I noticed in mine. Which you did not. That's the trouble with "pretty graphs". People get so impressed by the pictures, they forget to inspect them closely and take note of why they produced them in the first place. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]