Fun, kinda silly, and doesn't actually work on your trial set. Failures happen around points with large deviations from the smoothed concept.

my @also = sort {$a->[0] <=> $b->[0]} @points;
for my $i (reverse 0 .. $#points) {
    splice @also, $i, 1 if $points[$i] ne $also[$i];
}
[download]

More generally, I feel like there is a merge-sort-like solution in there somewhere, but can't quite seem to distill it in my brain. Not that it would be practical, since push-pop is already O(N), but that's not the point, right?

Streaming buffer w/ STDIN/STDOUT:

my $frame = 10;
my @buffer;
$, = ' ';
$\ = "\n";
while (<>) {
    my @row = split;
    if (@buffer and ($buffer[-1][0] < $row[0]) ^ ($buffer[-1][1] < $ro
+w[1]) ) {
        pop @buffer;
    } else {
        print @{shift @buffer} if $frame < push @buffer, \@row;
    }
}
print @$_ for @buffer
[download]

Note the ridiculous micro-optimization for comparison.