I took calculus as part of my electrical engineering degree. The irony of majoring in EE is that I'm not that hot at math, at least abstract math. I like hard numbers that represent real things. The professor explained it like this. He graphed a function on the board and started working out how to find the area below the curve. As an aside, he started to explain that a chemist, confronted with the same problem, would simply draw the graph out 10 times and weigh the resulting graphs to arrive at the answer. Depending on the needed accuracy, that chemist might perform the graph and weigh 100 or 1000 times. After he explained this, I thought to myself, 'crap, I guess that means I should have gone with chemistry'.

Recently, I came up against a real life situation involving complex curves and the area under them. I dutifully combed the internet and dug out my old probability and statistics text book to figure out the answer to the question, "How do I find the union of two normal distributions?" (Normal distribution - the famous "bell curve").

I found many many texts, some of them starting out with just the basics and working their way up. I found lot's of complex formulas that did not do quite what I was looking for, formulas that looked like they were written in sanskrit. After 5 or 6 hours of frustration, I decided to look at the problem from a chemist's point of view: empirically.

Empirically comparing normal distributions: A normal distribution is a graph of probabilities, really just an array. Perl is great for handling this stuff, so I wrote a method that converts a mean and standard deviation into an array plotting out what a population of 100 would look like. I did this again for the second mean and SD and then compared the results. If a point appeared in both arrays, it was counted. Then, a simple look at the relationship between the number of 'common' points to the population and presto! A simple percentage of similarity!

As an added bonus, I could increase the population to arbitrarily high numbers (1000 or 10000 points) to get better resolution on the graphs and hence, more accurate percentages. I would make my script plot out the 100's of graphs and weigh them for me.

A quick search found me a GPL'd method for finding the percentage of a population at a datapoint based on the mean and SD: 

sub int_gen_curve
{
        my ($self, $x, $mean, $sdev ) = @_;
    my $pi = 3.14159265358979323844;
        return 1 / ( sqrt( 2 * $pi * $sdev * $sdev ))
                * exp( -($x-$mean)*($x-$mean) / ( 2 * $sdev * $sdev ))
+;
}

[download]
Where '$x' is a point on the x axis. Next, I wrote a simple routine for turning my mean and standard deviation into an array: 
sub compare_bell_curves
my ($self,$m1,$sd1,$m2,$sd2) = @_;

my $upperbound = sprintf("%.1f", ($m1 + (1.75 * $sd1)));
my $lowerbound = sprintf("%.1f", ($m1 - (1.75 * $sd1)));
my $area = 10000;

my $data;
for (my $x=$lowerbound; $x < $upperbound; $x = $x + 1)
{
   $x = sprintf("%.0f", $x);
   my $posarg =  $self->int_gen_curve($x,$m1,$sd1);
   $data->[$x] = int($area*$posarg);
 }

# do the above again for the $m2 and $sd2
# then subtract $data1 from $data, any $data->[x]
# that remains positive is a point of difference

[download]
In the end, I just had to count the positive points in $data after the subtraction and find out what the percentage relationship was to the population. If there would be NO positive points ($data1 removed all of the points from $data) I would have two equal distributions.

Perl allowed me to reduce a quite complex problem into it's basic elements and solve it using real numbers, empirically. As an added bonus, I had datasets that could easily be plugged into GD::Graph, to give that extra, visual, representation of what the data said.

Does anybody else think like this, or am I just kooky?

oakbox

In reply to Empirically solving complex problems by oakbox

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.