IMHO¹ the surface of a convex hull already describes the (optimal) planes where all atoms of your molecule are on one side of the plane (or of course in the plane).

So your problem reduces to calculate the margins of your convex hull:

At least one edging plane should separate the atoms into distinct groups!

          |
   \      |
    x  /  o   o
  x   x   | o   
    x  \  o   o
   /      |
          |

[download]

Of course, when going into real world physics the distance between two molecules should be bigger than the distance between the atoms, to be considered different molecules. But this can be calculated with those surface planes by adding a tolerance.

IMHO those are too close 

    x  |  o
  x   x|o   o
    x  | o o

[download]

I don't know if you need a simple or a fast algorithm, but in many cases you can shorten the cases by approximating a (minimal) sphere which includes all atoms of a molecule.

Only if those spheres (maybe widened by the mentioned minimal distance) intersect you will need to calculate the surfaces of the molecule.

And you will only need to calculate the surfaces close to this intersection.

IMHO this should be considerably fast and intuitive, for further optimization better rely on the WP article on Support vector machines BrowserUK was linking to.

Cheers Rolf

¹) prove should be trivial.

UPDATE: It should be mentioned that IMHO physics knows examples of molecules which are distinct but intersect from a mathematical point of view. Something called bucketball rings a bell for me...

In reply to Re^3: Confused about 3D geometry / algebra by LanX
in thread Confused about 3D geometry / algebra by tomdbs98

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.