IMHO¹ the surface of a convex hull already describes the (optimal) planes where all atoms of your molecule are on one side of the plane (or of course in the plane).

So your problem reduces to calculate the margins of your convex hull:

At least one edging plane should separate the atoms into distinct groups!

          |
   \      |
    x  /  o   o
  x   x   | o   
    x  \  o   o
   /      |
          |
[download]

Of course, when going into real world physics the distance between two molecules should be bigger than the distance between the atoms, to be considered different molecules. But this can be calculated with those surface planes by adding a tolerance.

IMHO those are too close

    x  |  o
  x   x|o   o
    x  | o o
[download]

I don't know if you need a simple or a fast algorithm, but in many cases you can shorten the cases by approximating a (minimal) sphere which includes all atoms of a molecule.

Only if those spheres (maybe widened by the mentioned minimal distance) intersect you will need to calculate the surfaces of the molecule.

And you will only need to calculate the surfaces close to this intersection.

IMHO this should be considerably fast and intuitive, for further optimization better rely on the WP article on Support vector machines BrowserUK was linking to.

¹) prove should be trivial.

UPDATE: It should be mentioned that IMHO physics knows examples of molecules which are distinct but intersect from a mathematical point of view. Something called bucketball rings a bell for me...