Bertrand Meyer (the inventor of Eiffel) uses 'abstract data types' (ADT) as the underlying concept for objects in his book Object Oriented Software Construction. Googling for ADTs brings up, e.g. this or that.

I'm not so sure whether this is of any practical help for you but you were asking for a mathematical notation ...

-- Hofmator