I think what you really want, is an expression parser. Regexes can't handle this kind of stuff, especially if you start using recursive definitions, with subexpressions between parens... You might want to check out
Parse::RecDescent, which was created with this kind of application in mind. Start by the tutorial, included with the package on CPAN. n.b. You'll also need to install the module
Text::Balanced, which used to be part of this distribution.
In addition, I found an example script for a calculator on the web, here.
Despite it requiring a perl 5.8.0, it appears to work well with perl 5.6.1, if you comment out the "use 5.008;" line. Don't forget to call the sub shell.