I'm just guessing, but it sounds like you want something other than regular expressions. It feels like eventually you'd like to support nested (un)balanced expressions -- which I don't think you'll get to work with REs. Maybe Text::Balanced or something like it would help.

LaTeX: ${a^nb^n\inL(G)}$