$texttosub =~ s/(?!log10)|([a-zA-Z][A-Za-z_0-9]+)/$cats{$1}/i
[download]

But I am getting an infinite loop for some reason.

You don't need the alternation operator |. Lookaround Assertions like (?!...) are zero-width (tutorial). So the regex / (?!foo) | bar /x means "match a zero-length string as long as the next thing isn't foo, or match bar".

However, even with the alternation operator removed, the regex still won't do what you expect: the pattern basically means "any letters or numbers, as long as the next thing isn't log10", so the result will be "l(5)", because the substring og10 matches the pattern and there is no key og10 in %cats!

Although I've already said that I agree with Corion that using a proper parser is better, and I've shown a different solution, just for the sake of completeness and TIMTOWTDI, here are two additional ways to do what you want. First, you can use the word boundary \b to make sure that the string being matched isn't just a portion of a longer identifier: s/(?!log10)(\b[a-zA-Z][A-Za-z_0-9]+)/$cats{$1}/ works on the sample input you've shown.

Second, you could look at all identifiers, and then figure out what they are once you've matched them. Here, I'm taking advantage of the /e modifier to execute the replacement part as Perl code:

use warnings;
use strict;

my %cats = (blackcat=>5, whitecat=>10,orangecat=>20);
my %funcs = map {$_=>1} qw/ log10 sin cos /; # etc.

my $texttosub = "log10(blackcat)*whitecat*(log10(orangecat))";

$texttosub =~ s{(\b[a-zA-Z][a-zA-Z0-9_]+\b)}{
    my $repl;
    if ($cats{$1}) { $repl = $cats{$1} }
    elsif ($funcs{$1}) { $repl = $1 }
    else { die "Unknown identifier '$1'" }
    print "Matched '$1', replacement '$repl'\n"; # Debug
    $repl }eg;

print $texttosub, "\n";

__END__

Matched 'log10', replacement 'log10'
Matched 'blackcat', replacement '5'
Matched 'whitecat', replacement '10'
Matched 'log10', replacement 'log10'
Matched 'orangecat', replacement '20'
log10(5)*10*(log10(20))
[download]