Sorry to reply to myself, but I think I found something. The problem seems to manifest itself more clearly when using the /i modifier and the regex fails. It seems the engine is wasting a lot of time normalizing case.

$s = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxyyyRRRyyyy\n" x 300;
$n = 0;
$n++ while ($s =~ /(.*?)RRRR/isg);
print "$n matches\n";
[download]

To summarize: 5.6.1: 0 matches, 0.32 s; 5.8.0: 0 matches, 2.2 s.

But note that if I change the regex to /x(.*?)RRRR/isg the results are reversed: 5.6.1: 9.2 s; 5.8.0: 1.4 s. That's because now 5.6.1 can't get away with the fake anchor. Interesting...