The Devel::SawAmpersand module has a bit more info on why:

There's a global variable in the perl source, called PL_sawampersand. It gets set to true in that moment in which the parser sees one of $`, $', and $&. It never can be set to false again. Trying to set it to false breaks the handling of the $`, $&, and $' completely.

If the global variable PL_sawampersand is set to true, all subsequent RE operations will be accompanied by massive in-memory copying, because there is nobody in the perl source who could predict, when the (necessary) copy for the ampersand family will be needed. So all subsequent REs are considerable slower than necessary.