As others have pointed out, the key here is [^'] to keep inside the '....'.

This is a little shorter:

  while ($value =~ s/href='[^']*?\K<br>//gsi) {} ;
[download]

but requires n+1 passes across $value, where n is the maximum number of <br> between href' and '.

Or, following massa, above:

  $value =~ s{href='[^']*?\K<br>([^']*)}{local $_ = $1, s/<br>//g, $_}
+gsie;
[download]

which does one pass only, and only processes any href'...' which contain one or more <br>. (A (?=') before the first } will stop it processing an unterminated href'...', should you care.)

And that fills my quota of regex finagling for the day.