What version of Perl are you running that you have to use $'? Explicit backreferences have worked for ages. Just use a pattern like /^#(\d+)/ (modify to taste) and assign $1.

Using $' or $` anywhere in your program slows down *every* RE possibly dramatically. So Don't Do That.