The problem seems to be that $modifiers is not interpolated in time for the compiled regex to have seen it. This will work,

if ( $text =~ m/(?$modifiers)$pattern/ ) {
    print "it matches\n";
}
[download]

That is a facet of "Cloistered Pattern Modifiers", as covered on p186 of the Camel Book(3ed.).