What you describe is accomplished not by using a modifier, but by being explicit in the definition of the char class:

m/0x[\da-fA-F]{1,4}/;
[download]

There's no reason to have or use modifiers on only part of a regex, when one can simply use the mechanisms already available to accomplish the same goal.