Well I was looking for a command-line solution, which could also work within the debugger...

And after investigating Pod::Man I discovered that the X-tags (and more) ARE actually (almost correctly) exported to man's *roff- format.

lanx@nc10-ubuntu:/tmp$ grep 'X<qr>' perlop.pod
X<qr> X<s> X<tr> X<'> X<''> X<"> X<""> X<//> X<`> X<``> X<<< << >>>
X<qr> X</i> X</m> X</o> X</s> X</x> X</p>
[download]

lanx@nc10-ubuntu:/tmp$ grep 'Xref' perlop.man |grep qr
.IX Xref "operator, quote operator, quote-like q qq qx qw m qr s tr ' 
+'' "" """" ` `` << escape sequence escape"
.IX Xref "qr  i  m  o  s  x  p"
[download]

So it's more a question about finding an appropriate viewer/pager to be able to jump into the correct line.

Identifying the file is no big deal...