My target is Linux so I think the point about compiling dlls is moot although please correct me if I am mistaken.

Moot how?

Why is it desireable that each CWD gets a copy of the module built for the occasion?

Why is acceptable that "installing" a module isn't the end of the installation process?

Oh look, maybe https://metacpan.org/pod/Inline#WRITING-MODULES-WITH-INLINE is slightly better than more appealing than my advice ... its certainly better than recompiling the module after installation ... but it adds a dependency for end users on Inline which isn't always easy to install (only author needs Inline to generate the XS, everybody after author doesn't need it)