Well you say I'm inclined to add everything I can find, so that could in principle go far beyond users who have explicitly made a link in the other direction.

Privacy is a fraught business. Making this an option available to people sounds like a good thing, but my cautionary instincts suggest making it opt-in would be a wise approach.