Well, in my case it's wife, and it should actually read, "I only sleep with..."

"mostly" doesn't have the connotation of exclusivity that would be more accurate. :)