The problem with basing it off week numbers is that week numbers are based on weeks starting on Monday, not Sunday, so you would need to check that your start date is not already a Sunday before continuing.
Is this purely an intellectual exercise or is there a reason you don't just subtract the right number of days from your start date to get the beginning of the week?