But what is the goal of having the poll? I can think of two reasons: 1) you have an interest in the outcome of the poll; that is, you want to result to reflect the opinion or experience of your visitors. Or 2) you do it to attract people to your site, or to enhance their visit with the voting experience.
In case 1) shutting out people is as bad as having people vote more than once. In case 2) shutting out people is worse than having people vote more than once.