If you read perlvar carefully it covers this:

Similarly, a value assigned to $) must also be a space-separated list of numbers. The first number sets the effective gid, and the rest (if any) are passed to setgroups(). To get the effect of an empty list for setgroups(), just repeat the new effective gid; that is, to force an effective gid of 5 and an effectively empty setgroups() list, say $) = "5 5".

So you probably want "$gid $gid" to drop all groups but $gid. And the truly paranoid will check $! afterwards . . .