One and a half passes.

my ($i, %seen) = (0);

my @no_dup =
    grep { 2 < push @{$seen{/(pattern)/}}, $i++ }
    @with_dup;

@no_dup = map { 2 < @$_ ? $no_dup[$_->[0]] : () } values %seen;
[download]

Makes use of the little known return value from push but probably no more efficient than the two pass solutions due to creation of a lot of anonymous arrays. Hmm.. I can do better..

my ($i, %seen) = (0);

my @no_dup =
    grep { -1 != ($seen{/(pattern)/} = exists $seen{$1} ? -1 : $i++) }
    @with_dup;

@no_dup = @no_dup{grep $_ != -1, values %seen};
[download]

Update: fixed a thinko.

Makeshifts last the longest.