Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid

remove duplicates from array

by mkmcconn (Chaplain)
on Feb 22, 2001 at 02:58 UTC ( #60068=snippet: print w/replies, xml ) Need Help??
Description: Evolution of a solution

Combing code, I found I have used these 5 different solutions to a common task for my work:
   A command returns an unordered list of integers, with some duplicates.
   Remove duplicates (after sorting the list).
   (Listed below from older to more recent):


#!/usr/bin/perl -w
use strict;
use Benchmark;
my $n     = 10000;
my $o     = rand($n);
my @rray  = (1..$n,$o..$n);     #  make an array and pollute it
                                #  with duplicates
my @array = ();

@array    = sort {$a <=> $b} @rray;

my %tests = (
    totemp => sub {
        my @newarray    = ();
        my $i           = 0;
        for (  @array ){ 
          my $temp  = $array[ ++$i ] ? $array[ $i ]: "";
          push @newarray, $_ unless $_ eq $temp;
    ternary => sub {
        my @newarray    = ();
        my $i           = 0;
        for ( @array ){ 
          push @newarray, $_ unless $_ eq $array[ $array[ $i+1 ] ? ++$
+i : 0 ];
    toend => sub {
        my @newarray    = ();
        my $i           = 0;
        for my $list (@array){
           if ( $array[ ++$i ] ){
              push @newarray, $list unless $list eq $array[ $i ]; 
              push @newarray,$array[-1];
    cookbook => sub {
        my @newarray    = ();
        my %duplicates  = ();
        @newarray       = grep { ! $duplicates{$_} ++ } @array;
    modulus => sub {
        my @newarray    = ();
        for (0..(@array-2)){
             $array[$_] % $array[$_+1] and push @newarray, $array[$_];
        push @newarray, $array[-1];

timethese ( shift @ARGV || -5, \%tests);

Thanks chipmunk
ternary is a bizarre solution - and as you discovered (and I did not realize, but it's obvious looking at it now), it breaks completely when there's a 0 in the array! The original was echo'd from the commandline, and appears to have been used to remove duplicates from a list of words.
modulus also works as long as there is no zero in the list. If there is more than one zero, it will not run. If there is one zero, it will be skipped.
Thanks for taking time to comment, chipmunk. Some small adjustments made in the post, to prevent complete failure of the tests.

Replies are listed 'Best First'.
Re: remove duplicates from array
by chipmunk (Parson) on Feb 22, 2001 at 10:11 UTC
    Unfortunately, ternary and modulus return incorrect results when the array contains 0. ternary pushes an extra 0 into the results, because the conditional operator doesn't distinguish between 0 and undef.

    If 0 is the first element in the array, modulus omits it from the results. If the array starts with a negative integer and contains 0, modulus tries to divide by zero and dies horribly.

    These solutions could be fixed with some extra conditionals, but I think the other solutions are better.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: snippet [id://60068]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (2)
As of 2023-06-03 19:13 GMT
Find Nodes?
    Voting Booth?
    How often do you go to conferences?

    Results (16 votes). Check out past polls.