My first instinct is to tell you to use split for the second parsing action.

Now, a big question I have is what does "duplicate" mean? For example,

<17>11,2002,FORD    ,ECONO CO,        ,34567676xxxxx   
<17>11,2002,FORD    ,ECONO CO,       ,34567676xxxxx
[download]

Are those duplicate? If those are not duplicate strings, then:

my %seen;
while (<TEMP>)
{
    next if $seen{$_}++;

# Rest of loop here.
}
[download]

However, if those are considered duplicates, you have to do duplicate checking in your second parse statement. Do something similar with the %seen hash. :-)

------
We are the carpenters and bricklayers of the Information Age.

Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.