In terms of performance, the single most important change to be done is to use a hash enabling direct lookup instead of an array to store ther $xref data, as already suggested by Corion. Depending on the size of the $xref file, this change alone can make your program thousands times (or even much more) faster.

A few other changes might improve your code. Insert these pragmas:

use strict; use warnings;
at the top of all your Perl programs, and declare your variables (with the my keyword).

You don't need to store your customer file into an array (@lines) and then read the array, this leads you to process twice the same data, which is a waste of time. Do your checks when reading your customer file line by line;

Use the three-argument syntax of open and use lexical file handles (see examples below).

This could lead to something like this (untested):

use strict; use warnings; # Read the values from XREF into a hash my $xref = "input_file.txt"; # put here real filename open my $XREF, "<", $xref or warn "Could not open $xref $!"; # A more + detailed message can sometimes be useful my %xreflines = map { chomp; $_ => 1 } grep /\S/, <$XREF>; # popu +lating the hash with the content of the xref file close $XREF; my $goodfile = ...; # insert the name +of the output file here open my $XFILE, ">>", $goodfile or warn "Could not open $goodfile $!"; my $outfile = ...; # insert the name +of the customer file here open my $CUSTOMERFILE, "<", $outfile or warn "Could not open $outfile +$!"; while (my $line = <$CUSTOMERFILE>) { my ($dist, $cust) = (split /;/, $line)[0,1]; # no need to chomp + the fields, there are other fields afterwards in the line my $key = "$dist$cust"; # Building the loo +kup key unless (exists $xreflines{$key}) { # hash lookup print $XFILE "$line"; # printing the lin +es whose ID is not found in the cross ref file } } close $XFILE; close $CUSTOMERFILE;
Update: removed some extra quote marks left out from the OP code when changing the open syntax. Thanks to choroba for pointing out.

In reply to Re: best way to use grep by Laurent_R
in thread best way to use grep by vbynagari

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.