As previously said by other monks, you don't provide enough information, but depending on the approximate ratio of lines that get updated between two runs, you might just start by comparing the full lines and decide to split the lines and compare the individual columns only for those lines which are different.

Otherwise, we don't have enough details about your procedure and your data, but, in general, comparing 20,000 lines in less than 2 minutes seems to be a very realistic aim (if coded reasonably efficiently). I am quite often comparing 30 million lines in 10 to 15 minutes or even less if the comparison to be performed is simple or the lines relatively short, on a platform which is far from being a racing horse.

Finally, as already pointed out, if you open your files in read mode, there is no danger to alter them. But show your code to confirm this as well as my previous (quite general) comments.

In reply to Re: Compare CSVs FILES using REGEX or pattern matching by Laurent_R
in thread Compare CSVs FILES using REGEX or pattern matching by edge99off

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.