I'll admit, I skimmed your requirements, and focused on the problem of diffing data files rather than specifics.

That being said, you're combining two different problems:

The first is a complex problem, and if you're trying to do it in a simple fashion, you should probably use some kind of relational database to solve it. Others have already addressed that in previous postings

You never made mention of the size of the data or performance requirements. Comparing data files against eachother becomes a very expensive process as the files get large. You mentioned that you only want to process one line at a time for each file. How do you know you're on the right line? How do you know the file is sorted properly?

The solution we used worked for us, because we needed to know if *ANYTHING* changed. That, and it had to be lightweight and extremely fast. User-friendliness wasn't an issue, because all it had to do was generate a file of SQL statements.

The issue of user-friendliness is a totally separate problem. It sounds to me like your web-based UI should be a very simple application that presents sort/diff criteria to use user, and passes those criteria to your data-processing package. In your place, I'd use CGI::Application to create that lightweight front-end for the user.

While your other comments aren't relevant anymore, I feel like I should answer them in some part. sed was not being used to parse the CSV, only to re-format it into something that made life easier for sort.

Newlines were never an issue, because newlines aren't legal CSV. When newlines appeared, we complained to the vendor to get them fixed. If they didn't fix it, we made a note of the processing error and chucked the row.

sort takes delimiter arguments - that is how we were able to sort on an arbitrary field.

Whatever you end up doing, good luck - it's definitely a challenging problem.

In reply to Re^3: CSV Diff Utility by swngnmonk
in thread CSV Diff Utility by Limbic~Region

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.