In a perfect world, I would agree with you.
The application layer should be where all changes are handled and tracked, but too often that is not the case.
At my company, the source system has been through several different applications, with data from each still being used by some users. Also, there are folks with direct access to the data (via sql) who perform updates directly without benefit of an application.
That leaves me (and my teammates in the data warehouse) with 2 choices: rely on the change/update dates in the source system and just accept that there are changes we won't track, or use some kind of Changed Data Capture (CDC). We have survived for 3 years on the former, and are now trying to institute the latter.
CDC can take several forms, but the best, in my opinion, is one that uses the database engine itself. The database will generally do logging of transactions, and scanning that transaction log for the tables you are interested in tracking is the best way to capture all changes.
We have gone from full table scans of the last 3 days worth of changes, and still missing a good percentage of them, to capturing 100% of the changes we are interested in and loading 25% of the data we did before.

Plus, we took the burden off the source system and put it on our newer servers.

In reply to Re^4: Conditional many to many relationships with Class::DBI by poqui
in thread Conditional many to many relationships with Class::DBI by BigLug

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.