Better is a question answered utilizing criteria, and its unclear what has led you to seek a replacement... perhaps an 'ility' like maintainability, or perhaps scalability or extendability.

So, to focus this discussion a bit more, could you comment on what 'better' means to your team and company? Often we think of the 'triangle' of Cost/Features/Schedule and the trade-offs one makes to add or subtract one of these 'qualities'.

Could you share a bit more about what led to your decision to replace vs. refactor, and what is important to your team, what are the relative costs of hardware / software to the problem.

Long, big, complex jobs on huge server farm (think, Census data mining) presents a much different problem than quick, distributed, jobs that are isolated to prevent data corruption.

I'm personally interested in the reasoning behind the 'Temp DB' approach, and what, if any, transactions post back to the 'master' database.

I'm guessing these multiple-scripts grew up around a need to be fault tolerant and insure consistency in the data model within a database not using constraints.

If so, your design should probably begin by clearly identifying your capacity, concurrency and consistency related design constraints, and building your architecture based upon these requirements withing the budget and schedule you have available.

Depending on the degree and type of parallelism involved (if any) there might also be a need to focus on a job control framework (such as Load Sharing Facility aka LSF) to simplify the chain of dispatch/setup/execute/return_results/cleanup. If a significant portion of your code is focused on the 'control' issues, and the actual 'work' is a much smaller piece, then a batch processing framework could save you from reinventing the wheel.

You tell us!

-- Patrick Hennessey

"Strictly speaking, there are no enlightened people, there is only enlightened activity." -- Shunryu Suzuki

In reply to Re: SQL execution engine - best approach? by spectre9
in thread SQL execution engine - best approach? by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.