Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re: SQL execution engine - best approach?

by spectre9 (Beadle)
on Feb 18, 2009 at 00:22 UTC ( [id://744597]=note: print w/replies, xml ) Need Help??


in reply to SQL execution engine - best approach?

Better is a question answered utilizing criteria, and its unclear what has led you to seek a replacement... perhaps an 'ility' like maintainability, or perhaps scalability or extendability.

So, to focus this discussion a bit more, could you comment on what 'better' means to your team and company? Often we think of the 'triangle' of Cost/Features/Schedule and the trade-offs one makes to add or subtract one of these 'qualities'.

Could you share a bit more about what led to your decision to replace vs. refactor, and what is important to your team, what are the relative costs of hardware / software to the problem.

Long, big, complex jobs on huge server farm (think, Census data mining) presents a much different problem than quick, distributed, jobs that are isolated to prevent data corruption.

I'm personally interested in the reasoning behind the 'Temp DB' approach, and what, if any, transactions post back to the 'master' database.

I'm guessing these multiple-scripts grew up around a need to be fault tolerant and insure consistency in the data model within a database not using constraints.

If so, your design should probably begin by clearly identifying your capacity, concurrency and consistency related design constraints, and building your architecture based upon these requirements withing the budget and schedule you have available.

Depending on the degree and type of parallelism involved (if any) there might also be a need to focus on a job control framework (such as Load Sharing Facility aka LSF) to simplify the chain of dispatch/setup/execute/return_results/cleanup. If a significant portion of your code is focused on the 'control' issues, and the actual 'work' is a much smaller piece, then a batch processing framework could save you from reinventing the wheel.

You tell us!

-- Patrick Hennessey

"Strictly speaking, there are no enlightened people, there is only enlightened activity." -- Shunryu Suzuki

  • Comment on Re: SQL execution engine - best approach?

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://744597]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others lurking in the Monastery: (3)
As of 2024-04-24 04:20 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found