in reply to Basic Perl trumps DBI? Or my poor DB design?

For a single-tasking, (ie. no concurrency issues) application, a well-written, non-DB (RDBMS) solution will always be able to out-perform an RDBMS solution for any given task.

Simple logic. Whatever steps are required to produce a given set of results, all those steps still need to happen regardless of whether you code them yourself, or whether they are performed by the RDBMS on your behalf.

When you code them yourself, you can optimise those steps specifically to the your requirements.

When the equivalent steps are performed by the DB, they will be optimised for the general case. I addition, the RDBMS will also have the overhead of:

  1. Translating SQL (or other data manipulation language) into it's internal form
  2. Optimising the query.
  3. Communications
  4. Locking
  5. Security

However, performance is not the primary reason for storing your data in an RDBMS--all of those overheads are.

Once you store your data in an RDBMS, when a second application comes along that needs to access the same data in a different way, or combine it with other data:

  1. You no longer have to write the code to store, access, search, format or update that data.
  2. You no longer have to optimise that code.
  3. You no longer have to implement, test, debug your own bespoke communications protocols to retrieve your data from remote locations.
  4. You no longer have to implement, test, debug your own locking to achieve concurrency.
  5. You No longer have to implement, test, debug or administer your own security protocols.

These are all very good reasons for using an RDBMS. It's not all sweetness and light. SQL is a pain, normalising data to allow for efficient access for conflicting uses is a (black) art form. Database optimisation is hard, simply because you do not have full control, and there are many conflicting issues that need to be considered.

The very worst reason for considering moving data into an RDBMS is ultimate performance. That's not to say that achieving a reasonable performance isn't possible with an RDBMS. Nor even that, given the amount of effort expended by generally very clever people, over long periods of time, to develop, refine and tune RDBMS performance, that it won't do a better job, more quickly, than most of us will be able to achieve at our first (or even second or third) attempts at a given problem.

It just means that you cannot have all of the additional benefits that you get from using them, without paying some price; and that price is a reduction in ultimate performance.


Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
"Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
  • Comment on Re: Basic Perl trumps DBI? Or my poor DB design?

Replies are listed 'Best First'.
Re^2: Basic Perl trumps DBI? Or my poor DB design?
by jfroebe (Parson) on Oct 23, 2004 at 15:02 UTC
    For a single-tasking, (ie. no concurrency issues) application, a well-written, non-DB (RDBMS) solution will always be able to out-perform an RDBMS solution for any given task.

    Not necessarily. In over simplified theory you would be correct but, what if you have terabytes or exabytes of data to search?

    You make the inaccurate assumption that DBMS == RDBMS, which is not true. It may be that 90% or more of all databases are built around the relational model, there are plenty that are not.. consider DataWarehousing, an often misunderstood concept. RDBMS solutions rarely fit the needs of Datawarehousing. What about ODBMS (object database management systems) such as Poet?

    Jason L. Froebe

    No one has seen what you have seen, and until that happens, we're all going to think that you're nuts. - Jack O'Neil, Stargate SG-1

      No over simplified theory. The volume of data is irrelevant. A "well-written" application wouldn't search, it would index.

      That is no inaccurate assumption, it is a deliberate limiting of scope of my assertion.


      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "Think for yourself!" - Abigail
      "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon

        A "well-written" application wouldn't search, it would index.

        Now I understand your reasoning... you really aren't talking about the use of a DBMS or not.

        It would be more accurate for you to be suggest the embedding of a simple database-like mechanism (a.k.a. DBMS) into the application level rather than accessing a DBMS backend (i.e. MySQL, Postgres, Sybase ASE, Oracle, etc).

        That's the crux of your argument: embedding is always better than not with single user access. I believe the cliche "it depends" applies here. It depends on the type of data, the amount of data, what you're going to be doing with the data, etc. There is much to consider before choosing between an embedded dbms or a backend dbms (and also which dbms as well).

        please don't misunderstand me, you bring up excellent points but I don't think you were conveying them like you intended.

        Jason L. Froebe

        No one has seen what you have seen, and until that happens, we're all going to think that you're nuts. - Jack O'Neil, Stargate SG-1