background

sample scenario one

My current boss said the following:
5. We should revisit all error messages currently being generated to determine if they adequately describe errors that occur and are sufficient for Sean and Scott to perform remediation.

This implies that all error messages are searchable and typed. In the best case, I would be able to list all error messages and list each messages attributes, ie, loglevel, which module has the message, etc.

sample scenario two

Now, in my job before this, someone else (thank god it was not me), had to answer this following question:
How many different queries are we using in our system (which was a huge system with easily 1000 different queries)? Also, what tables are these queries accessing? How many of our queries are SELECT statements? How many are UPDATE statements?

Again, the questions imply that all SQL queries in the Perl application are searchable.

While I never ever had to solve this problem myself, being a forward-thinking kinda guy, I went ahead and wrote SQL::Catalog in anticipation of one day wanting a means of cataloging SQL queries.

questions

  1. When someone writes code, even though searchable "metataggable" attributes of the code might be there, they probably dont stop to think about such things nor make the code searchable as such. How much of a program should consist of Structured, Typed, Searchable Catalogs (e.g. SQL::Catalog has done this to an extent for SQL), what other domains should we do this for? In fact, perhaps it should be a shame to write comments in code. Perhaps we should create database-ready attributed descriptions of each line so that such program characterization is possible.
  2. Have you ever made a program catalog? Would you use Class::Phrasebook to build your catalog? Or would you create a set of database tables?

conclusion

Perhaps I am appearing to be overly anal. Overly paranoid. Overly structural. Overly typed. But it took me 30 minutes and grep to do what might have taken a query of some sort had my code been structured and typed and described to begin with.

Carter's compass: I know I'm on the right track when by deleting something, I'm adding functionality

Replies are listed 'Best First'.
Re: [meditation] Structured, Typed Program Catalogs
by dws (Chancellor) on Nov 18, 2002 at 19:17 UTC
    Perhaps I am appearing to be overly anal. Overly paranoid. Overly structural. Overly typed.

    Perhaps so. Perhaps you're being too specific in your solution.

    On any sizeable project I've been involved with, we eventually "externalize" all significant strings into a set of repositories, where the repository can be a simple flat text file with (key,value) tuples. This process, which sometimes happens very early on in development, is part of "internationalization"(I18N) -- those steps you take to prepare for eventual "localization" (L10N) of the application for some target language. Once you have strings in repositories, you can switch in translated versions of the repositories to localize.

    If you're careful where you put externalized strings, "types" (classifications) can be determined either by which section of a repository they're in (e.g., group all SQL queries together in one section, and error messages in another) or by which repostitory they're in (e.g., keep separate repositories for different types of strings).

    While it's common to externalize end-user visible text, it's a little less common to externalize SQL. But if your application is well structured, SQL should be limited to a set of readily identifiable modules.

[reply]
      While it's common to externalize end-user visible text, it's a little less common to externalize SQL. But if your application is well structured, SQL should be limited to a set of readily identifiable modules.
      But then how do you answer the type of questions I mentioned? Ie, what types of queries, what tables are they querying. Also, when you have > 1000 queries (and queries can be quite large), they take up memory space and should be externalized. I list all the motivations for SQL::Catalog in its docs. Kind of in a hurry now...

      Carter's compass: I know I'm on the right track when by deleting something, I'm adding functionality

[reply]
        I like to keep it simple. Personally, for me I usually store all of my SQL in a seperate module for easy swapping.
        If I had a 1000 queries, I would probably use a tied BerkeleyDB hash instead to cut down on load time, etc.
        If I had to figure out what was doing what, I would probably write an SQL parser, (probably building off of SQL::Statement or similar) running it on all the queries in the module and verifying the unparsable ones by hand.

        -Lee

        "To be civilized is to deny one's nature."
[reply]
Re: [meditation] Structured, Typed Program Catalogs
by perrin (Chancellor) on Nov 18, 2002 at 21:44 UTC
    I don't know, I think 30 minutes of grep might be less pain than the time it would take to try and anticipate everything you might be asked some day and keep it in a separate database. I would certainly separate content like error messages, because I expect someone else will want to edit them, but I wouldn't go to that much trouble with much else unless I knew for sure it would pay off. The extra complexity will not be worth it in most cases. To me, this is a case "You aren't gonna need it."
[reply]
Re: [meditation] Structured, Typed Program Catalogs
by Abigail-II (Bishop) on Nov 19, 2002 at 13:50 UTC
    Statistics on the SQL queries? That should be easy. Just turn on the appropriate feature of your database such that all queries are logged, and run the test suite of your program(s). You do have a good test suite that runs all parts of the code, don't you?

    I would also assume that any large program doesn't contain any SQL code - that all SQL code is found in a separate layer. Preferably two layers, one in "application space", in a library (or module), which calls stored procedures in the database. The stored procedures themselves will contain the queries. Programs should never touch tables themselves - that's as dirty as using global variables, or accessing attributes of alien classes directly.

    Abigail

[reply]
      Resonable, IF your database supports stored procedures.

      I don't think mysql supports them yet, does it?

      Still, you can get most of the same benefits by having a SQL details abstraction layer in Perl.
      --
      Mike

[reply]
        That just shows that MySQL is just there for toy applications. Noone is interested in a break-down of SQL queries of (or has 1000 queries in) a toy application.

        I just find it mind boggling people use MySQL in applications they take serious. But then, people run services they find important on Windows machines as well.

        Abigail

[reply]

Back to Meditations