Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

How to display all the content of the database?

by Alex the Serb (Monk)
on Jan 09, 2003 at 11:48 UTC ( #225509=perlquestion: print w/replies, xml ) Need Help??

Alex the Serb has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

haven't wrote for a while! (hope my popularity went up :))

Anyway, this is general database problem and hopefully it can be done using DBI modul for Perl.

For example, database is a catalog of all of departments of some company (no matter which one). Each department can contain divisions, each division can contain something, and it goes like this for unknown number of times. The depth level is unknown. The script must be general in a way that it displays whole content no matter how deep it goes.

Any ideas?

  • Comment on How to display all the content of the database?

Replies are listed 'Best First'.
Re: How to display all the content of the database?
by davorg (Chancellor) on Jan 09, 2003 at 12:17 UTC

    Well, something is going to need to know about the structure of your database (even it it's just a list of tables to dump). This knowledge can be encoded in one of two places. The most common is that your program needs to understand the relationships between the various tables.

    The more flexible and generic (but far less common) solution would be if the database designers encoded this information in the database itself by using referential integrity constraints on all of the tables in the database. In this case you would be able to query the metadata tables in order to construct SQL to query the real tables. One possible problem is that the metadata tables vary a lot between different database systems - so a solution like this wouldn't work on different systems. I believe that the DBI team are working on adding generic metadata support to the DBI spec, but I don't know how far they've got or how well supported it is in individual DBD modules.

    Executive summary - you probably can't do it.


    "The first rule of Perl club is you do not talk about Perl club."
    -- Chip Salzenberg


      sector(s_id, d_id, name);
      department(d_id, dd_id, name);
      division(dd_id, ... , name);
      goes for unknown depth.

      I thought that it have something to do with data dictionary, but can it be done using DBI. DBI is independent of database, does DBI have some part that addresses common things of every data dictionary, like geting informations about relations between tables, informations about primary and foreign keys ...

      Am I more specific now?

      Some people did this, but they charge money for this!

      Anyway, I tried this but using Perl's Pg modul for PostgreSQL. There is table pg_databases, pg_tables and stuff like that from which you can determine some of stuff, but can it be done using DBI and MySQL. And it have to be general.

      If the company add some new relation that is in relationship with another one, it should be displayed without changing the script.

        We are making some progress here.

        However, I have some concerns.

        You say:

        sector(s_id, d_id, name); department(d_id, dd_id, name); division(dd_id, ... , name);

        That looks a bit backwards from a normalization point of view. It appears that the higher level entity contains a pointer to the lower level entity, which is exactly backward from how it ought to be.


        sector(s_id, name, ...) department(d_id, s_id, name, ...) division(dd_id, d_id, name, ...)

        In this rough schema, a department has a sector ID. In your schema, either a sector has to have multiple columns for department IDs or there have to be duplicate sector entries if a sector has more than one department.

        Now, to get back to your actual question, if your requirement is simply to be able to use DBI, then without prior knowledge of the set of tables, you cannot depend on being able to discover the relationships, since not all DBDs offer that kind of support (DBD::CSV, DBD::RAM, for instance).

        If you must use DBI with any arbitrary DBD for the data store, you will have to know up front the set of tables to examine. You won't even be able to count on getting a list of tables in the "database".


        does DBI have some part that addresses common things of every data dictionary, like geting informations about relations between tables, informations about primary and foreign keys ...

        I think I already answered that. Please reread my previous answer substituting your phrase "data dictionary" for my phrase "metadata" - they mean the same thing :)


        "The first rule of Perl club is you do not talk about Perl club."
        -- Chip Salzenberg

Re: How to display all the content of the database?
by rdfield (Priest) on Jan 09, 2003 at 11:59 UTC
    Use the 'connect by' SQL construct if they're all stored in the same table and there is a parent/child link defined.


Re: How to display all the content of the database?
by CountZero (Bishop) on Jan 09, 2003 at 21:27 UTC

    Just a wild thought: why would one like/need to have so many different tables (one for each level)? I would rather have one table which in each record would hold its own ID, its level number (say: section = 1; department = 2; ...), its name and the ID-number of its parent (and any other info you need to store for this entity.

    Traversing such a structure or selecting would be rather simple with the help of some well chosen SQL-queries:
    SELECT name FROM organization WHERE level = 3 and parent = 10
    would give you all level 3 entities which are overseen by entity 10.

    Finding out how deep your organization is, is as simple as finding the maximum of the level numbers.

    You could store meta-data about the levels (not the individual entities) in a separate table, the ID of which is of course the level number, ... etc. etc.

    No need to access the meta-data of the database and you can implement it in any type of database and use DBI::DBD in a portable way.

    Of course if the structure of your organization is much more complex, you can always try to catch it in an XML-structure, which is more flexible but more difficult to use.


    "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

      I agree with you completely, seems that the person who designed the database has no database experience....

      In Oracle I would do:

      create table organization( orgid number not null primary key, ordparent number, orgname varchar2(40) not null); alter table organization add constraint fk_parent foriegn key (orgpare +nt) references organization(orgid); -- start inserts for organizations insert into organization ....etc. -- end inserts for organizations create table orglevels( levelid number not null, levelname varchar2(20) not null); insert into orglevels values (1,'Sector'); insert into orglevels values (2,'Department'); insert into orglevels values (3,'Division');

      You would probably get lots of data by running a query like this one (untested):

      select orgid,orgname,orgparent,level,l.levelname from organization o s +tart with orgparent is null connect by prior orgid=orgparent left join orglevels l on l.levelid=level
Re: How to display all the content of the database?
by Starky (Chaplain) on Jan 09, 2003 at 18:46 UTC
    Rather than answer the question directly, I'd like to ask some questions about the question.

    Display of "all the content of the database" "no matter how deep it goes"? If you want a generic script that recurses through foreign keys to display data, then consider 2 possibilities:

    1. Your database has a large number of tables with a large number of associations. You will potentially be writing out a massive stream of data. Why would you want to have such a massive file? Either for archiving or for a human to use.
      • Is it for a human to look at? How many trees would you sacrifice to print out a sizable chunk of a 5 gig database? How would a human find something in the file? Wouldn't it be better just figuring out what the human wants to know and writing some nice queries for them?
      • Is it for archiving? Then why not just dump the database (e.g., via mysqldump)?
    2. Your database is small with a small number of tables. In this case, why not just write a one-off script? You can change it later if you need to rerun it after a change to the schema is made. Trying to conceive of the general case with a recursive search through the data dictionary just seems like it will take alot of time. And recursive code is never the most readable if someone else needs to work with it later. There is a time and place for quick one-off hacks, and this may be one of them.

    Although I don't know exactly what it is you are trying to do, my impression is that reanalyzing your initial approach may be fruitful.

    Hope this helps :-)

Re: How to display all the content of the database?
by Alex the Serb (Monk) on Jan 10, 2003 at 08:06 UTC
    Thanks everyone, but I somehow think that everything is not so clear yet.

    Its not the question of which company it is, the script should work for any company or any catalog of some store or anything else. The large number of data would probably be limited with LIMIT and OFFSET clauses. But, I'm under impression that the most of you are telling me that this could be done with SQL without Perl? I have a version of MySQL which does not support subqueries. The script should work on any DBMS and use common stuff to any DBMS. The script should be generic and work in a way so that if anyone transfers it to store/stores that have different catalogs of different stuff, with different set of tables, it would still work perfectly.

    For example, if we have:

    section(s_id, ...);
    division(d_id, s_id, ...);

    the script should give:
    sector 7 ... division 1 ... division 2 ... ... ... sector 8 ... division 1 ... ... ... ... ...
    Then one day, they decide to add a new type of entity called department (its just a new level of depth for the script, nothing more!)

    The script should then give:
    sector 7 ... division 1 ... department 1 ... department 2 ... ... ... division 2 ... department 1 ... ... ... ... ... sector 8 ... division 1 ... department 1 ... ... ... ... ... ... ...
    Of course some of divisions would not have departments, but that does not matter for the script, the script would display it in the correct way, because its generic. And even if some day someone add a new type of entity it would still work correctly :)

    I know this is huge problem and I'm not asking you to do it, I'm just asking the small example of it with Perl using DBI, or code example of some idea or at least point me to the right direction :)

Re: How to display all the content of the database?
by OM_Zen (Scribe) on Jan 09, 2003 at 20:31 UTC
    Hi ,

    my $tables = "select column_name ,object_name from all_tab_colum +ns,dba_objects where owner_name = ? and column_name in (select colum +n_name from all_tab_columns where table_name = object_name)";

    This subselect is correlated and shall take some like a few minutes and can be optimized .

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://225509]
Approved by Tanalis
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others surveying the Monastery: (3)
As of 2023-12-06 04:27 GMT
Find Nodes?
    Voting Booth?
    What's your preferred 'use VERSION' for new CPAN modules in 2023?

    Results (29 votes). Check out past polls.