Call it a personal choice, but prefer to keep the DTD separate from the *ML files. This goes back to my SGML days, when I would write a DTD and distribute it to people who would write compliant SGML.

HTML, sadly and frequently, breaks from adherence to the DTD, but XML at least makes an attempt to return to it.

Since one DTD may engender many XML document, I prefer to keep it separate, but as I said in the beginning, it's a personal choice.

What is missing from XML::Twig to make it easier for you to write your code? If you need fixed/default attributes to be filled, I think I could add it.

Sorry for the delay in responding -- I'm the only IT person in my company, and a few days after posting this question I got snowballed -- this is the first chance I've had to get back to my programming.

I'm not sure I can say something is "missing" from Twig. It is currently working just fine -- in my initial testing and such. The only "problem" I am having is a design issue.

The way version-2 of my system works, each data record is in a unique file; each file was plain ascii; each file had a data layout like this:

fieldname|whatever data is in the field\n

Everything worked fine until they started doing the cut/paste thing and dumping in full email text and/or web pages and/or resumes, etc. So I started tinkering about with encoding and such, with limited success.

Then I stumble on this whole XML concept, and realize that this is just what I've been needing to overcome some of the issues I was working with -- so now I'm moving my data to XML (and now it is v3.1), and the layout looks like:

<fieldname>whatever data is in the field</fieldname>

I had a "VOC" file (a carryover from my old PickBASIC days) that had the layout of each "fieldname" -- and converting that to XML (esp using XML::Twig) was a beautiful thing. It now works like a champ, and I love it! There is one XML file (voc.xml) that is easy to read and modify and maintain; reading it into the program is almost too trivial; extracting the layout info is a piece of cake!

Now, I could do as some suggested, and that is have one file per database, and have the multiple records in it, and that'd be okay. I can see how that might be good (and how Twig could be exploited to make that work). But my one-record-per-file concept makes things work well for the web-based projects I am working on (limited record locking issues), so I don't want to walk away from it just yet.

The point that I was looking to get around was that I wanted to make a generic enough data entry and data display routine that would grab any of the records (no matter what the defined layout was) and edit it or display it. To do that, my thoughts were that I had to either: 1) read in and process the DTD, or 2) have Twig automagically read in and process the DTD. But XML::Twig doesn't deal with the DTD, so that was my problem.

So that was the basis of my comments and question. If XML::Twig processed the DTD and could return to me the "complete list" of fields to be expected, then nobody would have heard from me. I would have stuck it into each item, and that would have been that. As it is, I am now getting back to trying to figure out how to grab that info from either the standalong DTD file or the internal DTD info in the XML file.

And since I spent the time since I posted this original message putting out fires instead of working on the project, I am just today starting to look back at that question again.

I guess the bottom line question is: how do folks keep up with the full list of what fields are supposed to be in the XML file?

Lee Crites
lee@critesclan.com

You can wrestle some information out of XML::Twig: you need to use the load_DTD option to read the external DTD, then get the list of elements in the dtd and their models using the model method. It is not perfect, and you then have to do some (simple) post-processing to get just the sub element names.

For example:

#!/usr/bin/perl -w
use strict;
use XML::Twig;

my $t= XML::Twig->new(load_DTD => 1, error_context => 1)->parsefile( '
+test.xml');

my @elts= $t->model;
print "all elements: ", join( ', ', @elts), "\n";

foreach my $elt (@elts)
  { my $model= $t->model( $elt);
    print "model of $elt: $model\n";
    my @subelts= grep { $_ && $_ ne 'PCDATA'} split( /[^\w:-]/, $model
+);  # keep only element names
    # if you want an array of unique element names you then need this
    my %subelts= map { $_ => 1 } @subelts;
    @subelts= sort keys %subelts;
    print "elements in $elt: ", join( ', ', @subelts), "\n";
  }
[download]

Does this help?

If that's the case, then I think a single DTD file per group would be best. And when you use your main script to access ALL groups, I would expect that you just need to create a new Twig object for each group, and access the DTD for that group to know about its structure. (You shouldn't need to add structure to the XML data in order to describe the structure, though perhaps this might not be a bad idea -- assuming it's done consistently across all groups/records so that they all have something useful in common, regardless of anything else.)

The only way it would make sense to have the DTD included with every record is if every record could potentially have a different structure, regardless of what group it's in. (For that matter, I wonder about needing DTD's at all -- I thought the one of the design goals of XML was to make the markup parseable without requiring any "ad hoc" DTD a priori -- in contrast to the more cumbersome SGML.)

In my case, each file will be one record. That record could belong to any "database" I have. I have defined (for myself) certain fields (e.g. 'name' or 'city') that have predefined meaning and a common layout. I put that layout into a global INI type file that all scripts will access. When a new (and unique) field is added to any database, I stick that into the global file as well. So there is one file (voc.xml) that has every field name and all of it's particular layout definitions.

What I am doing is to write a single display and edit CGI script. It will read in the voc.xml file, and then somehow (automagically) know what to do with the record. This worked just fine before the migration to XML, so I know the process works, I'm just trying to find the "best practice" for the XML world in duplicating it.

It seemed to me that the perfect place to do this was in the DTD. Each database would have a DTD file that contained a list of all of the fields it used, etc. But how do I tie the record I am processing to the DTD that relates to it? The internal DTD info seemed like the perfect location for that as well. But XML::Twig didn't process it correctly, and when I looked, I found out that this was a known problem.

But my philosphical question remains: what's the best practice for linking the data file to the DTD file? I can read the documentation, but so far I haven't really gotten a good feel for what is really being done by those folks who are actually using this.

Lee Crites
lee@critesclan.com

"Each record is a separate XML file."

Why? Does it not make more sense to say each table is a seperate XML file. In reality, each logical table could indeed map to multiple physical files.

XML::Simple is easy but does not really provide DTD features. I have similar concerns as you in XML reader and checker and have found XML::LibXML pretty useful.