ORM for XML to DB - does one exist?

learnedbyerror has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

I am a long time user of perl, since v3, and a long time lurker here on Perl Monks. For the first time, I have not be able to find an answer or at least a pointer to get to the an answer my question. I guess that means my perl wisdom may finally be reaching a higher level :)

In any case, I am working on a program that queries publicly available data from multiple sources on the internet and loads it into a relational database for analysis. Most of these return the reply to their query in XML. Most of the sites provide a brief explanation of what a query will return; however, none, as of yet, have a published XSD. With a few spits and spurts, I have been successful in parsing the responses using XML::LibXML and then loading the data into the database with DBI. However, I am currently hand coding the XML to SQL mapping and the related inserts/updates. I do not like this pattern because it means that I have to embed my mapping and update/insert logic in code for each source. This does not scale well and will create a support headache down the road.

I am doing some testing with DBIx::Class to clean up at least the insert/update portion of the code and this will be a significant step if it works out right. But DBIx::Class got me thinking about my question:

Is there a perl module or pattern that I can use to map the XML to the database so that implementation is simply configuration and execution is a call?

I have spent a lot of time over the last week investigating here and across the internet; but I have not found much. I did find a few old modules, 8 - 10 years old that attempted to do this; however, none appear to have gotten off the ground.

Any guidance in this direction with be appreciated!!

lbe

Comment on ORM for XML to DB - does one exist?

Replies are listed 'Best First'.
Re: ORM for XML to DB - does one exist? by bart (Canon) on Jan 29, 2011 at 23:13 UTC
I recommend using SQL::Abstract, possibly on top of DBIx::Simple. In this case, inserting the data into the DB is as simple as: `$db->insert($table, \%row);` [download] where %row is a hash representing the row data, with the field name as key and the value as value. All you still have to do is extract the data from the XML and loading it into the hash (one row at a time). If the data comes out of a web page, perhaps there's only one record per XML file? In that case, you might get along with just using XML::Simple, which does convert XML into a simple Perl data structure. If you have the problem that the XML tagnames and the database field names don't fully agree, you can convert from one to the other using a mapping hash. For example if the XML has a tagnames (just making some names up now) "fullname" and "product-id" and the DB has the equivalent column names "name" and "product_id", you can map one to the other using `%map = ( 'fullname' => 'name', 'product-id' => 'product_id' );` [download] and you can do, getting the data out of %xmldata: `my %row; @row{values %map} = @xmldata{keys %map};` [download] after which you can do the insert. If XML::Simple will not do you can use a module like XML::Twig (or my own XML::Parser::GlobEvents, which I wrote some 10 years ago, after the docs of the former module didn't make any sense to me) to build the data structure while parsing the XML.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re: ORM for XML to DB - does one exist?
by bart (Canon) on Jan 29, 2011 at 23:13 UTC

SQL::Abstract

DBIx::Simple

$db->insert($table, \%row);
[download]

All you still have to do is extract the data from the XML and loading it into the hash (one row at a time).

If the data comes out of a web page, perhaps there's only one record per XML file? In that case, you might get along with just using XML::Simple, which does convert XML into a simple Perl data structure.

If you have the problem that the XML tagnames and the database field names don't fully agree, you can convert from one to the other using a mapping hash. For example if the XML has a tagnames (just making some names up now) "fullname" and "product-id" and the DB has the equivalent column names "name" and "product_id", you can map one to the other using

%map = ( 'fullname' => 'name', 'product-id' => 'product_id' );
[download]

my %row; @row{values %map} = @xmldata{keys %map};
[download]

If XML::Simple will not do you can use a module like XML::Twig (or my own XML::Parser::GlobEvents, which I wrote some 10 years ago, after the docs of the former module didn't make any sense to me) to build the data structure while parsing the XML.

[reply]
[d/l]
[select]