Re: Datastructures to XML

Perhaps it is just me, but this seems quite complicated when the goal is to simplify generating XML for record sets. Many people find looping constructs (foreach, etc) hard enough when they are close to the actual data. Abstracting them via a template I feel would only add to the confusion.

I'm also not so crazy about all of the options to control filtering of hash keys. The problem with templates like this is that you have to anticipate all of the possible key selection behavior. A routine that took a simple filter function, rather than all of those template options, would give you all of the flexibility of grep and let you reuse what you already know about Perl to do the filtering.

For most use cases there are only a handful of structurally distinct ways (about 6) to convert an array of hashes into a set of XML records. Beyond that most of the variation comes either from tag names, or the need to filter out certain hash members. On that basis, I think you could get away with something much more simple than templates.

I personally would prefer a routine that I could call like this:

    my $sXML = genRecord($aData); #default options
    #or
    print genRecord($aData, $hOptions);
[download]

where options let me set something to filter keys, change tag and attribute names, and decide whether field values should be attribute values, or element text. It would be much easier to use (and explain in SOPW replies).

The following module definition (265 lines of POD and examples, 102 lines of code) would accomplish all that:

use strict;
use warnings;
package XML::HoA;

use base 'Exporter';
our @EXPORT_OK=qw(genRecord);

=pod

=head1 SYNOPSIS

  # $aData    reference to an array storing zero or more
  #           hash references. Each hash reference represents
  #           a different record.  The key-value pairs of the
  #           hash store the value assigned to each field in
  #           the record.  The keys are the field names and
  #           the value for each key is the value assigned to
  #           that field
  #
  # $hOptions reference to a hash storing formatting options
  #           for each record.
  #
  #           key        meaning
  #           ----       -------
  #           format     6 possible values - see below
  #                      defaults to NAME_EQ_VAL
  #           record     tag name for XML element storing
  #                      each record - defaults to 'record'
  #           field      tag name for the XML element storing
  #                      each field - defaults to 'field'
  #           name       tag name for the XML element storing
  #                      the field name - defaults to 'name'
  #           value      tag name for the XML element storing
  #                      the value assigned to a field
  #                      - defaults to 'value'
  #           filter     reference to a subroutine that filters
  #                      hash keys and determines which get
  #                      included in the record
  #
  # $iDepth   number of tabs to indent the first record
  #           the "tab" is defined via $XML::HoA::TAB and
  #           is initially set to "   "; $iDepth defaults to 0

  my $sXML;
  $sXML = genRecord($aData);
  $sXML = genRecord($aData, $hOptions, $iDepth);

  # Example

  my $aData
     = [ { lname => 'Krynicky', fname => 'Jenda' }
       , { 'Site' => 'PerlMonks', 'Nick' => 'Jenda' }
       ];
  my $hOptions = { format => XML::HoA::NAME_TEXT_VAL_TEXT
                   , record => 'page'
                   , name => 'id' };

  print genRecord($aData, $hOptions);


=head1 DESCRIPTION

Module for converting arrays of hashes to XML.  This module
supports custom names for tags and six different ways of
representing the fields of a record.

You can choose which of the six you want by setting the C<format>
key of options hash (2nd parameter of C<genRecord(...)>. Other
keys of the options hash will be interpreted based on your selected
format.

=head2 Configuring formats

The six format values are:

=head3 XML::HoA::NAME_EQ_VAL

This format converts field values to C<fieldName="value">, like
this:

  <record
     lname="Krynicky"
     fname="Jenda"
  />

You can change the name of the record element using the option
hash key C<record>.  For example, if your option hash was

 { format => XML::HoA::NAME_EQ_VAL, record => 'page' }

Then your output would be:

  <page
     lname="Krynicky"
     fname="Jenda"
  />


=head3 XML::HoA::NAME_TAG_VAL_ATTR

This format converts each field to an element whose tag is the
the same as the field name. The value is an attribute:

  <record>
     <lname value="Krynicky"/>
     <fname value="Jenda"/>
  </record>

As with C<NAME_EQ_VAL>, you can set the C<record> element of the
option hash to change the record tag.  You can also use the
C<value> element to change the attribute name.  For example if
your option hash was:


 { format => XML::HoA::NAME_EQ_VAL
   , record => 'page'
   , value=>'input'
 }

Then your output would be:

  <page>
     <lname input="Krynicky"/>
     <fname input="Jenda"/>
  </page>

=head3 XML::HoA::NAME_TAG_VAL_TEXT

This format converts each field to an element whose tag is the
the same as the field name. The value is element text:

  <record>
     <lname>Krynicky</lname>
     <fname>Jenda</fname>
  </record>

As with C<XML::HoA::NAME_EQ_VAL>, only the C<format> and C<record>
keys of the option hash will be used.


=head3 XML::HoA::NAME_ATTR_VAL_ATTR

This format has one element for each field. The value is
stored as an attribute of each of these elements:

  <record>
     <field name="lname" value="Krynicky"/>
     <field name="fname" value="Jenda"/>
  </record>

All of the hash keys may be used to change the name of tags.
The following value of the options hash:

  { format => XML::HoA::NAME_TEXT_VAL_TEXT
    , record => 'person'
    , field  => 'property'
    , name   => 'ID'       #attribute name
    , value  => 'VALUE'    #attribute name
  }

would produce the following XML:

  <person>
     <property ID="lname" VALUE="Krynicky"/>
     <property ID="fname" VALUE="Jenda"/>
  </persn>

=head3 XML::HoA::NAME_ATTR_VAL_TEXT

This format has one element for each field. The value is
stored as the text of each of these elements:

  <record>
     <field name="lname">Krynicky</field>
     <field name="fname">Jenda</field>
  </record>

This format uses the same option hash keys as
C<XML::HoA::NAME_ATTR_VAL_ATTR>. TheC<value> key is ignored
since there is no value attribute.

=head3 XML::HoA::NAME_TEXT_VAL_TEXT

This format assigns the field name and field value to separate
elements:

  <record>
     <field>
       <name>lname</name>
       <value>Krynicky</value>
     </field>
     <field>
       <name>fname</name>
       <value>Jenda</value>
     </field>
  </record>

All of the hash keys may be used to change the name of tags.
The following value of the options hash:

  { format => XML::HoA::NAME_TEXT_VAL_TEXT
    , record => 'person'
    , field  => 'property'
    , name   => 'ID'
    , value  => 'VALUE'
  }

Would result in the following XML:

  <person>
     <property>
       <ID>lname</ID>
       <VALUE>Krynicky</VALUE>
     </property>
     <property>
       <ID>fname</ID>
       <VALUE>Jenda</VALUE>
     </property>
  </person>


=head2 Filtering record data

Sometimes you don't want to print all of the keys in a hash. If
you need to filter out some of the keys you can write a
routine. This routine takes three arguments, in order:

* the current hash key

* the value assigned to the current hash key

* a reference to the hash storing the record data

It returns 1 if the key should be included and 0 otherwise.
The default filter is a no-op: it always returns 1 so that
all keys are selected.

The following example only includes fields whose names are all
lowercase in the XML record:

   my $crFilter = sub {
      my ($k,$v,$hRecord) = @_;
      return $k =~ /^[a-z]+$/;
   }

   print genRecord($aData, { filter => $crFilter });


=head1 CAVEATS

Tested only via inspection.

=head1 AUTHOR

Elizabeth Grace Frank-Backman

=head1 COPYRIGHT

Copyright (c) 2008- Elizabeth Grace Frank-Backman. All rights
reserved.

This program is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.

=cut

#==================================================================
# SHARED DATA AND CONSTANTS
#==================================================================

our $TAB="  ";
use constant {
  NAME_EQ_VAL => 1
  , NAME_TAG_VAL_ATTR => 2
  , NAME_TAG_VAL_TEXT => 3
  , NAME_ATTR_VAL_ATTR => 4
  , NAME_ATTR_VAL_TEXT => 5
  , NAME_TEXT_VAL_TEXT => 6
};
my $DEFAULT_FORMAT=NAME_EQ_VAL;

#==================================================================
# FUNCTIONS
#==================================================================

sub genEndTag {
  my ($sIndent, $sTag, $bAttributes) = @_;
  return $bAttributes ? "/>\n": "$sIndent</$sTag>\n";
}

#-----------------------------------------------------------

sub genField {
  my ($k, $v, $hOptions, $iDepth) = @_;
  $hOptions = {} unless defined($hOptions);
  $iDepth=0 unless defined($iDepth);

  #set up options
  my $iFmt = $hOptions->{format};
  $iFmt=$DEFAULT_FORMAT unless defined($iFmt);

  my $sFieldTag = $hOptions->{field};
  $sFieldTag='field' unless defined($sFieldTag);

  my $sNameTag = $hOptions->{name};
  $sNameTag='name' unless defined($sNameTag);

  my $sValueTag = $hOptions->{value};
  $sValueTag='value' unless defined($sValueTag);

  #generate field XML
  my $sXML='';
  my $sIndent=$TAB x $iDepth;
  if ($iFmt eq NAME_EQ_VAL) {
    $sXML .= "$sIndent$k=\"$v\"\n";
  } elsif ($iFmt eq NAME_TAG_VAL_ATTR) {
    $sXML .= "$sIndent<$k $sValueTag=\"$v\"/>\n";
  } elsif ($iFmt eq NAME_TAG_VAL_TEXT) {
    $sXML .= "$sIndent<$k>$v</$k>\n";
  } elsif ($iFmt eq NAME_ATTR_VAL_ATTR) {
    $sXML .= "$sIndent<$sFieldTag $sNameTag=\"$k\" "
      ."$sValueTag=\"$v\"/>\n";
  } elsif ($iFmt eq NAME_ATTR_VAL_TEXT) {
    $sXML .= "$sIndent<$sFieldTag $sNameTag=\"$k\">"
      ."$v</$sFieldTag>\n";
  } elsif ($iFmt eq NAME_TEXT_VAL_TEXT) {
    my $sIndentTag= $TAB x ($iDepth + 1);
    $sXML .= "$sIndent<$sFieldTag>\n";
    $sXML .= "$sIndentTag<$sNameTag>$k</$sNameTag>\n";
    $sXML .= "$sIndentTag<$sValueTag>$v</$sValueTag>\n";
    $sXML .= "$sIndent</$sFieldTag>\n";
  }
  return $sXML;
}

#------------------------------------------------------------

sub genRecord {
  my ($aData, $hOptions, $iDepth) = @_;
  $iDepth=0 unless defined($iDepth);
  $hOptions = {} unless defined($hOptions);

  #set up options
  my $iFmt = $hOptions->{format};
  $iFmt=$DEFAULT_FORMAT unless defined($iFmt);

  my $sRecordTag = $hOptions->{record};
  $sRecordTag='record' unless defined($sRecordTag);

  my $crFilter = $hOptions->{filter};

  my $sIndent=$TAB x $iDepth;
  my $sXML = '';
  my $bAttributes = ($iFmt eq NAME_EQ_VAL);
  foreach my $hRecord (@$aData) {
    $sXML .= genStartTag($sIndent, $sRecordTag, $bAttributes);
    while (my ($k, $v) = each(%$hRecord)) {
      if (defined($crFilter) && ! &$crFilter($k,$v,$hRecord)) {
        next;
      }
      $sXML .= genField($k, $v, $hOptions, $iDepth+1);
    }
    $sXML .= genEndTag($sIndent, $sRecordTag, $bAttributes);
  }
  return $sXML;
}

#------------------------------------------------------------
sub genStartTag {
  my ($sIndent, $sTag, $bAttributes) = @_;
  return "$sIndent<$sTag" . ($bAttributes ? "\n" : ">\n");
}

#==================================================================
# MODULE INITIALIZATION
#==================================================================

1;
[download]

The other advantage of a functional approach like the one above is that it is quite easy to enhance to handle things like complex objects and records nested within records. One would only need to add an entry in the option hash that stored a hash that itself contained option hashes keyed by field name. When a field name was found in that hash, the value would be printed out by calling genRecord(...) recursively.

I've also attached a small demo script outputting the data for the specific example that you give:

use strict;
use warnings;
use XML::HoA qw(genRecord);

my $aData = [ { 'lname' => 'Krynicky'
                , 'fname' => 'Jenda'
                , PageId => 1, Name => 'Civil name'
              }
              , { 'Site' => 'PerlMonks'
                  , 'Nick' => 'Jenda'
                  , PageId => 2, Name => 'Online identity'
                }
            ];

my $crFilter = sub { my $k = shift;
                     return $k !~ /^(?:PageId|Name)$/; };
my $hOptions={format=>XML::HoA::NAME_TEXT_VAL_TEXT
              , record=>'page'
              , name=>'ID'
              , filter => $crFilter };
print genRecord($aData, $hOptions);
[download]

Best, beth

Comment on Re: Datastructures to XML Select or Download Code

Replies are listed 'Best First'.

Re^2: Datastructures to XML
by Jenda (Abbot) on Mar 17, 2009 at 23:47 UTC

Who said anything about records? Who said the datastructure is gonna be AoH? I would like to support any kind of datastructure. And be able to produce even more complex (read crazy) XML. I think you'd find this style getting quickly out of hand as soon as you attempted to support AoA,HoA,HoH,HoHoH,HoAoH, ... or as soon as you needed to produce things like

<record id="1">
  <foo>Hello</foo>
  <bar>World</bar>
</record>
[download]

For the simpler task of converting AoH to (more or less) record based XML your solution is probably simpler. Whether easier to use I'm not so sure. The templates as I see them, let the user specify how does he/she want the result to look like and then mark what is to be repeated for the A and what for the H, where to put the key and where the value from the hash, specify the static tags or data, etc.

Thanks for the comment anyway of course, I actually think your module might be a nice little addition to CPAN. Or maybe it could be added to XML::Records or XML::RAX. As a means to go the other direction than what the modules were originally made for.

Jenda
Support Denmark!
Defend the free world!

[reply]
[d/l]

Re^3: Datastructures to XML

by ELISHEVA (Prior) on Mar 18, 2009 at 02:22 UTC

Who said anything about records? Who said the datastructure is gonna be AoH?

Well,your question began: how would you convert this data, and this data was AoH. And you also stated that your inspiration for the meditation was SOPW that was also about an AoH. Conceptually, many people think of an AoH as an array of records, hence the terminology "record".

I would like to support any kind of datastructure.

That is an excellent goal, but it sits in tension with the goal of "making it easy" - as you observe about writing documentation when you responded to zentara. Much of the appeal of templating solutions comes from the fact that it is sort of WYSIWYG - there is less guess work in what the output will be. However, this benefit usually lessens when templating languages try to add more and more features to handle various edge cases. As extra syntax accumulates the template begins to look less and less like the actual output.

I think you'd find this style getting quickly out of hand as soon as you attempted to support AoA,HoA,HoH,HoHoH,HoAoH, ... or as soon as you needed to produce things like...

If you want to handle more complex data structures, it can be done without option keys multiplying like rabbits or creating hundreds of format constants. The key is to understand the meta structure of the problem. As with the templating approach, you need to do two things:

Provide a way to handle H, A, AoA, and AoH
Provide a way to handle array elements and hash values that are non-scalar: references to H, A, AoA, AoH, and blessed objects

With those two ingredients you can handle data structures of any complexity - whether you are using a functional approach or a templating approach. Handling fields that have non-scalar values was briefly discussed (though perhaps not very clearly) in my note above in the paragraph about adding support for recursion and a option hash key that stored a value representation rule: a hash reference storing option hashes keyed by field name or regex. This is really little more than a change in representation from the templating approach: the regex that appears in your template becomes the hash key; the written out example XML becomes the option hash assigned to the regex.

As for H,A,AoH,AoA. AoH is already handled. H is equivalent to AoH with one element, so modifying genRecord(...) to work with H rather than AoH is trivial. A is also equivalent to AoA with one element. So that leaves AoA and objects. To support AoA, you would need to deal with two scenarios: (a) indexes get mapped to names (b) each array element gets mapped to a nested element that differs only in the value assigned to it. (a) can be handled by modifying the filter function to return the field name rather than just a boolean value. Alternatively one could add a option hash key "indexToName" that has an array reference listing the field names in order. (b) can be handled by adding support for two additional format constants: NO_NAME_ATTR_VAL, NO_NAME_TEXT_VAL

Blessed objects raise other issues: (a) is the object opaque or can you just extract the data associated with the underlying blessed reference? (b) if the object is opaque, one needs to identify which methods should be used as getter methods. Whether one adopts a functional or templating approach one will still need to find a way to provide the same information: (opaque or not/which methods are getters).

OK, so you add a way to specify which "field" is gonna become an attribute of the record tag. Then you find out you sometimes need more. That sometimes the name of the field doesn't match the name of the attribute. ...

Some people would handle things that way, but I wouldn't. The functional approach I described actually does allow one to put "fields" in as attributes already as well as a number of other XML syntax variations. Check the format choices for details. What it didn't allow you to do is rename "fields" or pick and choose which fields are record attributes and which are nested elements.

However, providing support would require (for the user) no more than a minor modification to the filter function. Instead of returning a simple boolean, the filter function would return:

undef if the field should be skipped
the name of the field if the field should be included in the record. If only a name is returned the field will be a record attribute or nested element according to the option hash passed into genRecord(...)
a reference to an array or hash containing two bits of data: the field name and an option hash that overrides the one passed into genRecord(...) but only for that particular field.

I would consider a souped up filter function a better choice than the regex being used in the template spec because the logic involved in the choosing field names and field placement (attribute/nested element) may not be reducible to a regex.

For the simpler task of converting AoH to (more or less) record based XML your solution is probably simpler. Whether easier to use I'm not so sure

Each person has their own style and you (and probably many others) may simply prefer templating. I find templating approaches more limiting and "less easy", mainly because (a) even when there are obvious defaults, one still needs to spell out everything in a template - one could say it lacks Huffman encoding. (b) when I really do need to do fancy things like treat some fields as record attributes and some as nested elements (or rename fields) my logic may not be reducible to a regex. A filter function gives me the full power of Perl, including closures. (c) the implementation is more re-usuable. I can always layer a template language and parser over the functional approach. But I can also experiment with other ease of use interfaces.

Best, beth

Update: moved discussion of WYSIWYG and templating to start of post.

[reply]
[d/l]
[select]