I think the idea seems sound enough just as long as you allow some sort of way to map XML elements to objects based on some other way than string. In other words storing a user defined (possibly through a higher level meta-language) hash entry such as <chapter> --> Publish::Chapter seems like a good idea, but parsing the element name string and automatically mapping that to an element seems like an inherently bad idea.
Another thing to keep in mind is that XML by its nature is inherently structured and Perl by its nature is inherently unstructured so expect monstrous amounts of wrapping.
One thing I would like would be the ability to support persistence using XML...ie mapping from XML to object and back to XML.
Anyway, my $.02