As far as XML parsers go, XML::Bare is one of the fastest. Since it was released there are some other faster parsers, but they exchange faster parsing for slower access to the data once parsed.
Based on what you are saying, you are outputting all, or most of your data each time you parse. Because of this it is unlikely to could gain higher speed via another parser.
There are speed losses because an in between structure is created ( a perl hash ). That is bloated and inefficient when you have large XML files ( over 3 MB ). If you have XML files that large you should look at using a SAX parser. XML::Bare has a SAX parser as well, but it is pre-beta and very unstable. ( there are a bunch of other SAX parsers on CPAN )
Note that I wrote XML::Bare, and have been using it in a way similar to what you are describing for the last several years, so I am sort of biased.
Note also that there are various glitches and problems with the current version of XML::Bare. The biggest one is that you cannot have nodes named value ex: '<value>'. Using such a node -will- crash the parser. | [reply] |
I basically want to now how much better performance I can get by using for exampel XML::XSLT
Who says it will be faster;) Although I'm biased towards doing things the XML way, i.e. use XSLT to transform a XML document, I sometimes encounter performance issues with XSLT. How complex is the resultset and what is the kind transformation your aiming for? The 78kb doesn't sound to scary to me. I suppose you can also limit the size of the resultset, much like you can limit the returned number of rows for a mySQL query? I wonder why mySQL was the bottleneck and what product you use now. Please provide some more info and maybe an example of a (representative) resultset and the kind of transformation you want to do.
Cheers
Harry
| [reply] |
Ok. Let's see...
The xml result consists of 'articles', where each row starts with 'article', containing 36 attributes (There is some header data also)
<response>
<serv>
<host>xxx</host>
<app>yyy</app>
<port>12345</port>
</serv>
<function>
<query>
<func>aaa</func>
<parameters>
<parameter>
<name>bbb</name>
<value>1</value>
</parameter>
<parameter>
<name>ccc</name>
<value>10</value>
</parameter>
<parameter>
<name>filter</name>
<value>ggg</value>
</parameter>
</parameters>
</query>
<result>
<articles>
<!- article starts here ->
<article>
<attribute1>..</attribute1>
...
<attribute36>..</attribute36>
</article>
</articles>
</result>
<time>13</time>
</function>
</response>
Every article entry varies from c:a 1.4 - 2.6 kb in size, which gives us an average of 2 kb/article. The number of articles varies from 1-36 (it's hard to give an average here, since it's varies a lot), so let's say that it varies between 2-72 kb using the average value, so let's say 1-100 kb to be on the safe side.
So, it's not really that much data to process. As you said I can limit the size of the resultset, but I want to calculate using these values, also to be on the safe side
The MySQL database is a bottleneck because it holds so much more data than just article information. The new data source (which I can't reveal) is designed for just this kind of data, and will provide better performance.So, I'm basically tries to make the parse as good as possible compared to the "query". | [reply]
[d/l] |
I'm not convinced of the business case for using XML. It's hard to believe that a relational database is outperformed by the new datasource. But then again the datasource is a bit of a mystery:) I worked with different types of databases including native XML databases. In my experience it is hard to beat the RDBMS in terms of retrieval speed. Unless you have some exotic data structure and need many joins to grab the data together. You can often de-normalize data to fix that though. Size of the database is hardly ever a problem in my experience.
I agree that it's not much data to process but XML is (very) verbose, you probably have even less data to process when you use a RDBMS. From what I can derive from your example the data structure is quite simple, i.e. rows in a table with a variable number of columns (your attributes). So I guess the transformation is also not too complex. I would not call my att attributes attribute1 .. attribute36 though, but give them meaningful names.
It's not too difficult to write some test cases and Benchmark the stuff to gain confidence in the solution.
Cheers
Harry
| [reply]
[d/l] |
Either your mysql is set all wrong or you're running the new datasource on a much quicker machine. Unless you are looking for an excuse to use XML, you'd better fix your mysql.
Jenda
Enoch was right!
Enjoy the last years of Rome.
| [reply] |