Re: Data. Lots of Data.

Could this be a job for XQL?

If you convert your data to XML, then you could use one of the XQL modules at CPAN.

XQL allows you to use SQL-like syntax to access an XML tree of data, and it could be what you want.

A potential problem is that some implementations of XQL hold the entire XML tree in memory, others do not. An implementation which did the former might use up all of your memory.

The documentation for Enno Dirksen's XML-XQL at CPAN says that it uses the module XML-DOM, which means that it holds the entire XML tree in memory. The documentation says that future releases might use the module XML-Grove. Using XML-Grove would not hold the entire tree in memory, since avoiding memory-hogging XML apps is the purpose of XML-Grove.

There is also an XML-miniXQL by Matt Sergeant at CPAN. It is not clear from reading its documentation what its memory requirements are.

If I were going to do this project I would look into XML/XQL. The modules install easily and one can do simple applications in an afternoon.

Comment on Re: Data. Lots of Data.

Replies are listed 'Best First'.
Re (tilly) 2: Data. Lots of Data. by tilly (Archbishop) on Feb 07, 2001 at 17:55 UTC
XML may be many things, but high-performance and memory efficiency are not two of its strengths. Quite the opposite in fact. The format is portable, flexible, extensible...but this comes with considerable overhead.	[reply]
Re: Re: Data. Lots of Data. by mirod (Canon) on Feb 07, 2001 at 18:13 UTC
XML::XQL might be your hammer but we are not dealing with nails here! XML::XQL is based on XML::DOM as you said, which means that it eats up RAM like there's no tomorrow, about 10 times the original size of the XML data (which is already much more than CSV for example). XML::Grove does not seem to be very well supported from comments to the review and from the fact that the last release is dated September 9th 1999, and is slow as a dog in any case. XML::miniXQL is not exactly supported either, the last version was released June 16th 1999. While you might like XML::XQL it is certainly not the ultimate in terms of data munging. Sorry for the plug but you might want to have a look at Processing XML with Perl or Ways to Rome to see a couple of comparisons of various XML modules along with benchmarks. In this case a data base solution seems like the only way to go (and I am as much of an XML evangelist as you might find around here ;--)	[reply]