in reply to Re^7: Multithreaded xml writing using XML::Writer
in thread Multithreaded xml writing using XML::Writer

Ok, thank you for a very detailed and good answer!
To answered my unanswered question first: No, the ordering of data is irrelevant, my thought is that whoever of the subroutines that has data ready should write it immediately, unless another one of them currently writes data. I.e., a queue.

I will benchmark the time it takes to execute the queries in parallell/serial "fashion" to determine if it buys me time.

But for now, I'd be glad to learn how to do this, in case it actually buys me time:-)

I can't go in too much on detail how the fetched data looks. What I can say is that the data contains
1. Product information, such as article number, article description etc, a mix of ID's and text fields, around 10 of each. The number of rows can vary from 10 to 15000, where the last one is more common.

2. A website database. his one holds websites, and can generate c:a 1000 rows.
3. Another website database, with approximately the same specs as number 2.

It feels like "Pass a cloned or (externally) shared XML::Writer object to your threads and (externally) serialise use of it through locking." would be an option. Can you describe how you would do it?
  • Comment on Re^8: Multithreaded xml writing using XML::Writer

Replies are listed 'Best First'.
Re^9: Multithreaded xml writing using XML::Writer
by BrowserUk (Patriarch) on May 04, 2010 at 14:41 UTC
    It feels like "Pass a cloned or (externally) shared XML::Writer object to your threads and (externally) serialise use of it through locking." would be an option. Can you describe how you would do it?

    Still not enough info. I don't need to see the details of your content,nor even the real field names. But some idea of the structure. For example, would something like this be acceptable as output: 

    <?xml version="1.0" encoding="UTF-8"?>

<operations>
 <sites-list2>website 1</sites-list1>
 <product>product 1</product>
 <sites-list1>website 1</sites-list1>
 <product>product 2</product>
 <sites-list1>website 2</sites-list1>
</operations>

    [download]

    Or this (or some other essentially random ordering of it?):

    <?xml version="1.0" encoding="UTF-8"?>

<operations>
    <website-1-1>url</website-1-1>
  <product-2>
 <sites-list1>
  </product-1>
    <detail-2>stuff</detail-2>
    <website-2-1>url</website-2-1>
    <detail-1>stuff</detail-1>
    <detail-1>stuff</detail-1>
 <sites-list2>

    <website-2-2>url</website-2-2>
</sites-list1>
 <products>
  <product-1>
  </product-2>
 </product>
    <website-1-2>url</website-1-2>
</sites-list1>
    <detail-2>stuff</detail-2>
</operations>

    [download]

    That's shuffled rather than interleaved, but it makes my point. Depending upon the structure of the data from different sources, the locking requirements vary. If each of the data sources only produced a single line--one tag (pair) and lots of attributes--and it didn't matter how the lines from the 3 sources where interleaved, then you would use a different solution than if the data sources produce deeply nested structures of tags for each row.

    If, as I suspect, you want all the rows from each data source grouped together (possibly under a top-level tag for that data sources), and your " No, the ordering of data is irrelevant" means that you don't care which order the 3 blocks are in the file, you'd be better creating the output from each data source separately (perhaps in ram files using three separate instances of XML::Writer), and then combining them into an output file as 3 large chunks.

    But if that is the case, I would seriously consider using the -xml flag on the mysql or mysqldump commands.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom