Re^2: querying multiple databases in parallel -- attach problem #2

with forking (or threading) i'm trying to speed up the process, the databases are quit large and when i query it as a one db then my one core is on 100% and the rest of them are just sitting there. so to speed up the process even twice it could mean 4h saved, not to mention running it on 8 cores, which i'm planing to do.

so my question is focused on how to enable this.

i'm also open for any alternative suggestions that would speed up my process

thnx

Comment on Re^2: querying multiple databases in parallel -- attach problem #2

Replies are listed 'Best First'.
Re^3: querying multiple databases in parallel -- attach problem #2 by MidLifeXis (Monsignor) on Nov 14, 2008 at 15:14 UTC
By breaking each DB handle out to a different core, you could still have idle time (one DB handle consistently takes longer to complete the query, for example). Can your problem be broken down more at the higher level? In other words, instead of breaking each handle out to a different thread, break the top level data set into N smaller sets and distribute those smaller jobs out to the individual cores. Each core would still have M (4 in this case?) DB handles, but each core could stay busy (less the slight overhead for managing the data sets). Now, if all of the top-level data is interdependent, then never mind :-) --MidLifeXis	[reply]

Replies are listed 'Best First'.

Re^3: querying multiple databases in parallel -- attach problem #2
by MidLifeXis (Monsignor) on Nov 14, 2008 at 15:14 UTC

By breaking each DB handle out to a different core, you could still have idle time (one DB handle consistently takes longer to complete the query, for example).

Can your problem be broken down more at the higher level? In other words, instead of breaking each handle out to a different thread, break the top level data set into N smaller sets and distribute those smaller jobs out to the individual cores. Each core would still have M (4 in this case?) DB handles, but each core could stay busy (less the slight overhead for managing the data sets).

Now, if all of the top-level data is interdependent, then never mind :-)

--MidLifeXis

[reply]