|
in reply to Re^3: Module uses loads of CPU.. or is it me
in thread Module uses loads of CPU.. or is it me
Wow thanks for digging this deep to find this problem! That is awesome.
And thanks also for the suggestions, though writing a new XML parsing tool seems a little much (if not a little bit daunting too:), and not knowing what the actual return will be makes me wonder if it's worth it in this case.
Now I wonder if this exact issue has not been encountered in other XML applications, and if so, how it was improved.
Thanks again,
-H
|
Re^5: Module uses loads of CPU.. or is it me
by redhotpenguin (Deacon) on Dec 11, 2007 at 17:22 UTC
|
Well I think you should take a serious look at some of the lower level methods in Net::S3::Amazon and try to develop a parallelized application. It isn't likely that you will be able to increase the efficiency of the parser by double, but I think with a few hours hacking you could get a parallelized version of your program that you can make Nx speedups with.
| [reply] |
|
|
Without hacking too deeply, some parallelization might be acheived by using the "marker" method already in the module itself, and having parallel fetches in batches. Looking further into the module I think I found an error or bug (maybe it's just a documentation bug) http://rt.cpan.org/Public/Bug/Display.html?id=31381.
Going much further is going to be a question of (my) available time right now -- but I really do appreciate the push!
-Harold
| [reply] |
|
