in reply to Parsing a lot of data. Very slow. Need suggestions
I count 16 process invocations (17, including the script), and from the look of things, the script is contributing noise to the overall time. It should be trivial to add some timing code to measure how much time each subprocess is taking.
Then you can figure out how to optimize or combine your subprocess scripts. For example, do you really need to run parsecache twice to generate separate files? Or could that script be changed to spit out two files from a single invocation? Ditto for report. That would cut subprocess invocation in half, which would have to be a win.
|
|---|