Re: How to speed up my Html parsing program? (Concurrently run Subroutines?)

You could try Parallel::Simple and run multiple queries at once, but there's no real speed-up unless you're running on multiple cores. It's probably more worthwhile to take a look at your code and see how to improve it. If you're doing a lot of regular expression matching you might want to call study on your string as that does some groundwork which might speed things up.
HTH, SSF

Comment on Re: How to speed up my Html parsing program? (Concurrently run Subroutines?) Download Code

Replies are listed 'Best First'.
Re^2: How to speed up my Html parsing program? (Concurrently run Subroutines?) by eye (Chaplain) on Jan 06, 2009 at 07:37 UTC
This page contains a list of links which then need to be retrieved by the get_page_html() method and there content passed to retrieve_info(); If the get_page_html() method retrieves pages over a network (rather than from disk), there is great potential for improving performance with forking or threads. In a single process/thread, network latency is additive. With multiple processes/threads, latency costs can run concurrently.	[reply]
Re^3: How to speed up my Html parsing program? (Concurrently run Subroutines?) by BobFishel (Acolyte) on Jan 06, 2009 at 16:34 UTC
Yes, it retrieves it over a network. I'm definitly going to look into forking, after doing some reading last night it seems this is my best bet at this point. Now I just need to figure out how to keep my variables indedpendant. I haven't dug into the code in the responses below yet but from the looks of it they seem like a great starting point!	[reply]
Re^2: How to speed up my Html parsing program? (Concurrently run Subroutines?) by BobFishel (Acolyte) on Jan 06, 2009 at 18:39 UTC
Wow I just got some time to look into study and I feel I will definitely make some gains incorporating this into my program. Thanks!	[reply]