Re^3: IPC::Open, Parallel::ForkManager, or Sockets::IO for parallelizing?

wonder if there is any way to parallelize the processing of the results in the same move

Have you tried just putting your processing in the handle_answer callback? That may be all you need. But I would still profile it first because it would be a big surprise if the feed processing step weren't dwarfed by the fetch times.

🦛

Comment on Re^3: IPC::Open, Parallel::ForkManager, or Sockets::IO for parallelizing? Download Code

Replies are listed 'Best First'.
Re^4: IPC::Open, Parallel::ForkManager, or Sockets::IO for parallelizing? by mldvx4 (Friar) on Sep 05, 2023 at 16:48 UTC
Have you tried just putting your processing in the `handle_answer` callback? Yes, I had started to look at that. As far as I can tell `handle_answer` keeps getting called only until the HTTP response is complete. I suppose there is a way to identify when the response is finally complete? For now, I will try building out the script as-is, with LWP in parallel and the rest serial. Some of the preparatory parts seem much faster than expected, based on trials in Perl compared to another scripting language. So it may very well not be an issue. Although I have lots of RAM on this computer, I'd like to find a way to process the responses as they come in so that all 700 to 800 of them don't sit around whole at the same time. Resuming more reading...	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^4: IPC::Open, Parallel::ForkManager, or Sockets::IO for parallelizing?
by mldvx4 (Friar) on Sep 05, 2023 at 16:48 UTC

Have you tried just putting your processing in the handle_answer callback?

Yes, I had started to look at that. As far as I can tell handle_answer keeps getting called only until the HTTP response is complete. I suppose there is a way to identify when the response is finally complete? For now, I will try building out the script as-is, with LWP in parallel and the rest serial. Some of the preparatory parts seem much faster than expected, based on trials in Perl compared to another scripting language. So it may very well not be an issue. Although I have lots of RAM on this computer, I'd like to find a way to process the responses as they come in so that all 700 to 800 of them don't sit around whole at the same time.

Resuming more reading...

[reply]
[d/l]
[select]