in reply to Threads and HTTPS Segfault

Instead of using threads, use coros. LWP::Protocol::AnyEvent::http makes it easy. Just start off like: 
#!/usr/bin/perl

use strict;
use warnings;
use LWP::Protocol::AnyEvent::http;
use Coro qw( async );

my $ua = LWP::UserAgent->new();
$ua->protocols_allowed([qw( http https )]);

[download]

Replies are listed 'Best First'.
Re^2: Threads and HTTPS Segfault
by onelesd (Pilgrim) on Aug 21, 2011 at 19:02 UTC
    Coro doesn't scale across multiple processors and that would hurt performance, but I suppose I could move the HTTPS request from the child threads into a Coro thread in the parent, but I'd need some way to send data from the children to the Coro thread.
[reply]

      If you cannot fix the problem and need bi-di communications with the kids, then you could do worse than to use a queue talking to threads that run lwp-requests via piped-opens. The https sessions run in separate processes, but you fetch the data back into the parent process where you can coordinate between them.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

        I tried this (using open and curl) and performance took too big a hit. The cpu was doing about 15% user and 85% system and I could only get about 40 req/s - I need to be around 70 to 100, which I am able to do when running many disconnected non-threaded scripts.

        Starting up a process for each request unfortunately is too expensive, thank you for the suggestion, though.

[reply]

      You say in your original post, that...

      Most of the time in each thread is spent making the HTTPS request ...

      If this is true you shouldn't lose much performance by using an event framework and/or Coro... depending on what you mean by "most" I guess...

[reply]

        Is that true for https connections?

        I thought SSL connections imposed a fairly high cpu loaded because of the decryption requirements. Even on a pretty low bandwidth connection, cpu usage can quickly become the limiting factor for throughput. Start trying to decode multiple concurrent streams on the same processor (as with Coro, POE and other event driven architectures), and cpu will definitely become the limit factor to throughput.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom