in reply to Re^8: Threads calling LWP causes exception
in thread Threads calling LWP causes exception

Would you mind running this code with a command line of "https://www.wellsfargo.com" ? I have a feeling that Crypt-SSLeay is not thread safe. 

package perl_module1;
use strict;
use v5.8;
use HTTP::Cookies;
use LWP::UserAgent;
use HTTP::Request::Common;
use URI::Escape;

use vars qw(@ISA @EXPORT);
use Exporter ();
our @ISA = qw(Exporter);
our @EXPORT = qw( GetIt );
our $DEBUG = 0;

sub GetIt {
    my ($url) = @_;

    my ($ua) = LWP::UserAgent->new(keep_alive => 1);
    $ua->timeout(60);

    # allow redirects for POST
    push @{ $ua->requests_redirectable }, 'POST';

    # set the cookie
    my ($cookie) = './ac_cookie.txt' . $$;
    $ua->cookie_jar(HTTP::Cookies->new('file' => $cookie));

    my ($response) = $ua->get($url);
    if ($response->is_success) {
        my ($page) = $response->content;
        print( $page );
    }
    else {
        printf "Failed with %s\n", $response->message;
    }
}

return 1 if caller;

package main;
use strict;
use warnings;
use v5.8;
use threads;
use threads::shared;

print("Starting LetsRock() \n");

ThreadFunc( $ARGV[ 0 ] );
LetsRock( $ARGV[ 0 ] );

print("LetsRock() ended \n");
exit(0);

sub LetsRock {
    my $url = shift;
    my (@kids);

    for (my $x=0; $x < 5; $x++) {
        my ($kid) = threads->create( \&ThreadFunc, $url );

        if (defined($kid)) {
            push(@kids, $kid);
        }

        sleep(1);
    }
    printf "Waiting for %s\n", $_->tid, $_->join for @kids;
}

sub ThreadFunc { return perl_module1::GetIt( $_[0] ); }

__END__

[download]

Replies are listed 'Best First'.
Re^10: Threads calling LWP causes exception
by BrowserUk (Patriarch) on Mar 11, 2006 at 05:55 UTC

    You may be happy to know that I finally reproduced your problem here using the above code, and I've reduce the failure case to this: 

    package main;
use strict;
use warnings;
use v5.8;
use POSIX qw[ _exit ];
use threads;

sub GetIt2 {
    require LWP::Simple;
    return LWP::Simple::get( $_[0] );
}

#print
GetIt2( $ARGV[ 0 ] );

#print
threads->create( \&GetIt2, $ARGV[ 0 ] )->join;

#_exit(0);

    [download]

    This seems to be a bad interaction during cleanup between libeay32.dll and thread.pm.

    The keys to the failure seem to be:

    1. Using LWP for fetch a https url.
    2. Calling the thread func directly from the main thread before using as a thread procedure.
    3. Retrieving the return values (joining) the thread.
    4. Allowing the perl cleanup code to run.

    It will require somebody with greater internals skills to resolve that further, so you should raise a perlbug. The code above should suffice as a testcase.

    I've two workarounds for you.

    1. Don't both call your ThreadProc() as a synchronous function and use it as a thread procedure.

      The failure does not seem to occur unless you do that.

    2. If you really cannot avoid that situation in your code, the only call it from the main thread after you've started the threads.

      The error does not seem to occur then either.

    3. And if you really, really cannot not avoid the problem in either of these two ways, then you might consider using POSIX::_exit() to terminate your code.

      This is not a good solution as it effectively bypasses Perl's cleanup, which means open files don't get closed properly, destructors don't get called etc. If your code relies on any of the cleanup events for proper function, it could give you problems.

      That said, if you ensure that all your files are closed and data is saved before you call POSIX::_exit(), it will avoid the exception, so it worth considering as a last resort until a proper solution is found and implemented.

      Disclaimer: It's at your own risk and I didn't tell you about it :)

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
      Thanks again.
[reply]

In Section Seekers of Perl Wisdom