Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

HTTP Request and Threads

by banhmi (Initiate)
on Sep 12, 2018 at 17:45 UTC ( #1222242=perlquestion: print w/replies, xml ) Need Help??

banhmi has asked for the wisdom of the Perl Monks concerning the following question:

Hi all. I'm aware Perl and threading don't have the best relationship, but this crash has been bugging me for a while. I'm wondering if anyone has encountered similar issues, or know what is going on here.

#!/usr/bin/perl -l use strict; use warnings FATAL => 'all'; use Data::Dumper; use threads; use threads::shared; use Thread::Queue; use HTTP::Headers; use LWP::UserAgent; my $counter_thread : shared; my $a = threads->create(sub () { my $method = "GET"; my $uri = "https://www.google.com/"; my $header = HTTP::Headers->new(); $header->push_header("Content-Type" => "application/json"); my $ua = LWP::UserAgent->new; $ua->ssl_opts(verify_hostname => 0, SSL_verify_mode => 0x00); my $request = HTTP::Request->new(); $request->method($method); $request->uri($uri); $request->headers($header); my $response = $ua->request($request); print $counter_thread++, "th :", "Dumped Data Returned:\n\n ", Dum +per($response->decoded_content); print "Response code: ", $response->code, "\tHTTP Response Messag +e: ", $response->message, " "; print threads->tid(); }); $a-> join; my $b = threads->create(sub () { my $method = "GET"; my $uri = "https://www.google.com/"; my $header = HTTP::Headers->new(); $header->push_header("Content-Type" => "application/json"); my $ua = LWP::UserAgent->new; $ua->ssl_opts(verify_hostname => 0, SSL_verify_mode => 0x00); my $request = HTTP::Request->new(); $request->method($method); $request->uri($uri); $request->headers($header); my $response = $ua->request($request); print $counter_thread++, "th :", "Dumped Data Returned:\n\n ", Dum +per($response->decoded_content); print "Response code: ", $response->code, "\tHTTP Response Messag +e: ", $response->message, " "; print threads->tid(); }); $b-> join;

Basically, I want to create two threads that do a similar GET HTTP request and print the response out (please ignore the clumsy code, I tried to repro the crash in a rush). The first thread would run fine. But the second thread would always cause a crash with one of these errors:

Process finished with exit code -1073741816 (0xC0000008) Process finished with exit code -1073741819 (0xC0000005) Process finished with exit code -1073741790 (0xC0000022)

I'm pretty sure one time both of the threads work, so the issue might be intermittent. I haven't had much luck looking for any clues. If anyone could point me in the right direction, I'd appreciate it greatly!

Edit: I've been spending some time looking more at this. What I noticed is for my machine, specifically the fail conditions are, the first request sent by the user agent has to be in a thread that's joined right after its created. On the second request, thread or non-thread, it will mostly crash (around 98% for me, there were 3-4 runs that worked randomly). That said, a quick workaround I'm implementing right now is sending a simple non-thread GET request just to "plant the seed". Then the subsequent requests in threads or non-threads would run without problems. I'm using This is perl 5, version 24, subversion 3 (v5.24.3) built for MSWin32-x86-multi-thread-64int

Edit 2: Per @ikegami and @Corion's comments, I tried using an http uri instead of https and that seems to run fine. It might be the case that SSL libraries (which are not pure Perl) are only loaded if we use https.

Replies are listed 'Best First'.
Re: HTTP Request and Threads
by hippo (Bishop) on Sep 13, 2018 at 08:11 UTC
    The first thread would run fine. But the second thread would always cause a crash

    I've tried running your code as-is and cannot get it to crash (over the space of a few runs) on my perl which is This is perl 5, version 20, subversion 3 (v5.20.3) built for x86_64-linux-thread-multi. To remove some possible sources of error I then made a few changes and come up with this, which I can't get to crash either.

    #!/usr/bin/perl -l use strict; use warnings FATAL => 'all'; use threads; use threads::shared; use HTTP::Headers; use LWP::UserAgent; my $counter_thread : shared; sub getgoog { my $method = "GET"; my $uri = "https://www.google.com/"; my $header = HTTP::Headers->new(); $header->push_header("Content-Type" => "application/json"); my $ua = LWP::UserAgent->new; $ua->ssl_opts(verify_hostname => 0, SSL_verify_mode => 0x00); my $request = HTTP::Request->new(); $request->method($method); $request->uri($uri); $request->headers($header); my $response = $ua->request($request); print $counter_thread++, "th :", "Dumped Data Returned:\n\n ", sub +str ($response->decoded_content, 0, 256); print "Response code: ", $response->code, "\tHTTP Response Messag +e: ", $response->message, " "; print "Thread id: ", threads->tid(), "\n"; } my $thra = threads->create (\&getgoog); $thra-> join; my $thrb = threads->create (\&getgoog); $thrb-> join;

    For clarity, the changes are:

    • De-duplicated the sub as both appear to be completely identical.
    • Remove the unused Threads::Queue
    • Removed use of Dumper (because it's acting on a scalar string)
    • Avoided use of the specials $a and $b
    • Truncated the output for visualisation reasons

    What happens if you try running my code? You might also want to consider experimenting with not making warnings fatal.

      Thank you for your response, and thanks for taking the time to make the code look prettier! I ran your code and the same crash still happened, unfortunately. It's encouraging to know it's working for someone at least though. I made an edit to my original post about a possible workaround if you're interested.

Re: HTTP Request and Threads
by ikegami (Patriarch) on Sep 13, 2018 at 10:50 UTC

    I suspect the SSL library. Everything else should be Pure Perl.

    I think there are two classes that Net::HTTPS can used for SSL; using the other might help?

      That's a good idea!

      I think that LWP::UserAgent will only load the SSL libraries once it encounters an https URL. To be really sure, you can try to load LWP::UserAgent only from within each thread. This has other drawbacks, but at least for diagnosis, maybe it helps:

      #use LWP::UserAgent; # don't load this in the outside program my $a = threads->create(sub () { require LWP::UserAgent; my $ua = LWP::UserAgent->new; $ua->ssl_opts(verify_hostname => 0, SSL_verify_mode => 0x00); };

      Thinking more about this, I'm less confident that this approach changes much, at least if the C part of the ssl library still only gets initialized once, but as it's a small change, I think it's worth a try.

        Thanks for the response. Loading LWP::UserAgent from within the thread unfortunately doesn't help. But from your comment, I did try using an http URI and that seems to work just fine. So at least I think suspecting SSL library seems to be the right way to turn!

Re: HTTP Request and Threads
by beech (Parson) on Sep 13, 2018 at 01:02 UTC

    Hi

    MS-ERREF: NTSTATUS Values

    0xC0000008 STATUS_INVALID_HANDLE An invalid HANDLE was spec +ified. 0xC0000005 STATUS_ACCESS_VIOLATION The instruction at 0x%0 +8lx referenced memory at 0x%08lx. The memory could not be %s. +0xC0000006 0xC0000022 STATUS_ACCESS_DENIED {Access Denied} A process +has requested access to an object but has not been granted those acce +ss rights.

    So, based on that, I try something simple, move the first join next to the second, and the program runs to completion.

    Starting over, I add exit 0; after the last join , and it also lets the program complete without exceptions

    However, redirecting the output to a file, exit solution crashes the same way. Same if I move some use statements around.

    However, the original solution doesn't crash no matter what, create threads first, join them later .

    However I'm using perl 5.16.1 a long unsupported windows OS , so :)

      Thank you for the response! I also got success in joining the threads after creating them. Unfortunately for the way that this code is implemented in the bigger project that I'm working on, making that change would be difficult. This is a good workaround though, and helped narrow the root of error down.

      Starting over, without the first change, adding exit 0 at the end still gave me similar errors (though there was 1 success run out of 5).

      In case you're interested, what I noticed is for my machine, in order for the code to fail, the first request sent by the user agent has to be in a thread that's joined right after its created. On the second request, thread or non-thread, it will mostly crash (around 98% for me, there were 3-4 runs that worked randomly). That said, a quick workaround I'm implementing right now is sending a simple non-thread GET request just to "plant the seed". Then the subsequent requests in threads or non-threads would run without problems.

Re: HTTP Request and Threads
by noxxi (Pilgrim) on Sep 14, 2018 at 18:15 UTC
    While the SSL support is mainly thread safe the initialization of the SSL library is special. The library should not be initialized per thread but should be initialized per program - before any threads are created. I therefore recommend that you do a "use IO::Socket::SSL" in your main code which will take care of initializing the SSL library in the main thread so that it will not attempt initialization in the other threads too.

      This solved my problem. Thank you noxxi!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1222242]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (6)
As of 2022-01-20 17:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:












    Results (57 votes). Check out past polls.

    Notices?