Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks I am taking baby steps in creating a Boss/Worker threading I have liberally used the example at Re: Perl Threads Boss/Worker Example to create the following:

  my ($accessToken, $appId, $userId) = (shift, shift, shift);
  use threads;
  use Thread::Queue;
  use WWW::Curl::Easy;
  use WWW::Curl::Form;

  my $threadLimit = 1;
  my $testUsers=listTestUsersFull($accessToken, $appId);
  my $q = Thread::Queue->new();    # A new empty queue
  # Worker threads
  my @thr = map {
    threads->create(sub {
      while (my @item = $q->dequeue(2)) {
        # assuming undef isn't a valid value and so can be a marker.
        last unless defined $item[0];
        last unless defined $item[1];
        deleteUser($item[1], $appId, $item[0]);
      }
    } , $appId
   )->detach();
  } 1..$threadLimit;

# Send work to the threads
@testUsers = @testUsers[1..5];
  foreach (keys %{$testUsers}){
    $q->enqueue($userId, $$testUsers{$userId}->{access_token});
  }

# send markers.
  $q->enqueue(undef) for (1..$threadLimit);

# terminate.
  $_->join() for @thr;

[download]

the deleteUser subroutine calls another which contains

 use WWW::Curl::Easy;
 use WWW::Curl::Form;

[download]
, thus sending a simple form. My problem is the subroutine with those WWW::Curl:: calls returns at the line that posts the form:  my $response = $curl->perform; however when called via a non-threaded subroutine, everything just works. The documentation for WWW::Curl states threading is hopefully possible.

I am hoping to gain some wisdom on my problem, I thank you!

Replies are listed 'Best First'.
Re: Thread::Queue and WWW::Curl
by Corion (Patriarch) on Sep 24, 2012 at 14:33 UTC

    You don't show where you create instances of WWW::Curl(::Easy). I recommend that you do so separately and only within each thread, because most likely the WWW::Curl objects are backed by corresponding C objects, and those may or may not like dancing between different threads and maybe are not even threadsafe unless precautions are taken.

    Personally, I would look at ditching threads and looking at the Curl-native routines in WWW::Curl::Multi. These should be able to run within a single worker thread and at least for the fetching part, you are most likely limited by network bandwidth/latency anyway.

[reply]
Re: Thread::Queue and WWW::Curl
by BrowserUk (Patriarch) on Sep 24, 2012 at 14:42 UTC
    My problem is the subroutine with those WWW::Curl:: calls returns at the line that posts the form: my $response = $curl->perform;

    Is that a problem description? Because I cannot see where it says something fails, much less what; how; what error mesages/code you get.

    the deleteUser subroutine calls another which contains use WWW::Curl::Easy; use WWW::Curl::Form;

    That kind of implies that the code that is failing is in the subroutines that you haven't bothered to post. Is that the case?

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    RIP Neil Armstrong
    network sites:
[reply]

      Thanks for the speedy replies. The 2 missing subroutines are as follows:

      sub deleteUser {
# in: app access token, app id, FB userId to delete
# deletes user using app access token
# out: nothing
  my ($accessToken, $appId, $userId) = (shift, shift, shift);
  my %form = (
    method => 'delete',
    access_token => $accessToken
  );
  my $url = 'https://graph.facebook.com/' . $userId;
  my $post = hitFbNew2('post', $url , \%form);
  return ($post);
}

      [download]
      and
      sub hitFbNew {
  my ($method, $endpoint, $form) = (shift, shift, shift);
  my $curl = WWW::Curl::Easy->new;
  my $curlf = WWW::Curl::Form->new;
  my $response_body;
  for my $key (keys %{$form}){
    $curlf->formadd($key, $form->{$key});
  }
  $curl->setopt(CURLOPT_WRITEDATA, \$response_body);
  $curl->setopt(CURLOPT_URL, $endpoint);
  $curl->setopt(CURLOPT_HTTPPOST, $curlf);
  print "\nHitting FB using the curl libraries...";
  my $response = $curl->perform;
  print "\nresponse was $response";
  my %reply  = (
    content => $response_body,
    responseCode => $curl->getinfo(CURLINFO_HTTP_CODE)
  );
  print "\nreply code is $reply{responseCode} and content is $reply{co
+ntent}";
  return (\%reply);
}

      [download]
      I will look at WWW::Curl::Multi in the meantime.
[reply]
[d/l]
[select]

        You still haven't said what fails? Or how you know something is failing?

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        RIP Neil Armstrong
[reply]
Re: Thread::Queue and WWW::Curl
by Anonymous Monk on Sep 24, 2012 at 14:25 UTC
    Amending title.
[reply]

Back to Seekers of Perl Wisdom