Hi monks i am here again to seek your wisdom..

I have a cgi script (which i am trying to turn into psgi app Change cgi script to PSGI app) which has to connect to different vendor api's and collect data and return in as a json/xml output

I am trying to fetch this data simultaneously from each vendor by creating threads for each vendor.
I need to create max 10 threads.
But now vendor api has changed and i will have to make additional api calls in each thread.
Example Call a vendor api get list of products then fetch details of each over a api call with product id.
If i start threads again in a thread already it crashes and memory increases too, plus i need to load too much modules (may be i don't know how to do it correctly)

I have been searching perlmomks and net for probable alternatives and found that as my script usually wastes lot of time in waiting for SOAP::Lites calls i may be better off with using Coro instead of threads. (one of the reference Why Coro?)
Again by reading on perlmonks i changed to using forks/forks::BerkeleyDB (forgive me i don't have reference to this now).
It worked fine till it was not needed to use threads in already running threads. If i fork again foreach product apache has too many children.
I actually tried it on a test system and my admin is not nice to me since :(

Can i safely use Coro to do this?
I am a bit confused about how to explicitly use Coro's cede function.(where to put it exactly inside threads) If i cede it inside a thread will that thread run again?
Will all my vendor functions that i start in main thread run in parallel?
I have read http://http://search.cpan.org/~mlehmann/Coro-6.41/Coro.pm this in details but i am confused.

Please enlighten me..

#!/usr/bin/perl
#use forks::BerkeleyDB;
#use threads;
use strict;
use Coro;
use CGI qw(:standard);
use XML::Simple;
use JSON qw(encode_json);
use MIME::Base64 qw(encode_base64);
use DBI;
use SOAP::Lite; 
use Data::Dumper;
use CHI;
my (@req,@odata,@errdet)=();
    my %threads=();
    foreach my $vnd (@req){
        #$threads{$vnd}=threads->create({ 'context' => 'list', 'exit' 
+=> 'thread_only' },\&startthread,$vnd,$SCH);
        $threads{$vnd}=async{&startthread($vnd,$SCH) };
    }
    cede;
    foreach my $vnd (keys %threads) 
    {
        my($err,$data) =$threads{$vnd}->join();
        if ($err eq 'N') {
            push @odata,@{$data};
        }
        }elseif($err eq 'Y'){
            push @errdet,$data;
        }elsif($threads{$vnd}->error) # with threads it was useful,
        {
            push @errdet,{ Vnd=>$vnd, ErrorCode=>26,ErrorMsg=>$threads
+{$vnd}->error};
        }
     }    


sub startthread
{
    my ($vendor,$SCH)=@_;
    my($err,$data) =();
    require './'.$vendor.'functions.pl"; #load vendor specific code
    ($err,$data)=&getvendordata($SCH);
    return($err,$data);
}

#vendor functions has somewhat code like this besides functions specif
+ic to each vendor api

sub getvendordata
{
    my $SCH=shift;
    #perform vendor specific tasks mostly SOAP::Lite calls to api
    #Has to perform multiple calls some of them depend on output of pr
+evious calls
    #Also have to make simultaneous calls which may not be related to 
+each other can be done parallelly.
    #e.g. first call to api returns a list of products subsequent call
+s to fetch details.
    #If i start threads/forks again here most of the time program cras
+hes :(
    my ($err,@data)=();
    my @products=getproduects();
        #Maybe i can start coro routines here to get all listed detail
+s simultaneously
    foreach my $product (@products)
    {
        #fetch product details over api
        push @data,$product->getdetails;
    }
    return ($err,\@data);
}

[download]

In reply to Do I need to use Coro instead of threads/forks by mohan2monks

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.