Hi monks i am here again to seek your wisdom..
I have a cgi script (which i am trying to turn into psgi app Change cgi script to PSGI app)
which has to connect to different vendor api's and collect data and return in as a json/xml output
I am trying to fetch this data simultaneously from each vendor by creating threads for each vendor.
I need to create max 10 threads.
But now vendor api has changed and i will have to make additional api calls in each thread.
Example Call a vendor api get list of products then fetch details of each over a api call with product id.
If i start threads again in a thread already it crashes and memory increases too, plus i need to load too much modules (may be i don't know how to do it correctly)
I have been searching perlmomks and net for probable alternatives and found that as my script usually wastes lot of time in waiting for SOAP::Lites calls i may be better off with using Coro instead of threads. (one of the reference Why Coro?)
Again by reading on perlmonks i changed to using forks/forks::BerkeleyDB (forgive me i don't have reference to this now).
It worked fine till it was not needed to use threads in already running threads. If i fork again foreach product apache has too many children.
I actually tried it on a test system and my admin is not nice to me since :(
Can i safely use Coro to do this?
I am a bit confused about how to explicitly use Coro's cede function.(where to put it exactly inside threads)
If i cede it inside a thread will that thread run again?
Will all my vendor functions that i start in main thread run in parallel?
I have read
http://http://search.cpan.org/~mlehmann/Coro-6.41/Coro.pm this in details but i am confused.
Please enlighten me..
#!/usr/bin/perl
#use forks::BerkeleyDB;
#use threads;
use strict;
use Coro;
use CGI qw(:standard);
use XML::Simple;
use JSON qw(encode_json);
use MIME::Base64 qw(encode_base64);
use DBI;
use SOAP::Lite;
use Data::Dumper;
use CHI;
my (@req,@odata,@errdet)=();
my %threads=();
foreach my $vnd (@req){
#$threads{$vnd}=threads->create({ 'context' => 'list', 'exit'
+=> 'thread_only' },\&startthread,$vnd,$SCH);
$threads{$vnd}=async{&startthread($vnd,$SCH) };
}
cede;
foreach my $vnd (keys %threads)
{
my($err,$data) =$threads{$vnd}->join();
if ($err eq 'N') {
push @odata,@{$data};
}
}elseif($err eq 'Y'){
push @errdet,$data;
}elsif($threads{$vnd}->error) # with threads it was useful,
{
push @errdet,{ Vnd=>$vnd, ErrorCode=>26,ErrorMsg=>$threads
+{$vnd}->error};
}
}
sub startthread
{
my ($vendor,$SCH)=@_;
my($err,$data) =();
require './'.$vendor.'functions.pl"; #load vendor specific code
($err,$data)=&getvendordata($SCH);
return($err,$data);
}
#vendor functions has somewhat code like this besides functions specif
+ic to each vendor api
sub getvendordata
{
my $SCH=shift;
#perform vendor specific tasks mostly SOAP::Lite calls to api
#Has to perform multiple calls some of them depend on output of pr
+evious calls
#Also have to make simultaneous calls which may not be related to
+each other can be done parallelly.
#e.g. first call to api returns a list of products subsequent call
+s to fetch details.
#If i start threads/forks again here most of the time program cras
+hes :(
my ($err,@data)=();
my @products=getproduects();
#Maybe i can start coro routines here to get all listed detail
+s simultaneously
foreach my $product (@products)
{
#fetch product details over api
push @data,$product->getdetails;
}
return ($err,\@data);
}