in reply to Re: LWP::UserAgent non-blocking calls
in thread LWP::UserAgent non-blocking calls
Open file handles, open database transactions and (other) open network connections. For forking to be clean without side effects, i would need to close all those, fork and then reopen them in the parent process.
My worker processes are cyclic executives that run many modules, each doing it's own thing. The main loop is basically something like this:
... ##### MAIN TIMING LOOP ##### sub run($self) { my $runok = 0; eval { # Let STDOUT/STDERR settle down first sleep(0.1); my $nextCycleTime = $self->{config}->{mincycletime} + time; while(1) { my $workCount = $self->{worker}->run(); my $now = time; if($now < $nextCycleTime) { my $sleeptime = $nextCycleTime - $now; #print "** Fast cycle ($sleeptime sec to spare), sleep +ing **\n"; sleep($sleeptime); $nextCycleTime += $self->{config}->{mincycletime}; #print "** Wake-up call **\n"; } else { #print "** Slow cycle **\n"; $nextCycleTime = $self->{config}->{mincycletime} + $no +w; } } $runok = 1; }; if(!$runok) { suicide('RUN FAILED', $EVAL_ERROR); } return; } ... #### WORKER CYCLE #### sub run($self) { my $workCount = 0; # Run cleanup functions in case the last cycle bailed out with cro +ak foreach my $worker (@{$self->{cleanup}}) { my $module = $worker->{Module}; my $funcname = $worker->{Function} ; #$workCount += $module->$funcname(); $module->$funcname(); } # Notify all registered workers about dead children while((my $child = shift @deadchildren)) { foreach my $worker (@{$self->{sigchld}}) { my $module = $worker->{Module}; my $funcname = $worker->{Function} ; $workCount++; $module->$funcname($child); } } # Run all worker functions foreach my $worker (@{$self->{workers}}) { my $module = $worker->{Module}; my $funcname = $worker->{Function} ; $workCount += $module->$funcname(); } # Run cleanup functions foreach my $worker (@{$self->{cleanup}}) { my $module = $worker->{Module}; my $funcname = $worker->{Function} ; #$workCount += $module->$funcname(); $module->$funcname(); } return $workCount; } ...
It may not be very elegant, but it's simple, easy to debug and easy to balance worker loads by moving module configurations between different XML config files. Yeah, the stuff dynamically configures itself on startup.
Yes, depending on strict limits to the configuration of a specific worker, i can support forking. But this makes it a pain to work with, which is why i have been going to full "use non-blocking stuff" route for the last decade in my workers. Most of the stuff is low level protocols or web APIs that are already fire-and-check-back-later. I haven't encountered a web API that blocks for up to a minute in like 15 years, so i was unaware which modules were available to solve/circumvent that specific problem.
|
---|