in reply to POE Component HTTP problem
I finally got around to have a look at this code and the first thing that jumped out and screamed at me was that you are blocking in client_start whilst reading and parsing the CSV input file. What is happening is that the while loop blocks the kernel, whilst you are enqueuing the requests to PoCo-Client-HTTP. These requests are sitting enqueued until the while loop ends and client_start finishes and then in one big whoosh are executed.
As you appear to be using strace to diagnose the problem you are using Linux I guess, so you can use POE::Wheel::ReadWrite and POE::Filters to parse your input file and be a lot more co-operative
Only amendments shown:use POE qw(Wheel::ReadWrite Filter::Stackable Filter::Line Filter::CSV +); # add extra handlers for "file_input" "file_error" POE::Session->create( inline_states => { _start => \&client_start , _stop => \&client_stop , got_response => \&client_got_response , transfer_complete => \&finalize_transfer , file_input => \&file_input , file_error => \&file_error } ); sub client_start { my ($kernel, $heap) = @_[KERNEL, HEAP]; ## $poe_kernel->sig(INT => "_stop"); my $fh = IO::File->new( 'dealermade_pictures.csv', 'r' ); $heap->{file} = POE::Wheel::ReadWrite->new( Handle => $fh, Filter => POE::Filter::Stackable->new( Filters => [ POE::Filter::Line->new(), POE::Filter::CSV->new(), ], ), InputEvent => 'file_input', ErrorEvent => 'file_error', ); $heap->{_header} = 0; return; } sub file_input { my ($kernel,$heap,$fields) = @_[KERNEL,HEAP,ARG0]; unless ( $heap->{_header} ) { $heap->{_header}++; return; } my ( $picid, $url, $lot, $is_primary ) = @$fields; my $temp = generate_tempname( $url ); if ( -e $temp && -f $temp && -s $temp ) { $kernel->post( dmua => request => got_response => HEAD( $url ) +); } else { $kernel->post( dmua => request => got_response => GET( $url, Ac +cept => 'image/*' ) ); } return; } sub file_error { my ($kernel,$heap) = @_[KERNEL,HEAP]; delete $heap->{file}; return; }
The reading and parsing of the input file is now asynchronous and HTTP requests are enqueued and happening asynchronously whilst the input file is being read.
|
|---|