in reply to Re^3: multi threading DBI
in thread multi threading DBI

Thanks for offering, I have cut out the main part here: 
my $dbh = DBI->connect("dbi:SQLite:dbname=event.db3", "", "", { RaiseE
+rror => 1, AutoCommit => 0 }, ) or die $DBI::errstr;
my @arr_uname = @{$dbh->selectcol_arrayref("select uname from event")}
+;
$dbh->disconnect();

foreach $uname(@arr_uname) {

   ###
   ###   all the LWP::Simple and the parsing happens here
   ###
   ###  [deleted]

   ### output result
   if ($result ne '') {
      $dbh->do("INSERT OR IGNORE INTO detail_info (id,uname,url,thumb,
+txt) VALUES (?,?,?,?,?)", undef, $$result{'id'},$uname,$$result{'url'
+},$$result{'thumb'},$$result{'txt'});
      $dbh->do("INSERT OR IGNORE INTO score_info (id,uname,tid1,tid2,t
+eam1,team2,) VALUES (?,?,?,?,?,?)", undef, $$result{'id'},$uname,$$re
+sult{'tid1'},$$result{'tid2'},$$result{'team1'},$$result{'team2'});
      $dbh->commit; 
   }
}

[download]

Replies are listed 'Best First'.
Re^5: multi threading DBI
by BrowserUk (Patriarch) on Oct 30, 2013 at 20:08 UTC

    Hm. That's not a "working, cut-down version" ...

    So this isn't working either, but it ought to be close once you've filled in the blanks:

    #! perl -slw
use strict;
use threads;
use Threads::Queue;
use LWP::Simple;
use DBI;

sub fetchNParse {
    my( $Qin, $Qout ) = shift;
    while( my $uname = $Qin->dequeue ) {
        my $content = get $uname;
        my @bits = $content =~ m[....];
        $Qout->enqueue( join $;, $uname, @bits );
    }
    $qOut->enqueue( undef );
}

our $THREADS //= 16;

my $Qunames = new Thread::Queue;
my $Qrets   = new Thread::Queue;

threads->create( \&fetchNParse, $Qunames, $Qrets )->detach for 1 .. $T
+HREADS;
my $dbh = DBI->connect( ... );
my $unames = $dbh->selectcol_arrayref("select uname from event");

$Qunames->enqueue( @$unames );

for( 1 .. $THREADS ) {
    while( my $ret = $Qrets->dequeue ) {
        my( $uname, @bits ) = split $;, $ret.
        $dbh->do( insert stuff );
    }    
}
$dbh->disconnect;

    [download]
    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
      Thank you, but I have a bunch of termination errors in "fetchNParse": Thread 1 terminated abnormally: Can't call method "enqueue" on an undefined value at test.cgi line 16. seems $Qout is causing problem?
[reply]

        I'm not psychic. Post the (exact) code that is giving you the errors.

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom