in reply to The question of URL queue selection module ?

Now I use BerkeleyDB module to store the URL queue, the script is able to work, but the efficiency is not high.Because when I use multiple script to read the database db, I found multiple script can't read at the same time . They were blocked. After completing the script will turn to a script.

Hi,

What do you mean by that? What are you using exactly and how?

  • Comment on Re: The question of URL queue selection module ?

Replies are listed 'Best First'.
Re^2: The question of URL queue selection module ?
by Perl_Love (Acolyte) on Aug 13, 2016 at 01:58 UTC

    Hi

    ......
my $env=new BerkeleyDB::Env
    -Home=>'/home/XXX/DB',
    -Flags=>DB_CREATE|DB_INIT_MPOOL
|| die;
my $db=tie(%hash,"BerkeleyDB::Btree",
    -Filename=>"URL.db",
    -Flags=>DB_CREATE,
    -Env=>$env)
|| die;
while(1){
    while(my ($k,$v)=each %hash){
        if($num == $coro_max){
            &coro_url;
        }
        else{
            ++$num;
            push(@pages,$k) if($k ne '');
        }
    }
}
......

    [download]
    BerkeleyDB is blocking? I need Non-blocking.
[reply]
[d/l]
[reply]

In Section Seekers of Perl Wisdom