in reply to perl script and queue

Hello k_manimuthu,

It is strange how you call your insert script that fast that MySQL is not able to handle the insert statements. The easy way for me would be to connect your scripts either by creating loops and when you reach the end of the loop call next insert.

Alternatively you can use Thread::Queue. But I would first try to see if I can create an OO approach for the insert statements instead of calling the script by each self.

But this is my idea on how to approach the problem, maybe another monk has another idea.

Update: You can also in case you use the MySQL DB you can call the function INSERT ... ON DUPLICATE KEY UPDATE Syntax.

Update2: Or as fellow monk hippo proposed Ensuring only one copy of a perl script is running at a time.

Hope this helps, BR.

Seeking for Perl wisdom...on the process of learning...not there...yet!

Replies are listed 'Best First'.
Re^2: perl script and queue
by 1nickt (Canon) on Aug 07, 2017 at 14:34 UTC

    The OP never stated that he or she is using MySQL.

    Using a queue for this situation is quite sensible, in theory. Can you explain how to implement one with Thread::Queue?


    The way forward always starts with a minimal test.
[reply]
[d/l]

      Hello 1nickt,

      You are right, the OP never stated that he is using MySQL so I will update the answer.

      Regarding Using a queue for this situation is quite sensible, in theory. Can you explain how to implement one with Thread::Queue? I would approach it like this. Example not real time code:

      I use forks instead of threads because I get the error:

      $ perl test.pl 1 2
This Perl not built to support threads
Compilation failed in require at test.pl line 3.
BEGIN failed--compilation aborted at test.pl line 3.

      [download]

      Having said that, sample of executable code:

      #! /usr/bin/perl
use forks;
use strict;
use warnings;
use Thread::Queue;
use feature 'say';

sub insert_to_db {
    say $_[0];
}

my $q = Thread::Queue->new();    # A new empty queue
# Send work to the threads
$q->enqueue($_) for @ARGV;

# Worker threads
my $thread_limit = 8;

my @thr = map {
    threads->create(sub {
        while (defined (my $item = $q->dequeue_nb())) {
           insert_to_db($item);
        }
            });
} 1..$thread_limit;

# terminate.
$_->join() for @thr;

__END__

$ perl test.pl 1 2
Argument "2.56_01" isn't numeric in numeric ge (>=) at /home/user/perl
+5/lib/perl5/x86_64-linux/forks.pm line 1570.
1
2

      [download]

      Of course the sample of code, in reality instead of ARG I would queue the number of insert statements and execute them in sequence.

      Any way, my approach would be to queue the insert statements in order to complete each insert before the next one.

      Hope this clears things more, BR.

      Seeking for Perl wisdom...on the process of learning...not there...yet!
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom