sachin raj aryan has asked for the wisdom of the Perl Monks concerning the following question:

hello Monks i need help in this code , I am not sure what is causing loop to fail

here is my code

#! /usr/bin/perl use threads; use Thread::Queue; use Net::SFTP; use Net::FTP; use IO::Uncompress::Gunzip qw(gunzip $GunzipError); use Date::Simple qw(d8); use DBI; use DBIx::Dump; use Time::Piece; use Time::Seconds 'ONE_DAY'; use Path::Tiny; use Cwd qw(cwd); use DateTime ; my $outdir = cwd(); my $host = "rptsvr3.app"; my $virtualbase = "/ftparea_bhopal"; my $basedir = "locations"; my $reportfldr = "reports"; my $reportdr ="Transaction_Reports"; my $filename = '/home/sbi/Desktop/SRA/branchList1.txt'; $date=DateTime->now->subtract(days=>3)->ymd(''); open(my $fh, '<:encoding(UTF-8)', $filename)or warn "Could not open fi +le '$filename' $!"; chomp(my @file = <$fh>); close $fh; my $datedel = d8($date); my $f1=$datedel->format ('%d/%b/%Y'); my $q = Thread::Queue->new(); # A new empty queue $q->enqueue($_ = substr $_,0,6) for @file; # Send work to the th +reads print "i m entering program,\n"; my $thread_limit = 3; my @thr = map { threads->create(sub { while (defined (my $item = $q->dequeue_nb())) { maincalc($item,$date); print " $item , i m checking branch ,\n"; } } ); } 1..$thread_limit; $_->join() for @thr; ############Fetching report from LHO Server########################### +############# sub maincalc ($brchid,$date) { my $brchid=$_[0]; my $date=$_[1]; print "Starting $branchname\t"; $brnchid = substr $brchid,1 ; print "$brnchid upload \n"; my $branchdir = join"/",$outdir,$brchid; system "mkdir $branchdir"; $branchfile=join",","Z_CCOD_".$brnchid.".txt"; print "------------------------------------------------------------- +------------I am printing branch file printing $branchfile\t"; open(my $fh4,'>',$branchfile) or die "Couldnt Open file "; close ($fh4); my $datedir = join"/",$branchdir,$date; system "mkdir $datedir"; my $dir = join"/",$virtualbase,$basedir,$brchid,$reportfldr,$date, +$reportdr; print "$dir"; print " FTP is ongoing in main file \n"; my %args =( user =>'sftp_bhopal',password=>'password',debug=>f +alse); my $f = Net::SFTP->new($host,%args,ssh_args=>[port=>'2202',opt +ions=>["MACs +hmac-sha1","StrictHostKeyChecking no"]]); $f->status; $f->ls($dir,sub {print $_[0]->{filename},"\n"}); my $file = join "/",$dir,"CC_OD_Balance_File_depd0580.ID_IN058 +0_ID.txt.gz"; print "$file, i m sachin 1 \n"; my $filerec= join"/",$datedir,"CC_OD_Balance_File_depd0580.ID_I +N0580_ID.txt.gz"; print "$filerec, i m sachin 2\n"; $f->get($file,$filerec); # print " FTP done \n"; # print "Unzipping files now\n"; for my $input ( glob $datedir."/"."CC_OD_Balance_File_dep +d0580.ID_IN0580_ID.txt.gz" ) { my $output = $input; $output =~ s/.gz// ; gunzip $input => $output or warn "Error Uncompressing '$input': $GunzipError\n"; } # print "Unziping done \n"; # print "starting uploading to database \n"; my $d = d8($date); my $f=$d-> format ('%d/%b/%Y'); if (open(my $fh,'< ',$datedir."/"."CC_OD_Balance_File_depd0580.ID_IN0 +580_ID.txt") or warn "Could not open file '$fh' $!") { } print " database upload done\n "; chdir "../.."; }

Here are the files and folders it is generating before failling b00280 b00310 b00318 b00327 Z_CCOD_00280.txt Z_CCOD_00310.txt b00296 b00317 b00320 Z_CCOD_00296.txt

below is the error message

Thread 1 terminated abnormally: threadccod_new.pl: open /home/sbi/Desktop/SRA/CCOD_TEST: fileneme is Z_CCOD_00327.txt$! at threadccod_new.pl line 77.

73 open(my $fh4,'>',$branchfile) or die "Couldnt Open file "; 74 close ($fh4);
75 my $datedir = join"/",$branchdir,$date;

Replies are listed 'Best First'.
Re: Need Help in threading perl
by GrandFather (Saint) on Jul 31, 2024 at 10:37 UTC

    Add strictures (use strict; use warnings; - see The strictures, according to Seuss) and fix up the issues reported by them to start with. Then tidy up your indentation so your code is at least somewhat readable.

    Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond
Re: Need Help in threading perl
by Corion (Patriarch) on Jul 31, 2024 at 10:25 UTC

    Does the problem happen without threads?

    You did not output $branchfile in your error message. Output $branchfile in your error message and inspect if the file already exists at the location and if the current user can create files there.

      currently the max thread is 3 , so it only making 3 txt file. when i increase number of thread to 10 , it makes 10 files and then fails.

      threadccod_new.pl: open : Permission denied at threadccod_new.pl line 75. <\p> Iam not sure why getting permission for 4th and so on file and not for 1st to third file .

        You are using chdir from within a thread. I would avoid that, since it is never clear between operating systems, whether the "current directory" is per-thread or global per program.

        Also, replace

        system "mkdir $branchdir";

        with mkdir:

        mkdir $branchdir;

        Also, output the filename and path in your error message. I already suggested that to you but you ignore it. Please explain why you ignore my advice.

      No

Re: Need Help in threading perl
by ikegami (Patriarch) on Jul 31, 2024 at 19:59 UTC

    One problem is your use of chdir. That changes the process's work directory, so it affects all your threads. This is bad, and probably the reason your open is failing.


    Some improvements:

    my $q = Thread::Queue->new(); my @threads = map { async { # Just nicer # You don't want dequeue_nb here!!! while ( defined( my $item = $q->dequeue() ) ) { # Don't kill the entire thread # if there's a problem with one job. eval { worker( $item ); 1 } or warn( $@ ); } }; } 1 .. $thread_limit; # Do this *after* the threads are created. # Otherwise, you could have a deadlock. # (Ok, not really since we don't place a size # limit on the queue so the queue can grow # to any size, but it's still nicer and # allows us to limit the size of the queue.) $q->enqueue( substr $_, 0, 6 ) for @file; # Make it so the threads exit when the queue becomes empty. $q->end; $_->join() for @threads;

      Thank you for nice improvement and program worked after removing chdir