Hi all,
I found a better solution so I'll just post it at the end of original post, as it does not explain the strange print issue of the original snippet, the new code is shorter less expensive and more correct
following is part of a threaded script for an HPC job scheduler ( cluster ), that notifies a file parser when a file ( job ) is copied in a specific folder using SMB. ( parser will remove it afterwards, job files are not supposed to be altered.
It works well, but for some reason, -and only when 2 clients are simultaneously copying different files to the watch folder- the print command at the end is called without a reason ( and without running the rest of the code ).
I hope someone can shed some light on this behavior. I also wonder if there's any other method to check if the SMB server has any filelock in place ( I obviously only want to parse complete files > perl should wait until file is complete ).
I properly commented it
$smbroot = shared folder by samba server
$sharedfolder = folder inside $smbroot
#!/usr/bin/perl
use strict;
use warnings;
use threads;
use threads::shared;
use Linux::Inotify2;
my @jobstack :shared ; # shared array for filenames
# spawn thread sub
# start inotify in separate thread
my $thr1 = threads->create(\&jobinotifydetect);
$thr1->detach();
#main
while (1) {
sleep 1;
{
# here comes main > for now remove every second a file out of stac
+k
lock (@jobstack);
# I can't splice shared arrays so I copy it onto a nonshared one >
+ later I need splice to pull out specific files 1st
# now it's just a pop replacement
my @temp=@jobstack;
my $t = scalar (@temp);
splice (@temp,$t-1,1) if ($t );
@jobstack=@temp;
}
}
sub jobinotifydetect {
my $smbroot = "/media/lrfp/";
my $sharedfolder = "jobs/";
my $path = $smbroot . $sharedfolder ;
my $jobinotify;
my $file;
my $count;
my $tt;
$jobinotify = new Linux::Inotify2 or die "unable to create new inotify
+ object: $!";
# add watcher
$jobinotify->watch ("$path", IN_MOVED_TO | IN_MODIFY | IN_CREATE, s
+ub {
my $e = shift;
if ( $e->IN_MOVED_TO || $e->IN_MODIFY || $e->IN_CREATE ) {
# wait for file locks to be removed > when data is still written b
+y server
my $file = $e->name;
my $t=$e->fullname;
my $exec = `smbstatus | grep "EXCLUSIVE" | grep "$sharedfolder" |
+grep "$file"`;
if (!($exec)) {
# check if filesize <> 0,don't if file has zero length
if ( -s $t ) {
{
# lock @jobstack while I push filename onto it
lock (@jobstack);
# don't push onto stack if already exist
if (!( grep( /^$file/, @jobstack ) ) ){
push (@jobstack,$file);
$tt=scalar (@jobstack);
print "$count \t $file \t $tt \n";
$count = 0;
}}}
} else {
# count how much times routine gets called just out of curiosity,
+I'll remove this later
$count +=1 ;
}
}
});
1 while $jobinotify->poll;
}
when I copy 2 files from a single smb client I get a result ( as it should ) that is similar to this
1358 a.dd 1
458 b.dd 2
( last nr is amount of files in array )
but when I copy 2 files simultaneously from 2 different smb clients ( 2 different computers ) I'm getting something like this which is according to me not even possible as the array size doesn't increase while calling the print command ( there is only 1 print command that is called after a push ). Even stranger is that results are printed with a 1 sec interval after the 1st file is finished copying.
1266 a.dd 1
18 a.dd 1
43 a.dd 1
34 a.dd 1
56 a.dd 1
49 a.dd 1
34 a.dd 1
3 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
0 b.dd 2
any help is greatly appreciated, an alternate method to check smb filelocks even more.
thanks in advance.
Jan
new code
#!/usr/bin/perl
use strict;
use warnings;
use threads;
use threads::shared;
use Linux::Inotify2;
my @jobstack :shared ; # shared array for filenames
# spawn thread sub
my $thr1 = threads->create(\&jobinotifydetect); # start inotify in se
+perate thread
$thr1->detach();
#main
while (1) {
sleep 1;
{
# here comes main > for now remove every second a file out of stac
+k
lock (@jobstack);
shift (@jobstack);
}
}
sub jobinotifydetect {
my $smbroot = "/media/lrfp/";
my $sharedfolder = "jobs/";
my $path = $smbroot . $sharedfolder ;
my $jobinotify;
my $file;
$jobinotify = new Linux::Inotify2 or die "unable to create new inotify
+ object: $!";
$jobinotify->watch ("$path", IN_MOVED_TO | IN_CLOSE_WRITE, sub {
my $e = shift;
my $file = $e->name;
my $t=$e->fullname;
#only do for files with non-zero size
if ( -s $t ) {
if ( $e->IN_MOVED_TO || $e->IN_CLOSE_WRITE ) {
# don't push onto stack if already exist
if (!( grep( /^$file/, @jobstack ) ) ){
{
lock (@jobstack);
push (@jobstack,$file);
}}}}
# }
});
1 while $jobinotify->poll;
}
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
| |
For: |
|
Use: |
| & | | & |
| < | | < |
| > | | > |
| [ | | [ |
| ] | | ] |
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.