Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Fresh eyes - linux::Inotify2 event delays

by ghenry (Vicar)
on Apr 06, 2022 at 12:40 UTC ( [id://11142730]=perlquestion: print w/replies, xml ) Need Help??

ghenry has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

The following code works fine and has done for a while, but sometimes I am seeing events for new mp3s coming two hours or random time amounts after they are saved to the directories. We never miss events, just as if they are buffered by the kernel. I'm thinking either my 'Retry', but doesn't show in syslog, or ->poll needs to be switched to ->read. Any pointers much appreciated.

I've left out find_call_details() and email_recording() as it's the "$fullpath was created fully." that's not appearing in syslog, does can do hours later. Not a syslog issue though, as we have this basic code running too which shows the mp3 being created and logs it to syslog - https://pypi.org/project/inotify/. Most time our Perl version will log at the same time, but we may get a burst of events later. We still get them though.
use strict; use warnings; use DBI; use Linux::Inotify2; use MIME::Lite::TT::HTML; use Sys::Syslog; use File::Basename; use File::Find; use DateTime; use Getopt::Long; use Retry::Backoff 'retry'; use Data::Validate::UUID qw( is_uuid ); use Net::SMTP; my $debug; GetOptions( 'debug' => \$debug, ) or die("Error in command line arguments\n"); ######################################### # 1. Build initial file list and poll ######################################### my %recording_emailed; my $directory1 = "/blah1"; my $directory2 = "/blah2"; openlog( 'callrecordingemailer', 'ndelay,pid', 'LOG_LOCAL0' ); my $inotify = Linux::Inotify2->new() or die "Unable to create new inotify object: $!"; my $age = 0; find( \&add_most_recent_dirs, $directory1 ); # Reset for second run $age = 0; find( \&add_most_recent_dirs, $directory2 ); $inotify->poll while 1; sub add_most_recent_dirs { my $name = $File::Find::name; return if defined $age && $age > ( stat($_) )[9]; if ( -d $name ) { $age = ( stat(_) )[9]; syslog( 'info', "Found: $name" ) if $debug; watch_for_files($name); } } ######################################### # 2. Watch for files ######################################### sub watch_for_files { my $directory = shift; syslog( 'info', "Going to watch: $directory" ) if $debug; $inotify->watch( $directory, IN_CREATE | IN_CLOSE_WRITE | IN_DELETE_SELF, sub { my $e = shift; my $fullpath = $e->fullname; if ( $e->IN_DELETE_SELF ) { syslog( 'info', "Stopping watching directory: $fullpat +h" ) if $debug; $e->w->cancel; } if ( $e->IN_ISDIR ) { syslog( 'info', "Now watching directory for mp3 files: $fullpath" +) if $debug; watch_for_files($fullpath); } if ( $e->IN_CLOSE_WRITE ) { my $cdr_uuid = basename( $fullpath, '.mp3' ); if ( is_uuid($cdr_uuid) ) { my $filename = basename($fullpath); syslog( 'info', "$fullpath was created fully." ) i +f $debug; # https://metacpan.org/pod/Retry::Backoff my ( $email, $call_details ); retry { syslog( 'info', "Trying to find CDR for call_u +uid $cdr_uuid..." ) if $debug; ( $email, $call_details ) = find_call_details($cdr_uuid); } initial_delay => 5, on_failure => sub { my $h = shift; syslog( 'info', $h->{error} ) if $debug; }; if ( defined $email ) { email_recording( $email, $fullpath, $filename, $call_details ); } } } } ); }

Replies are listed 'Best First'.
Re: Fresh eyes - linux::Inotify2 event delays
by choroba (Cardinal) on Apr 06, 2022 at 14:14 UTC
    Unfortunately, I don't see anything fishy at the first sight.

    I usually don't use Linux::Inotify2 (there were some problems installing it, IIRC), I just

    open my $inotify, '-|', do { no warnings 'qw'; qw( inotifywait -m -r -e close_write -e moved_to --exclude /\ +.git/|~$|/#[^/]+# . ) } or die $!; while (<$inotify>) { my ($path, $event, $file) = split; ...

    It seems you ignore IN_MOVED_TO while I react to it. Maybe it can be the problem, or not?

    map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]

      Thanks for taking the time to read and reply. Nah, we are watching folders that FreeSWITCH creates mp3 files in and only IN_CLOSE_WRITE is of interest when it closes the file descriptor.

      It's a tricky one. I'm going to dig into 'retry' and maybe try your way too.

        Just to offer a negative data point: we're doing something similar to your use case at $work, that is, we rely on inotify to postprocess call recordings that were finished by Asterisk, monitoring IN_CLOSE_WRITE events, just like you, and to my knowledge we've never seen such random delays in receiving events during the 3+ years this system has been in production. However, in our case the user side inotify implementation is in Go.

        What kernel version do you use? That may be relevant.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11142730]
Approved by hippo
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others sharing their wisdom with the Monastery: (7)
As of 2024-04-24 09:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found