bernuli has asked for the wisdom of the Perl Monks concerning the following question:

Greetings Monks,

Using Perl v5.36.0 on macOS configured with defaults.

I would like to use the macOS specific F_NOCACHE flag when opening a filehandle. For example:

use Fcntl; sysopen(my $FI, "testFile.txt", O_RDONLY | F_NOCACHE ) or die "Could not open 'testFile.txt' $!";

However, Fcntl.pm does not export F_NOCACHE even though it is defined in my fcntl.h:

 #define F_NOCACHE       48              /* turn data caching off/on for this fd */

Is there a configuration option to build perl with a Fcntl perl module that supports the F_NOCACHE flag to sysopen? Or is there another way to pass this flag through sysopen?

I am opening numerous large files that do not need to stay in memory, so the macOS file cache quickly fills up with files that will not be referenced again. Not that big a deal but macOS will drop other files from the file cache as my script is running, and those other files would be better off staying in memory.

Also, in benchmarking scenarios, I would rather read files from disk.

Thanks for any info!

Replies are listed 'Best First'.
Re: macOS Fcntl.pm missing F_NOCACHE
by kcott (Archbishop) on Oct 10, 2022 at 01:51 UTC

    G'day bernuli,

    Welcome to the Monastery.

    I haven't had a macOS system for some years, so I'm unable to test this for you; also, I don't recall ever using F_NOCACHE or seeing it documented.

    The following is a possible workaround. This is on v5.36 but using Cygwin; I'm not expecting F_NOCACHE to be found. I ran some demo tests using this common alias of mine which reports many types of problems:

    $ alias perle alias perle='perl -Mstrict -Mwarnings -Mautodie=:all -MCarp::Always -E +'

    O_RDONLY is exported by default but F_NOCACHE is not:

    $ perle 'use Fcntl; say O_RDONLY' 0 $ perle 'use Fcntl; say O_RDONLY; say F_NOCACHE' Name "main::F_NOCACHE" used only once: possible typo at -e line 1. 0 say() on unopened filehandle F_NOCACHE at -e line 1.

    F_NOCACHE cannot be explicitly imported:

    $ perle 'use Fcntl qw{O_RDONLY F_NOCACHE}; say O_RDONLY; say F_NOCACHE +' "F_NOCACHE" is not exported by the Fcntl module Can't continue after import errors at -e line 1. main::BEGIN() called at -e line 1 eval {...} called at -e line 1 BEGIN failed--compilation aborted at -e line 1.

    I can fake it like this (i.e. the "possible workaround"):

    $ perle 'use Fcntl; use constant F_NOCACHE => 48; say O_RDONLY; say F_ +NOCACHE' 0 48 $ perle 'use Fcntl; use constant F_NOCACHE => 48; say O_RDONLY | F_NOC +ACHE' 48 $ perle 'use Fcntl; use enum qw{F_NOCACHE=48}; say O_RDONLY; say F_NOC +ACHE' 0 48 $ perle 'use Fcntl; use enum qw{F_NOCACHE=48}; say O_RDONLY | F_NOCACH +E' 48

    I've no idea idea if sysopen will accept that on your system. Curiously, it doesn't complain on mine:

    $ perle 'use Fcntl; use enum qw{F_NOCACHE=48}; sysopen my $fh, "dummy_ +file", O_RDONLY | F_NOCACHE; close $fh'

    I don't know if there's a way to build Perl such that F_NOCACHE is recognised but someone else might. To help with that, tell us how you built the current Perl you're using. It may also be helpful to show the output of `perl -V` (that's a capital V).

    — Ken

      Thanks Ken and hv!

      Using the raw numeric value worked! After opening the FILEHANDLE I added

      fcntl($FI, 48, 1);

      Which accomplished what I wanted. But I found the read performance to be much worse than expected. Also CPU usage went way up. Probably due to disk IO having to be managed more closely VS just buffering right into RAM.

      So I will end up not using the F_NOCACHE for now. But good to know the option is available if I need it.

      Thanks again!

      Summary of my perl5 (revision 5 version 36 subversion 0) configuration +: Platform: osname=darwin osvers=19.6.0 archname=darwin-2level uname='darwin test-mac.local 19.6.0 darwin kernel version 19.6.0: +thu jan 13 01:26:33 pst 2022; root:xnu-6153.141.51~3release_x86_64 x8 +6_64 ' config_args='' hint=recommended useposix=true d_sigaction=define useithreads=undef usemultiplicity=undef use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define Compiler: cc='cc' ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.15 -fn +o-strict-aliasing -pipe -fstack-protector-strong -DPERL_USE_SAFE_PUTE +NV' optimize='-O3' cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.15 -fn +o-strict-aliasing -pipe -fstack-protector-strong' ccversion='' gccversion='4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.32.62 +)' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='cc' ldflags =' -mmacosx-version-min=10.15 -fstack-protector-strong' libpth=/Applications/Xcode.app/Contents/Developer/Toolchains/Xcode +Default.xctoolchain/usr/lib/clang/11.0.3/lib /Applications/Xcode.app/ +Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sd +k/usr/lib /Applications/Xcode.app/Contents/Developer/Toolchains/Xcode +Default.xctoolchain/usr/lib /usr/lib libs=-lpthread -ldbm -ldl -lm -lutil -lc perllibs=-lpthread -ldl -lm -lutil -lc libc= so=dylib useshrplib=false libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=bundle d_dlsymun=undef ccdlflags=' ' cccdlflags=' ' lddlflags=' -mmacosx-version-min=10.15 -bundle -undefined dynamic_ +lookup -fstack-protector-strong' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_USE_SAFE_PUTENV USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF Built under darwin Compiled at Oct 7 2022 11:20:10 @INC: /usr/local/lib/perl5/site_perl/5.36.0/darwin-2level /usr/local/lib/perl5/site_perl/5.36.0 /usr/local/lib/perl5/5.36.0/darwin-2level /usr/local/lib/perl5/5.36.0
        "Using the raw numeric value worked!"

        That's basically good news and allows you to move forward.

        Having said that, raw numbers appearing in your code is less good: their purpose would be unclear to a new maintainer, and possibly even to you when the code is revisited at some future time. I'd consider using F_NOCACHE, adding a comment to explain what it is, and maybe even a condition to only use it for macOS.

        Although moot if you've decided not to currently use F_NOCACHE, that advice is valid generally, whenever numbers just appear in your code. I often use constant to provide names for array indices: for example, $data[PHONE] would be clearer than $data[7].

        — Ken

Re: macOS Fcntl.pm missing F_NOCACHE
by hv (Prior) on Oct 10, 2022 at 01:56 UTC

    The Fcntl module only provides access to the list of defines it knows about. The best I can suggest is either to use the raw numeric value with a comment, or put it in a constant sub:

    # OSX-specific define: don't cache this file sub F_NOCACHE () { 48 }

    If you need greater portability, you could wrap it in some XS exactly as Fcntl does, or create something more generic using h2xs (ships with perl) or Alien::SWIG4 to export everything you find in fcntl.h.

    That said, the Fcntl documentation has claimed since docs were originally added in 1995 that "this uses the h2xs program", which does not appear to have been true even then. So you could also consider filing a bug report that Fcntl should do what its docs claim, and make available all constants defined in that header - or just add F_NOCACHE.

    Update: removed spurious '$' from prototype; thanks kcott.

        Oops, thanks, corrected.

      Or, equivalently,

      use constant F_NOCACHE => 48;
      
Re: macOS Fcntl.pm missing F_NOCACHE
by Bod (Parson) on Oct 09, 2022 at 22:17 UTC

    My guess is that you would have to explicitly export F_NOCACHE as you do with other flags according to the documentation.

    use Fcntl qw(F_GETFL F_SETFL O_NONBLOCK);

    So yours would be:

    use Fcntl qw(F_NOCACHE);

    I don't have anything with iOS to be test it.

      Thank you for the reply.

      Unfortunately that does not work as my Fcntl.pm knows nothing of F_NOCACHE

      Adding F_NOCACHE to the @EXPORT array of Fcntl.pm does not work either as the script dies with "F_NOCACHE is not a valid Fcntl macro".