jamescondron has asked for the wisdom of the Perl Monks concerning the following question:

Forgive me for posting this verbatim from a thread on another forum; and for it sounding a lot like a Linux issue- its not:

As the post suggests, I'm having some issues with Fuse and perl, namely the write_buffer size. Now, this could just be a Fuse issue, and if it is I'd ask a mod to remove this, but I don't think it is.

The background;
Firstly, my implementation is working, and working well. I'm getting reasonable consistency in my read/writes, even with different sizes, and so I'm pretty sure my perl is working well (Which is why I haven't posted any of it, though I can if you need).

I've implemented a very simple, bog standard filesystem with hashrefs, and similar data structures, all looks good. The problem is that the writes are being done in 4096 blocks; the Fuse standard.

Because this is the fuse standard, I did a grep for the write size:

root@jcmain:~# !1810 grep -inH max_write `locate fuse` 2>/dev/null /usr/include/fuse/fuse_common.h:105: unsigned max_write; /usr/include/linux/fuse.h:363: __u32 max_write; .....snip..... /usr/src/linux-2.6.30-custom/fs/fuse/file.c:786: bytes += min_t(size_t, bytes, fc->max_write - count); /usr/src/linux-2.6.30-custom/fs/fuse/file.c:823: } while (iov_i +ter_count(ii) && count < fc->max_write && /usr/src/linux-2.6.30-custom/fs/fuse/file.c:987: size_t nmax = +write ? fc->max_write : fc->max_read; /usr/src/linux-2.6.30-custom/fs/fuse/fuse_i.h:322: unsigned max_w +rite; /usr/src/linux-2.6.30-custom/fs/fuse/inode.c:763: fc->ma +x_write = arg->minor < 5 ? 4096 : arg->max_write; /usr/src/linux-2.6.30-custom/fs/fuse/inode.c:764: fc->ma +x_write = max_t(unsigned, 4096, fc->max_write); /usr/src/linux-2.6.30-custom/include/linux/fuse.h:401: __u32 max_wr +ite; .....snip..... root@jcmain:~# uname -a Linux jcmain 2.6.30-custom #1 SMP Fri Jun 12 22:15:25 BST 2009 i686 GN +U/Linux

and found the module which referenced 4096 on my kernel; /usr/src/linux-2.6.30-custom/fs/fuse/inode.c
I changed 4096 to 131072 (I forget why that number, I think I saw it somewhere whilst researching this, but no idea where/why), recompiled the module, unloaded the old fuse module and loaded this one. I was even convinced it'd worked:

jc@jcmain:~/steg$ sudo perl fs_simple.pl unique: 1, opcode: INIT (26), nodeid: 0, insize: 56 INIT: 7.11 flags=0x0000003b max_readahead=0x00020000 INIT: 7.8 flags=0x00000000 max_readahead=0x00020000 max_write=0x00020000 unique: 1, error: 0 (Success), outsize: 40
0x00020000 being:
jc@jcmain:~$ perl -e "print int 0x00020000;" 131072

Which tells me that perl it's self has set the 4096 from somewhere else. Its not, however, built into Fuse.pm or Fuse::Simple.pm - which is what I'm using:

jc@jcmain:/usr/lib/perl5$ grep 4096 Fuse.pm jc@jcmain:/usr/lib/perl5$ <snip> jc@jcmain:/usr/local/share/perl/5.10.0/Fuse$ grep 4096 Simple.pm jc@jcmain:/usr/local/share/perl/5.10.0/Fuse$

So what am I missing? Does anyone remember solving the same sort of problem? Can it be solved? Is it even considered a problem, more a feature?

Hell, is it even a pure perl issue I'm having? Cheers.

Replies are listed 'Best First'.
Re: Fuse / Fuse::Simple write_buffer issues
by almut (Canon) on Mar 25, 2010 at 11:47 UTC
    The problem is that the writes are being done in 4096 blocks
    (...)
    Which tells me that perl it's self has set the 4096 from somewhere else

    Not really sure, but maybe your problem has to do with PerlIO having a fixed buffer size of 4k (hardcoded in the Perl source).  See Re: 4k read buffer is too small.

      As far as I can see, everything in Fuse is being done by passing hashrefs about, I don't know the hash underlying structure well enough to say for sure, but I shouldn't think it uses the same IO model.

      That being said, this is testable by going into the perl source, I just don't really want to have to if I can help it

        That being said, this is testable ...

        If you post a little (but self-contained) snippet that shows how you tested that writes are being done in 4k blocks, I might run it with my 8k buffer perl I compiled back then, which I still have around.

        Update: ...unfortunately, I couldn't get Fuse to work (with a reasonable amount of effort) on my somewhat older SUSE system at work — module builds fine, but tests fail, although kernel-wise, etc. fuse is in principle working.  I'll see what I can do later at home on my Ubuntu system.

Re: Fuse / Fuse::Simple write_buffer issues
by ikegami (Patriarch) on Mar 25, 2010 at 18:03 UTC
    Perl's buffered IO (read, readline) always do 4k reads, repeating if necessary. sysread, on the other hand, simply passes the requested number of bytes to the OS.
    $ strace perl -e'read STDIN, $_, 12345' </dev/null 2>&1 | grep 'read(0 +' read(0, ""..., 4096) = 0 $ strace perl -e'sysread STDIN, $_, 12345' </dev/null 2>&1 | grep 'rea +d(0' read(0, ""..., 12345) = 0