ultrachrome has asked for the wisdom of the Perl Monks concerning the following question:

I have a script that reads Wireshark decoded output via a pipe, parses, and loads it into a MySQL table.

My current method involves writing the parsed delimited records to a file and then doing a "load data infile" to load MySQL.

This method is 50% faster than directly inserting each parsed record via DBI.

To make this even faster, I thought I'd like to try writing/loading data through a FIFO as shown on MySQL's LOAD DATA INFILE Syntax page but I can't seem to open the FIFO.

From the command line or within perl, I can create the FIFO with no problem. What I can't do is open it using either open or sysopen, the latter method taken directly from the named pipes example in Programming Perl.

The script just hangs at that point with no error.

I made the following example that simply opens then closes the fifo and it also hangs. It prints the warning but never returns a prompt

my $fifo = "fifo";
require POSIX;
POSIX::mkfifo($fifo, 0666) or die "can't mknod $fifo: $!";
warn "$0: created $fifo as named pipe\n";
sysopen(FIFO, $fifo, O_WRONLY) or die "can't write $fifo: $!";
close FIFO;

[download]

I'm running this on Ubuntu 8.04 but I've also tried the code on Solaris 10 x86 with the same result.

Any guidance greatly appreciated. thanks.

Replies are listed 'Best First'.
Re: Trouble opening FIFO for speedy MySQL loading
by Illuminatus (Curate) on Nov 04, 2008 at 03:12 UTC
    From the mysql page you reference:

    Note that you must run the command that generates the data to be loaded and the mysql commands either on separate terminals, or run the data generation process in the background (as shown in the preceding example). If you do not do this, the pipe will block until data is read by the mysql process.

    Trying to open a fifo as write-only will block until something opens it to read.
[reply]

      Yes, I've read that but I'm having trouble demonstrating what it really means.

      Are you saying that the open statement will complete once I begin reading from the file?

      I did try running the load data infile statement while the script was hanging at open but nothing happened.

      Any help would be appreciated. Thanks.

[reply]
        I would need to see exactly what you are trying in order to comment. I created the following script: 
        #!/usr/bin/perl -w
use strict;
use IO::File;

my $fifo = "fifo";
sysopen(FIFO, $fifo, O_RDONLY) or die "can't read $fifo: $!";
close FIFO;

        [download]
        I kicked off your script in background, where it hung. When I run this script, the original script (the one you previously posted) exits, indicating that the open completed.
[reply]
[d/l]

Back to Seekers of Perl Wisdom