bcrust has asked for the wisdom of the Perl Monks concerning the following question:

I have a script that connects to a Postgres DB and writes data. Simple enough - however, this script will be run from multiple hosts and send loads of data.

I would like to toy with compressing the data to save on some network bandwidth and came across dbiproxy. I have dbiproxy running on the DB host and my script connecting to it successfully on the client hosts. However, when I attempt to add compression, I have failures.

I am running the dbiproxy thusly:

/usr/bin/dbiproxy --debug --configfile proxy_postgres.cfg --compression gzip

My script is connecting as such:

$dsn = 'DBI:Proxy:hostname=build;port=12400;compression=gzip;dsn=DBI:P
+g:';
$db_user = 'foo';
$db_pass = 'bar';
$dbh = DBI->connect($dsn, $db_user,$db_pass, {RaiseError => 1});

[download]

From dbiproxy, I get:

err, Child died: Unexpected EOF from client at
/usr/lib/perl5/vendor_perl/5.8.8/RPC/PlServer.pm line 146

[download]

I've stepped through the dbiproxy and it appears that when I have compression set, it attempts to uncompress every single message that comes through the socket, including the 'connect'. However, I can't find any mention of compressing in DBI, DBD::Pg, etc.

I've played around with a couple of versions of the components (perl 5.8.8/DBI 1.50 and perl 5.8.7/DBI 1.48).

The man page for dbiproxy and DBI::ProxyServer don't mention compression either, though dbiproxy's help option lists it.

Frankly, the only place I have seen this mentioned is in O'Reilly's Programming with DBI book and some sundry Google hits.

My question? Is this supposed to work? Is there an alternative?

Replies are listed 'Best First'.
Re: dbiproxy and compression
by talexb (Chancellor) on Nov 06, 2007 at 16:16 UTC

    Silly question, perhaps, but do you have compression enabled on the DBI::ProxyServer server? I just had a look at the module page on CPAN, and I didn't see a 'compression' argument.

    Obviously, both the server and the client have to have compression (the same compression) turned on for this to work.

    Alex / talexb / Toronto

    "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

[reply]
      Yes, I ran the dbiproxy with --compression. It seems that my problem is on the client side.

      I ran the proxy in the debugger and stepped through the code as it received the connection. It certainly attempted to Compress::Zlib::uncompress on every message received from the socket. The problem was that the message was not compressed.

      However, I can find no mention in the client-side connection libraries that compression was an option. I added compression=gzip to the DSN (as suggested in the O'Reilly book, but not in any PODs) to no effect. Is this just a pipe dream?

      Has anyone successfully ever gotten DBI::Proxy to talk to dbiproxy (DBI::ProxyServer) with compression enabled?

      Alternately, is there another way to transparently compress data with the DBI modules?

      Ben

[reply]
          The problem was that the message was not compressed.

        OK -- so why wasn't the message compressed? Is Compress::Zlib not installed on the client side?

        Am I missing something obvious?

        Alex / talexb / Toronto

        "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

[reply]

Back to Seekers of Perl Wisdom