Re^5: dbiproxy and compression

Am I missing something obvious? Is this something folks use regularly?

From your description, it sounds like this might have been a feature that was contemplated but not completed. I expect you'll have to ask Tim Bunce or someone else in the DBI team to get the full story.

I don't bother doing compression for traffic going to a database -- in my case, the data's not that frequent or lumpy, I don't have that many servers, and the local network is a 100Mb/sec guy. However, I do have tasks that regularly rsync data between servers, and some of that is text and highly compressible. Happily, rsync takes care of that automagically, so I never have to think about it.

Are you trying this out as an experiment, or do you have good information that your system could greatly benefit from this feature? I'm just worrying you may be doing premature optimziation.

Alex / talexb / Toronto

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

Comment on Re^5: dbiproxy and compression Select or Download Code

Replies are listed 'Best First'.
Re^6: dbiproxy and compression by bcrust (Novice) on Nov 07, 2007 at 17:17 UTC
After a few days of digging, it really does seem that the data compression is probably just half-baked. Yes, this is more of an experiment - perhaps "polishing the turd" is more apt. I have a large number of application servers with large amounts of logs from legacy applications. Without touching the application logging, I am aggregating the logs to a central database. It is working fine. However, since our apps are not CPU-bound, I wanted to see if I received any sort of network benefit of adding compression to the mix. This appeared to be low-hanging fruit that simply required dbiproxy configured and running and a change the DSN on the client side. The data being sent to the DB is generally not that large, though it certainly can be - I wouldn't be surprised if the overhead of Compress::Zlib surpassed the transmission time savings, but I wanted to try it out. I see that Tim Bunce co-wrote the book that mentioned the compression as an alternative - I'll try that route. Thanks Ben	[reply]

Replies are listed 'Best First'.

Re^6: dbiproxy and compression
by bcrust (Novice) on Nov 07, 2007 at 17:17 UTC

Yes, this is more of an experiment - perhaps "polishing the turd" is more apt. I have a large number of application servers with large amounts of logs from legacy applications. Without touching the application logging, I am aggregating the logs to a central database.

It is working fine. However, since our apps are not CPU-bound, I wanted to see if I received any sort of network benefit of adding compression to the mix.

This appeared to be low-hanging fruit that simply required dbiproxy configured and running and a change the DSN on the client side.

The data being sent to the DB is generally not that large, though it certainly can be - I wouldn't be surprised if the overhead of Compress::Zlib surpassed the transmission time savings, but I wanted to try it out.

I see that Tim Bunce co-wrote the book that mentioned the compression as an alternative - I'll try that route.

Thanks

Ben

[reply]