in reply to DBI/DB2 slow on Fedora, fast on SUSE

If the two Linux distros are running on the same hardware and using the same filesystem, here are the three components I would investigate:
  1. Kernel config
  2. libc
  3. Perl (et al.)
Personally, I think it would take some really messed up configurations for any one of these to cause an order-of-magnitude performance difference out-of-the-box (meaning un-modified distro packages) but sufficiently un-optimized builds of all three together probably could.

The kernel is still a bit of black magick for many folks. I apologize if I'm telling you stuff you may already know, but my experience is that simple point-releases in the Linux kernel can have dramatically different performance characteristics. If the two distros are running different versions of the kernel, that can explain some of the difference.

Also, keep in mind the configuration of the kernel - are the IO-related pieces loaded as modules or compiled in? Are there additional modules loaded or compiled in that one kernel has and the other does not? Has the kernel been compiled for a more specific/newer processor? What about optimization settings?

Of course, the kernel alone does not seem to me that it could cause such a disparity in performance.

Next to check libc - grab the source RPMs for each and compare the .spec files to figure out if there's any significant differences in compiled-in options and optimization parameters. Since nearly all the software on a system uses it at run-time a poorly-optimized libc can hurt performance in myriad ways

Perl can get a little trickier - the compilation options are significant, but the bundled versions (and compilation options) of various modules and such can make a *big* difference.

For example (and this is what brought me to the Monks!) - on an Ubuntu 7.04 system I was happily using the DBM::Deep module from the distro repository. When I used CPAN to update this module, I ended up with an over 100x slowdown. (It may have been more - but after 24 hours I wasn't going to wait any longer for my script to finish)

After some basic profiling and some testing on another system, I discovered that perl was spending all it's time computing MD5 hashes - the MD5 module had been upgraded and was significantly slower. Not only that, but the newer version of DBM::Deep was slower as well. The combination was multiplicative. Being against a deadline, I didn't dig any deeper.

I won't pretend that I'm a tuning guru, and every environment is unique, but I've been very successful in squeezing the most out of some pretty loaded systems where I work and these are the things I tend to look at first when in a similar situation as yours.

However, all this is only necessary if you're somehow *stuck* with the distro that performs poorly. The best solution is likely the simplest - use the fast one! Of course PM is a place for curious people like ourselves and understanding is so often worth the effort... So best of luck to you!
  • Comment on Re: DBI/DB2 slow on Fedora, fast on SUSE

Replies are listed 'Best First'.
Re^2: DBI/DB2 slow on Fedora, fast on SUSE
by hrr (Monk) on Jul 26, 2007 at 13:38 UTC
    Hercynium, thank you for your checklist for hunting down performance issues -- it is helpful for me and will be for other people!

    Just for the record, here are the versions used on both systems:
       SUSE: kernel-default-2.6.18.8-0.3.i586, glibc-2.5-25.i686
       Fedora: kernel-2.6.20-1.2952.fc6.i586, glibc-2.5-18.fc6.i686

    Since the Fedora server is at a remote location, I decided not to install a different distribution on it; I will just use the perl packages from SUSE that perform better.

      You "will just use the suse packages on the fedora installation"???? Muahahahahhah - No wonder your sh1+t don't work, man!!! :-) You gotta get down and dirty promiscuously up close and personal with compiling from source.

      Across our workstations, but most importantly- on our development and application servers- we do minimal distro deployment (just the main stuff for the os, X11 gui, etc). And the rest from source. Screw the packages- they always do something stupid. Yum is great for little things like web browsers, image applications, whatever. Little things. But when you are talking performance - the safest thing is to go get the actual architecture independent source. And compile via the cli. It might be a litte bit of a pain sometimes, but the stuff works. Nothing like having that binary sit on your hardware like butter on bread. And your distro is practically irrelevant.

      You gotta get down and dirty promiscuously up close and personal with compiling from source. I've had stuff that felt slow all of a sudden double in speed because I've had the wrong stuff.

      Also, check that your yum conf actually is getting the right architecture for your system. If you run the nightly yum, it could cause trouble.

        Yes, compiling from source is fun and you can often gain a better performance if you do it right. Actually, my system currently runs at about 50% load, so a few extra percent of performance are not absolutely necessary (but a 10x slowdown would not work). So your suggestions, which might be reasonable for your many workstations, would be out of scale in my case.

        Also, no nightly runs of yum, just set up iptables and only allow ssh + http. If there is a security problem with one of these, it will make big tech news and I fix my server easily. Even if it is down for a few days, I will not loose thousands of dollars, so this is a small and acceptable risk.