|
hrr has asked for the wisdom of the Perl Monks concerning the following question:
When moving my DB2 database from system running SUSE to one running Fedora Core 6, I have noticed a surprising slowdown of perl accessing that database as follows:
Find the same row 10,000 times: 2x slower
Read 100,000 rows: 10x slower (!!)
A devastating slowdown...
Both computers have a dual core "Intel Pentium D CPU 2.80GHz" processor and I am using DBIx::Class (0.08003, most recent) to access a DB2 database (V9.1).
Now I have noticed that SUSE comes with perl compiled for the i586 architecture, while Fedora Core uses perl for i386. So I have installed the following i586 packages from the SUSE repository on my Fedora system:
perl perl-DBI perl-Net-Daemon perl-PlRPC
Then I have recompiled DBD::DB2 and now the queries on the Fedora system run almost as fast (10-20% slower) as on the SUSE system. I hope this is a helpful solution for others, even though it seems somewhat random.
Did someone experience similar problems? Does this have to do with SUSE vs. Fedora builds or with i386 vs. i586 builds?|
Re: DBI/DB2 slow on Fedora, fast on SUSE
by andreas1234567 (Vicar) on Jul 24, 2007 at 19:23 UTC
|
IBM says here:
Database server performance, typically measured in terms of the transaction throughput and application response time, depends heavily on the I/O subsystem performance. To attain the best possible I/O throughput, database table data layout demands special attention from database and system administrators.
I suggest you examine the file systems in use, the layout and the performance of your disks, and the individual settings related to file system performance. Generally, the ext3 file system is conceived as less performant than, say, XFS. The choice and setup of your file system heavily impacts database performance.
| [reply] |
|
|
All true, but the file system should not play a big role in my tests. (For the first test, the row should be in DB2's cache; for the second test, I have clustered the table on the key I have searched for, such that all 100,000 rows are on the same location on the disk.)
Also note that the speep-up I have observed when upgrading from the Fedora to the SUSE perl packages (all on the Fedora system), was a factor of 10, even if I made no other changes.
So I guess this is not a problem with DB2, but rather with perl accessing it.
| [reply] |
Re: DBI/DB2 slow on Fedora, fast on SUSE
by Hercynium (Hermit) on Jul 25, 2007 at 01:40 UTC
|
If the two Linux distros are running on the same hardware and using the same filesystem, here are the three components I would investigate:
- Kernel config
- libc
- Perl (et al.)
Personally, I think it would take some really messed up configurations for any one of these to cause an order-of-magnitude performance difference out-of-the-box (meaning un-modified distro packages) but sufficiently un-optimized builds of all three together probably could.
The kernel is still a bit of black magick for many folks. I apologize if I'm telling you stuff you may already know, but my experience is that simple point-releases in the Linux kernel can have dramatically different performance characteristics. If the two distros are running different versions of the kernel, that can explain some of the difference.
Also, keep in mind the configuration of the kernel - are the IO-related pieces loaded as modules or compiled in? Are there additional modules loaded or compiled in that one kernel has and the other does not? Has the kernel been compiled for a more specific/newer processor? What about optimization settings?
Of course, the kernel alone does not seem to me that it could cause such a disparity in performance.
Next to check libc - grab the source RPMs for each and compare the .spec files to figure out if there's any significant differences in compiled-in options and optimization parameters. Since nearly all the software on a system uses it at run-time a poorly-optimized libc can hurt performance in myriad ways
Perl can get a little trickier - the compilation options are significant, but the bundled versions (and compilation options) of various modules and such can make a *big* difference.
For example (and this is what brought me to the Monks!) - on an Ubuntu 7.04 system I was happily using the DBM::Deep module from the distro repository. When I used CPAN to update this module, I ended up with an over 100x slowdown. (It may have been more - but after 24 hours I wasn't going to wait any longer for my script to finish)
After some basic profiling and some testing on another system, I discovered that perl was spending all it's time computing MD5 hashes - the MD5 module had been upgraded and was significantly slower. Not only that, but the newer version of DBM::Deep was slower as well. The combination was multiplicative. Being against a deadline, I didn't dig any deeper.
I won't pretend that I'm a tuning guru, and every environment is unique, but I've been very successful in squeezing the most out of some pretty loaded systems where I work and these are the things I tend to look at first when in a similar situation as yours.
However, all this is only necessary if you're somehow *stuck* with the distro that performs poorly. The best solution is likely the simplest - use the fast one! Of course PM is a place for curious people like ourselves and understanding is so often worth the effort... So best of luck to you! | [reply] |
|
|
Hercynium, thank you for your checklist for hunting down performance issues -- it is helpful for me and will be for other people!
Just for the record, here are the versions used on both systems:
SUSE: kernel-default-2.6.18.8-0.3.i586, glibc-2.5-25.i686
Fedora: kernel-2.6.20-1.2952.fc6.i586, glibc-2.5-18.fc6.i686
Since the Fedora server is at a remote location, I decided not to install a different distribution on it; I will just use the perl packages from SUSE that perform better.
| [reply]
[d/l]
[select] |
|
|
You "will just use the suse packages on the fedora installation"???? Muahahahahhah - No wonder your sh1+t don't work, man!!! :-) You gotta get down and dirty promiscuously up close and personal with compiling from source.
Across our workstations, but most importantly- on our development and application servers- we do minimal distro deployment (just the main stuff for the os, X11 gui, etc). And the rest from source. Screw the packages- they always do something stupid. Yum is great for little things like web browsers, image applications, whatever. Little things. But when you are talking performance - the safest thing is to go get the actual architecture independent source. And compile via the cli. It might be a litte bit of a pain sometimes, but the stuff works. Nothing like having that binary sit on your hardware like butter on bread. And your distro is practically irrelevant.
You gotta get down and dirty promiscuously up close and personal with compiling from source. I've had stuff that felt slow all of a sudden double in speed because I've had the wrong stuff.
Also, check that your yum conf actually is getting the right architecture for your system. If you run the nightly yum, it could cause trouble.
| [reply] |
|
|
Re: DBI/DB2 slow on Fedora, fast on SUSE
by jasonk (Parson) on Jul 24, 2007 at 18:03 UTC
|
The SUSE packages are built for i586, so they use processor features and optimizations not available on i386 machines. SUSE has chosen to compile most packages as i586 or higher, which gives you more speed if you have fast hardware, but means you can't run it on older hardware. Fedora has chosen to compile most packages as i386, and provide i586/i686 versions of some key packages where the speed difference will be noticable (like glibc), this way it can still run on older hardware.
| We're not surrounded, we're in a target-rich environment! |
|---|
| [reply] |
|
|
Indeed, installing an i586 version vs. an i386 version should speed things up. That it can make a factor of 10 difference, however, I find very surprising. So I strongly suspect that there is some other explanation/problem somewhere...
| [reply] |
Re: DBI/DB2 slow on Fedora, fast on SUSE
by jfroebe (Parson) on Jul 24, 2007 at 17:26 UTC
|
Sounds more of a Linux distribution issue rather than a perl one. You may want to file a performance bug with Novell (SuSE) indicating this problem :) Good job!
Thanks andreas1234567 for pointing out that I transposed SuSE and Fedora!
| [reply] |
|
|
According to the headline, SuSE is the fastest of the two configurations, so I guess you mean Fedora.
| [reply] |
|
|
Yes, Fedora it is.
At this time, I do not understand why such a huge speed difference appears... So I do not yet have a good bug description; maybe the experience of other people here can help illuminating this subject.
| [reply] |
|
|
