Much slower execution from one machine to another

df4or has asked for the wisdom of the Perl Monks concerning the following question:

I understand that our problem might not(only|at all) be related to Perl, but since this is the language of choice and I know there is a lot of wisdom to be found here, I beg to ask...

When migrating several older Perl programs from an old to a new server we saw a dramatic decrease in execution speed, although the newer machine should be much faster.

Old machine:
Intel Celeron @2.2GHz, 4533 bogomips, 2GB RAM, Perl 5.10.1, Ubuntu 10.04 LTS, kernel 2.6.32-45, mysql 14.14 Distrib 5.1.66, SysV init

New machine:
Intel Core i3-4160 @3.6GHz, 7200 bogomips, 8GB RAM, Perl 5.20.2, Debian jessie, kernel 3.16.0-4, mysql 15.1 Distrib 10.0.23-MariaDB, systemd 215

The difference in one task was about 10-fold (i.e. the script took more than 10 times as long to run on the new machine, 0m58 vs. 11m). In another case the slow down was about 4 to 8-fold. Measurements were taken with the 'time' command on the command line.

The slowest program reads a dBase database table sequentially, mangles the data a bit and writes the records to a text and to a MariaDB SQL table. When commenting out the only $sth->execute statement the execution time jumps up from 11 minutes to 38 seconds. Some careful performance tuning on the MySQL/MariaDB server brought the execution time down to 2m40 (from 11m), still about 3 times slower than the old machine.

Another program, which does not use SQL, reads a dBase file, mangles the data a bit and writes them to a text file. Here the slowdown is about 4-fold. Some analysis with Devel::NYTProf found that most time is spent in IO::Handle which in turn is called by XBase.pm.

What I did and concluded so far:

Moved the location of the dBase files which I read, from the server to the local machine, slight speed up of maybe 5%. So I conclude that the network is not the issue.
Copied the same large numbers of small and large files from the server to the local machines. The new machine is about 3% faster, so I conclude that disk write IO is not the issue. I understand that this is a very crude and simple performance test, and real performance test takes much more. I at least cleared disk caches in advance of each test.
Added the 'noatime' option to all affected file systems in /etc/fstab. No discernible speed up seen.
Ran the two slowest programs with -d:NYTProf option.
- the program which uses SQL spends 80% in DBI::st::execute, 17% in IO::Handle::CORE::read (which is called by XBase.pm);
- the program which does not use the database spends about 87% of the time in IO::Handle::CORE::read (which is called by XBase.pm).
Changed some parameters on the SQL server, which resulted in visible speedup of the programs using the SQL server. Nice, but not enough. Parameters changed were mostly buffer sizes.
Compared the XBase.pm version, since a lot of time seems to be spent there.
On the old machine it is 0.241, on the new one it is version 1.02. I did not yet look into the source of XBase.pm to see if there are any visible changes. Nor did I install the old version on the new machine. In fact I am a little bit afraid to do that and throw modules around... instead I came here to ask.

My question:
What else could be the reason for the slowdown? Is there anything which makes Perl 5.20 so much slower at IO handling? Is there anything obvious that I am missing, maybe something affecting IO in general? Buffering?

Any help is highly appreciated.

Thanks in advance,
Ekki

Comment on Much slower execution from one machine to another Download Code

Replies are listed 'Best First'.
Re: Much slower execution from one machine to another by graff (Chancellor) on Jun 03, 2016 at 22:56 UTC
I gather from your list of specs that each machine was/is running its own db server. Are you sure that the relevant schemas have been set up the same way? E.g. is it possible that some "create index..." operations might have been done on the old mysql tables, but might not have been done yet (or might be different in some way) on the new MariaDB tables? (Have you done a comparison test of old machine mysql vs. new machine MariaDB using the "native" query utilities -- i.e. not via DBI?) I don't know what to say about the dBase file performance differences; if you've done the same benchmarks on the old and new machines, and the difference in run time is comparable to the difference in percentage spent in IO::Handle::CORE::read, then I would guess that the new system is somehow "not playing nice" with dBase data (or vice-versa) -- something like disk block size (i.e. how much data is physically fetched from the media on a single minimal read) could slow things down under some circumstances if it varies in some silly way from the previous setup.	[reply]
Re: Much slower execution from one machine to another by BrowserUk (Patriarch) on Jun 03, 2016 at 17:52 UTC
Could you post the output from -V on both setups; they might shed some clues. Another possibility is do both setups use the same connection mechanism (unix/network port; pipe; odbc etc.) to the DB? One change to do with IO I'm aware of between those two versions is the size of the internal buffers used which I believe went from 4K to 8K. For disk that's generally a good thing (16K would be better still!), but if the change also affects the relevant transport IO; it could be detrimental. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :) In the absence of evidence, opinion is indistinguishable from prejudice. Not understood.	[reply]
Re: Much slower execution from one machine to another by Laurent_R (Canon) on Jun 03, 2016 at 17:37 UTC
It seems that the problem might be due to the database. Look at the default config parameters for MySql and MariaDB.	[reply]
Re: Much slower execution from one machine to another by rdfield (Priest) on Jun 05, 2016 at 13:28 UTC
What about changes to the filesystem used, and the physical layout of the disks? If Perl is seeing increased IO wait, that would suggest to me that the disks are slower. Over the years I've seen this problem attributed to firmware, controller and RAID amongst other things. rdfield	[reply]
Re: Much slower execution from one machine to another by anonymized user 468275 (Curate) on Jun 05, 2016 at 15:00 UTC
Also, it could be that more extensive caching was configured for the database in the old system. I have seen something similar once: I had cached a whole production database and later someone wanted to benchmark a new disk array. In effect the database was one big swap file, but smaller than the total RAM of the system. So as long as the database server wasn't started or shutdown, the shiny new disk array would be unable to outperform even an array of 1970s floppy disks using any SQL benchmark code you like! So it could be that the new system is doing disk I/O for database transactions and the old system was doing less or even NO disk I/O for the database transactions. Also the same thing can be done for /tmp, i.e. in some production systems, /tmp is a virtual disk in memory, not a real one but your perl code coverage tools would never see the difference directly (one has to be careful what to infer). One world, one people	[reply]