in reply to Compile perl for performance

what options should I consider when trying to optimize against my specific code based?

My inclination would be none of the things that you listed. Your benchmarks sound about right for an optimizing compiler. However, often in a large application a 10x+ performance increase (or even much more) can be had by adjusting either the algorithm or the implementation coding of your existing algorithm.

Another factor to consider is that the higher you crank the compiler optimization level, the more chance there is of the compiler making a mistake. I've lost contact with a friend at the moment, but he was analyzing optimize levels in all the major C compilers as part of a PhD level paper. I learned that all of these things make mistakes if the code is complex enough - I'm sure that Perl itself is complex enough for that qualification. Since you are optimizing the compiled C code, some kind of flaw in what Perl does could be hard to track back - could be that some nasty thing happens that is very hard to figure out.

I personally would look for algorithmic and coding enhancements far before messing with compiling Perl itself at a higher optimization level.

Replies are listed 'Best First'.
Re^2: Compile perl for performance
by learnedbyerror (Monk) on Aug 15, 2018 at 00:19 UTC

    Marshall,

    Thanks for your post. I agree with everything that you have written. I neglected to say that the code that I am testing has been around for a while. My first version of it took 32 hours to run. Through refining algorithms I was able to reduce it down to about 12 hours. I then ran it through Devel::NYTProf and based on a half a dozen or so iterations and algorithmic changes reduced the run time to about 8 hours.

    At this point, I'm being greedy and seeing what else I can get. The vast majority of the time is spent in the LMDB driver writing to the database in this case. This accounts for about 80% of the run time. The next chunk is about 10% for Sereal to serialize the HashRef which is written to the db. The next chunk after that is about 5% to parse and analyze the input data into the HashRef in the previous chunk. The last 5% covers the reading of the input files and other miscellaneous.

    My belief, based upon observing perl magic at a distance, is that between 5.28, usemyalloc and O3 that there is a net improvement on I/O, XS Integration and complier optimization that gets me to down to 6 hours.

    If I just applied the last perl version and compiler optimization, I would only be down to 24 hours from 32. That vast majority of getting from 32 to 6 hours, all but 2 hours of the reduction, is due to algorithmic improvement.

    I am somewhat concerned about the possibility of instability that you mentioned. In my experience, I have found O2 a reliable optimization level for gcc in general. I have run into problems with O3 where it helped on some code and actually made it worse on other code. One of the things that I love about App::perlbrew is that I can easily have multiple versions of perl installed. The version that I use every day is compiled with no additional flags. I do usually have one version available compiled with O2 for those programs where through testing I know that I receive a needed boost.

    Thanks again for your advice!

    lbe

[reply]
      Hi Ibe (learnedbyerror)!

      Thanks for your informative response!
      I think that we are "on the same page" and you know what you are doing!

      You are quite correct to be suspicious of gcc -O3 level. I TA an Advanced Assembly class whenever it "makes", which is only about every 5 years. It is a difficult class and it takes years to get enough qualified students in order to justify running the class.

      Sometimes we play "beat the compiler". This is possible at even at the highest optimization level. I agree that -O2 is "fairly safe". At -O3 the compiler gets increasingly bizarre in what it does - it writes ASM that no human would ever think of. It may even write code that winds up slower! The Phd guy I alluded to in an earlier post was one of our students.

      Here is one suggestion which may or may not help you:

      Your application is very DB intensive.
      The DB will have two important general limits:

      1. the number of operations per second
      2. a much smaller number, the number of transactions per second
      As it turns out, the commit of 1 million inserts doesn't take much longer than the commit of just one single insert.

      See if you can reduce the number of DB transactions per second. This can have a huge impact upon performance! You may or may not be done with the first part of optimization (algorithm and coding enhancements).

      I suspect that there is still more than a 25% improvement that can be had without resorting to an optimized compile of Perl itself.

[reply]

        Marshall

        Thanks for your informative follow up. I picked up on the large transaction count optimization jointly from reading up on LMDB as well as using it previously with BerkeleyDB. I borrowed KENTNL's single_txn trick that he used in CHI::Driver::LMDB module. This is what took a lot of the time off of the original 32 hours. Most of the remainder improvement was due to setting LMDB ReadMode true and setting the environment flags to MDB_WRITEMAP | MDB_NOMETASYNC | MDB_MAPASYNC | MDB_NORDAHEAD

        I am currently committing after 100 000 inserts. I did some testing up to 1,000,000 and found insignificant improvements after 100,000. So I decided to keep 100,000 as my limit

        Thanks again for your sage advice!

        lbe

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom