goupilInside has asked for the wisdom of the Perl Monks concerning the following question:

Hello fellow Monks.

I have encountered a tiny annoyance that has me seeking your knowledge.

I have a script, a CGI, using Class::DBI running far too slow that I must "cure".
It is not a very fancy script but it calls a lot of external modules particular to the project.

The platform is Linux and Perl version is 5.8.8.
I'm testing it from the command line to avoid interactions with the web server.

"Yipee" I thought, "I will be using those crazy profiling facilities Perls offer".

And so I proceeded.
After reading some holy writings of St D Foy and the laws of the CPAN, I tried to start using DProf.

Alas, poor me, the program starts but then produces a tense and pretty insulting "segmentation fault" -- then dies. The resulting tmon.out seems truncated and only contains information regarding environment startup.
Retrying with strace shows me a long serial of memory allocations before the segfault (brk).

Being still joyful and naive, I moved to Devel::Profiler, the CPAN having warned me of the possible deprecation of the old DProf ways.
This time, the process started, ate 2G of RAM, then 2G of swap and proceeded to freeze the computer.

"Well", said I, "why not trying Devel::SmallProf".
It worked ... sort of.
It did not crash or freeze, but produced an empty "smallprof.out" .

Devel::FastProf produced a 4K fastprof.out file containing nothing useful (with or without withfork option)

DBI::Profile is working well and indicating that the DB aspects are mostly correct, so I'd really like to be able to profile the sneaks of client side code.

Beside being stuck on the head until Satori, I seem to have tried every reasonable way.

Is there some wise voice in here who would show an humble disciple the way of illumination ?

UPDATE:

It turns out to be quite stupid and fully due to my script.

I have some module called lib/TheProject/DB.pm containing mostly the connection management and some small tools common to all DB oriented classes.

Then all Dabases classes are in modules like lib/TheProject/DB/SomeDBBackedClasses.pm, which requires using TheProject::DB.

Except that one of the developpers (it can't be me, can it ? ...cof, cof), has typed use DB; instead of use TheProject::DB;, including the Debugging related libraries causing a mess of respectable proportions (it still worked because below he/she/it added a use TheProject::DB.

Removing the guilty use makes all the profilers happy.

Thanks to all who answered and gave me directions.

/me is going to check in the SVN whom to apply the clue stick to.

Replies are listed 'Best First'.
Re: Profiling woes
by perrin (Chancellor) on Apr 09, 2008 at 13:02 UTC
    DProf can be a pain, but it is the best one. I'd suggest you try to narrow down where it crashes and try to remove that code temporarily. Since DProf uses the debugger hooks, it probably crashes in the same place when run in the debugger.

    Two pieces of general advice:

    • Always use wall clock time, not CPU time, when profiling a database application. It could be spending the majority of time waiting for responses to queries, which won't show up at all with the default of measuring CPU time.
    • When you find the problem, fixing Class::DBI performance problems often involves replacing Class::DBI calls with straight DBI calls inside a small performance-sensitive section of code.
[reply]

      Thanks I will try that.

      Feedback when DProf cease to crash ... or never if I do before.

      ;-)

[reply]
Re: Profiling woes
by derby (Abbot) on Apr 09, 2008 at 12:04 UTC

    Hmmm ... I would start with logging and timing before I jumped into any of the profile modules. That way I could narrow down the chunk(s) of code that are problematic. BTW ... I would use Benchmark's timediff method: 

    my $t1 = Benchmark->new()
#
# chunk of code
#
my $t2 = Benchmark->new();
my $td = timediff( $t2, $t1 );
$log->debug( "Chunk x took " . timestr( $td ) );

    [download]

    -derby
[reply]
[d/l]
      Theoretically, Devel::DProf should be a good way to find the parts of the program that need attention. It can tell which subs are taking up the time.
[reply]

      That's my "plan B" if I can't make the profiler work.

      But I'd rather not touch too much code.

[reply]
Re: Profiling woes
by stiller (Friar) on Apr 09, 2008 at 11:29 UTC
    I'm not exactly wise, but anyway: what makes your application slow is probably tightly related to the cause of profilers deaths. I'd try to narrow the problem down as much as possible, to isolate the module / set of modules that cause trouble. Try commenting out, e.g. with POD. Is there some heavy lifting in an inner loop somewhere?

    Do you have tests for the project spesific modules? They should probably be prime suspects... If you make simple tests to check their sanity, you can also to profile these tests in isolation.

    Good luck!

[reply]
      try to narrow the problem down as much as possible
      Exactly. If it's a CGI that uses DBI, create two test programs, one that only uses the CGI functionality, the other that only performs the DBI functionality. Find which one runs slow. Break that test program down...
[reply]
        I actually only mentioned CGI because of eventual interaction with the profiling environment.
        But the script is really dependant of getting its data from the database, hard to do without (ok I could do without if I had some stub objects and data, but I don't for now).
        That's why I really thought I had a good approach with the profiling (for once).
[reply]
      what makes your application slow is probably tightly related to the cause of profilers deaths
      Probably not, actually. The deaths are most likely caused by a bug in perl's debugging hooks. There's no reason to think that the part causing the crash is slow.
[reply]

      Well that's what I was trying to avoid thinking that profiling would give me a more elegant approach.

      What makes it more tedious is that some of the modules do have documentation and tests, and tend to have been used in various other programs. Some are more in the philosophy "write once", read never ....

      Thanks anyway

[reply]
Re: Profiling woes
by jplindstrom (Monsignor) on Apr 09, 2008 at 13:31 UTC
    I profiled a CDBI app four years ago and wrote a journal entry which may be of interest.

    Oh, and I can feel your pain regarding non-working profilers.

    /J

[reply]

      Thanks for the pointers.

      I indeed find surprising that those modules would break so easily.

[reply]
Re: Profiling woes
by salva (Canon) on Apr 09, 2008 at 13:10 UTC
    Without seeing any of your code it is difficult to guess what is happening.

    Maybe some of the modules module you are using, uses (or abuses) the debugging API, which may conflict with the profilers.

    Are you able to run the same script under the debugger successfully?

[reply]

      The modules used should not mess with the debugger, but who knows indeed.

      I did not try the debugger, but I will.

      Thanks for the answer.

[reply]
        BTW, I am the author of Devel::FastProf and the maintainer of Devel::SmallProf and would really appreciate having bug reports for those modules describing your problems in detail!
[reply]

Back to Seekers of Perl Wisdom