This is a followup on some recent posts of mine .. in case anyone new to Perl wonders if it is ever used for real work.

My web application handles document processing (conversion from PDF to XML), and it's currently running on a dozen servers, the busiest of which is used by two teams, one here in Toronto and the other in Mumbai, India. The work day starts about midnight local time (930am in Mumbai) and goes to about 8 or 9pm, and lately we've been running 7 days a week to keep up with demand. If things go South, it lands in my lap. I hate pages at 4am, so I do what I can to avoid that situation.

Anyway, recently, I wrote about a problem I'd been having with a sleepy CGI, and eventually rediscovered the excellent tool strace thanks to some respondents. But that led me to my next problem, as to why the CGI would be freezing at 

select(5, [4], [], [4], NULL

[download]
as strace was showing me. After some more research, I discovered this was the connection to the database. Could it be as simple as the CGI waiting for the database that was causing the problem?

Oh boy. The answer is yes, as it turns out. And the CGI was not 'going to sleep' -- it was continuing to run, but was patiently waiting for the database, not sleeping.

And the solution to the database slowdown (PostgreSQL, in my case) was the simple application of 

ANALYZE VERBOSE DOCUMENTS;

[download]
and the response time for the main query went from 20 seconds to about 550ms. Yay!

Since I've seen this performance problem before, I'm now going to keep an eagle eye on the system today and find out how long it takes before the performance starts to drop, then set up a cron job to ANALYZE the suspect table again, most likely every four hours or so.

This is all a result of watching my system closely, something I think is extremely valuable in any Engineering job -- staying on top of the performance of your system is very important in my current situation of developer, supporter and maintainer.

The moral of the story is, follow the data (I know, it sounds like CSI). Processes very rarely 'go to sleep' unless they're told to. Where's the CPU time (or bandwidth, or your other resources) going? Follow that lead, and you'll find the answer to your problem.

Alex / talexb / Toronto

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

In reply to Make sure you're solving the right problem by talexb

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.