I have an application that listens for SNMP traps using Net::SNMPTrapd. It examines the trap, discards most, and does some work (taking about 5ms) with a few of them. The rate of traps coming in is considerable (>100/s). The current version can handle that rate, but I'd like to make it more scalable. To that end, I've rewritten it with a master process controlling multiple worker threads.

The master process creates a lockfile that the worker threads then use to control access to port 162. The worker threads sit in a loop blocking on flock(). When they succeed in getting the lock, they get a trap from port 162 using Net::SNMPTrapd and give up the lock. Here are the basics of what each worker thread does:

my $snmptrapd = Net::SNMPTrapd->new(ReusePort => 1);
while (1) {
    flock($lockfile,LOCK_EX);
    my $trap = $snmptrapd->get_trap();
    flock($lockfile,LOCK_UN);
    # filter traps based on IP, etc. here
    $trap->process_trap();
    # do some work here, usually taking 3-5ms
}

[download]

All this works. However, I find that regardless of the number of worker threads I create, one of them does most - about 70% - of the work. That is OK for now - one thread can handle it at the current volume of traps - but that seems to defeat my object of scalability.

I have tried several methods to encourage the hardest-working thread to sit back and allow the others a turn. The one that I thought made sense was having each thread keep track of how many traps it's received and do a sub-second sleep when its trap count crosses a threshold. That just resulted in dropped traps.

Is there some technique I'm missing here? Running on Perl 5.16 on CentOS 7. Thank you!

In reply to worker threads - one does all the work by jvuman

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.