Update: I had a chance to run this on a Linux machine. IPC wise (fetching - dequeue) is slower from a Linux OS than from a BSD variant: e.g. FreeBSD, darwin. Running with threads also takes longer. See this post for more info. Below, I've updated the main script to show the enqueue time, pending count, and finally the duration time. A production environment might have a load balancer and 4 pizza-box (1-inch) servers. Together, the 4 servers can handle 1 million traps per minute if that's the scale needed. I reached 4.5k traps per second on my Linux machine.

Update: Loading threads at the top of the script will have MCE spawn threads instead. Doing so may cause consumers to run slower for some reason on Linux. Maybe it's from running an older Perl 5.16.3 release. I'm not sure. For maximum performance check to see if Perl has Sereal installed. MCE defaults to Sereal::Encoder 3.015+ and Sereal::Decoder 3.015+ for serealization when available.

use threads;

[download]

Update: I've updated the code to synchronize STDOUT and STDERR output to the manager process. Furthermore, specified user_output and user_error MCE options so that one can send to a logging routine if need be. Omitting them will have the manager process write directly to STDOUT and STDERR respectively.

Hello again jvuman,

Generating traps is handled by another script. Doing this allowed me to move the trap generator (producer) to another machine. My laptop running the server script can process 6k per second.

use strict;
use warnings;

# perl snmp_server.pl | wc -l

use Time::HiRes qw( sleep time );
use Net::SNMPTrapd;
use MCE::Flow;
use MCE::Queue;

my $queue = MCE::Queue->new( await => 1, fast => 1 );
my $max_consumers = 30;
my $start = 0;

MCE::Flow::init {
    user_output => sub { print {*STDOUT} $_[0]; },
    user_error  => sub { print {*STDERR} $_[0]; },
};

mce_flow { max_workers => [ 1, $max_consumers ] }, \&listener, \&consu
+mer;

printf {*STDERR} "duration : %0.03f seconds\n", time - $start;

exit(0);

sub set_start {
    $start = $_[0];
}

sub listener {
    my $snmptrapd = Net::SNMPTrapd->new( ReusePort => 1 );
    my $count = 0;
    my $start;

    while ( 1 ) {
        my $trap = $snmptrapd->get_trap();

        if ( !defined $trap ) {
            MCE->printf(\*STDERR, "$0: %s\n", Net::SNMPTrapd->error())
+;
            next;
        }
        elsif ( $trap == 0 ) {
            next;
        }

        $start = time(), MCE->do('set_start', $start) unless $start;

        # important, remove the file handle inside the object
        delete $trap->{_UDPSERVER_};

        # enqueue the trap for a consumer to process
        $queue->enqueue($trap);

        # leave the loop after 10,000 traps
        last if ( ++$count >= 10000 );

        # safety to prevent the queue from consuming memory out of con
+trol
        # $queue->await(2000) if ( $count % 4000 == 0 );

        # if ( ( $count % 4000 == 0 ) && $queue->pending() > 3000 ) {
        #     MCE->say(\*STDERR, "Warn: blocking temporarily");
        #     $queue->await(2000);
        # }

        # reset the counter to not overflow
        $count = 0 if ( $count > 2e9 );
    }

    $queue->enqueue((undef) x $max_consumers);

    MCE->printf(\*STDERR, "enqueue  : %0.03f seconds\n", time - $start
+);
    MCE->printf(\*STDERR, "pending  : %d\n", $queue->pending());
}

sub consumer {
    while ( defined ( my $trap = $queue->dequeue() ) ) {
        $trap->process_trap();

        MCE->printf( "[$$] %s\t%i\t%i\t%s\n",
          $trap->remoteaddr, $trap->remoteport,
          $trap->version, $trap->community );

        sleep 0.004;
    }
}

[download]

Here is the producer script for generating traps in parallel. This is useful for load testing.

use strict;
use warnings;

# perl snmp_producer.pl

use Net::SNMP;
use MCE::Flow;
use MCE::Queue;

mce_flow { max_workers => 2 }, \&producer;

exit(0);

sub producer {
    my ( $session, $error ) = Net::SNMP->session(
        -hostname  => '127.0.0.1',
        -version   => 2,
        -community => 'public',
        -port      => 162
    );

    if ( !defined $session ) {
        printf "Error: Starting SNMP session (v2c trap) - %s\n", $erro
+r;
        return;
    }

    for my $i ( 1 .. 5000 ) {
        my $result = $session->snmpv2_trap(
            -varbindlist  => [
                '1.3.6.1.2.1.1.3.0',      0x43, int( time() ),
                '1.3.6.1.6.3.1.1.4.1.0',  0x06, '1.3.6.1.4.1.50000',
                '1.3.6.1.4.1.50000.1.3',  0x02, 1,
                '1.3.6.1.4.1.50000.1.4',  0x04, 'String',
                '1.3.6.1.4.1.50000.1.5',  0x06, '1.2.3.4.5.6.7.8.9',
                '1.3.6.1.4.1.50000.1.6',  0x40, '10.10.10.'.((++$i % 1
+00) + 1),
                '1.3.6.1.4.1.50000.1.7',  0x41, 32323232,
                '1.3.6.1.4.1.50000.1.8',  0x42, 42424242,
                '1.3.6.1.4.1.50000.1.9',  0x43, int( time() ),
                '1.3.6.1.4.1.50000.1.10', 0x44, 'opaque data'
            ]
        );
    }

    $session->close;
}

[download]

To benchmark, run snmp_server.pl on machine A. Then, run snmp_producer.pl on machine B. Remember to change the IP address to host A inside producer if running on another host.

Host A : perl snmp_server.pl | wc -l
Host B : perl snmp_producer.pl

[download]

Regards, Mario.

In reply to Re^2: worker threads - one does all the work by marioroy
in thread worker threads - one does all the work by jvuman

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.