(*)Updated units per pc88mxers post below.

By serialising the counts through a threaded udp server, batching them on input before transfering them to the background writing thread, I managed sustained throughput rates of close to 600k/minute on a single core 2.6 M*GHz machine running both producer and consumer, and with half the bandwidth taken up with a download going on simultaneously:

Consumer run:

c:\test>s-udp.pl -MAXBUF=500 Total: 347176 throughput: 9777/sec [max: 9987]

MAXBUF=500 seems to be about the sweet spot on my machine. I can get higher throughputs by upping the priority of the server process. I was just writing the counts out to a file in the background, but you could probably do batch updates to a db without slowing things much.

Consumer:

use strict; use threads; use threads::shared; use Thread::Queue; use Time::HiRes qw[ time usleep ]; use IO::Socket; use List::Util qw[ max ]; $|++; our $PORT ||= 9999; our $FILE ||= 'theLog'; our $LEN ||= 6; our $MAXBUF ||= 1000; my $Q = new Thread::Queue; open my $log, '+> :raw', $FILE or die "$FILE: $!"; my $start = time; my $n :shared = 0; my %log; my $total = 0; async { my $max = 0; my $thru = 0; while( my $in = $Q->dequeue ) { ++$log{ $_ } for split chr(0), $in; sysseek $log, 0, 0; my $logRec = join "\n", map{ $_ . ':' . $log{ $_ } } sort keys + %log; syswrite( $log, $logRec ); $max = max( $max, $thru = int( $n / ( time() - $start ) ) ); printf "\r\t Total: %8d throughput: %5d/sec [max:%5d]", $total, $thru, $max; $total += $n; $n = 0; $start = time(); sleep 1; } }; my $srv = IO::Socket::INET->new( LocalPort => $PORT, Proto => 'udp', ) or die "Socket: $@ : [$^E]"; my $buffer = ''; my $msg; while( $srv->recv( $msg, $LEN ) ) { next unless length $msg; $n++; if( length( $buffer .= chr(0) . $msg ) > $MAXBUF ) { $Q->enqueue( $buffer ); $buffer = ''; } }

Producer:

#! perl -slw use strict; use IO::Socket; our $N ||= 1000; our $PORT ||= 9999; our $DELAY ||= 0.0001; my $sock = IO::Socket::INET->new( Proto => 'udp', PeerPort => $PORT, PeerAddr => 'localhost' ) or die "$@ [$^E]"; my $sent = 0; for ( 1 .. $N ) { ++$sent; $sock->send( int rand( 32767 ) ); select undef,undef, undef, $DELAY; } print "Sent: $sent";

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

In reply to Re: Multiple write locking for BerkeleyDB by BrowserUk
in thread Multiple write locking for BerkeleyDB by dino

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.