I don't see anything dramatically wrong with your threading code. There are some anomalies, like why you take a reference $session to the session object $session0, and then indirect through that instead of using the latter directly--but I'll put that down to artifacts of trimming your real code down for test purposes.

I'm also personally uncomfortable with code that doesn't use strict, and relies on package globals for communications between subs, but it's your code. There are a few errors that would show up were you using strict and warnings. For example, you have

my $result6;

$result6 = $$session->get_entries(-columns => [
        $ipCidrRouteIfIndex, $ipCidrRouteProto, $ipCidrRouteType 
]);

$result6 = $$session->get_entries(-columns => [
    $ipRouteNextHop, $ipRouteIfIndex, $ipRouteMask, $ipRouteProto, $ip
+RouteType
]) unless %$result6;
[download]

If the first call fails and returns undef, then %$result6; would produce a warning Use of uninitialized value in hash dereference ... if they were enabled. Better to use just unless $result6 or as you have on the next line unless defined $result6;, but it probably doesn't affect the result.

More important is what goes on under the covers of Net::SNMP--though take anything I say with a handful of salt as I have no devices to run it against and it's a huge amount of complicated code to run in your head.

The main comment I have is that it is never intended for use with threads. A large part of what makes it so complex, is that it has it's own built in select-based, event driven, multiprocessing model.

Ostensibly, you only need to add -nonblocking   => 1, to the constructor, and -callback => \&somesub, to the get_entries() calls, and you can overlap all your requests without using threads.

Of course, with your program being 100,000 lines, it indicates that you are doing some pretty complex and probably time consuming processing of the data you are getting back. If you just stuck all that into the callback, the timeliness of the event-driven model will go right out the window and the performance will suck. And trying to break your code up into small chunks to maintain the responsiveness of the event loop, without your being able to register your own code with their internal event queues, means that you would effectively have to build a second event-loop model on top, within your code.

The problem of the memory growth arises because they are using several 'singleton' instances internally.

• There's one, $DISPATCHER in SNMP.pm.
• Another, $MESSAGE_PROCESSING; inside Dispatcher.pm.
• There's $INSTANCE and a global hash $MSG_HANDLES to hold all it's messages, in MessageProcessing.pm.

And there are no less that 12 packages in the suite, and many of these load others--often large--packages, Math::BigInt, Digest::MD5 etc. And all of these, and all of the global data represented by the singleton objects and global tables above are being loaded and duplicated into every one of your 100 threads.

In essence, you are building a program that contains 100 event-loops, and then running each as a 1-shot event.

I also suspect, but cannot confirm from a dry run analysis, that the memory growth occurs because global data-structures get built and then replicated across 100 threads, but that the clean-up of the memory associated with any single request will only happen in the thread that processes it. Ie. Each request causes 100 copies of the data structures required to service it, but only one of those copies gets cleaned up upon completion, leaving 99 dead copies of everything from your 1000 requests kicking around doing nothing but taking up space.

It would certainly explain the growth.

The upshot.

You either need to forget using threads, use the built-in non-blocking multi-processing model, with all the inherent difficulties of trying to break your extensive post-processing into bite-size chunks to maintain the responsiveness of the internal event loop. Which probably means building a second event-loop mechanism in your own code.

Or, entreat on the authors to move away from using singletons, global data structures and the (redundant) event loop comms model, when running in blocking mode.

Neither is a particularly enticing decision.

Update: There is another alternative. Run the Net::SNMP event loop in non-blocking mode in the base thread. Have the callbacks simply queue the returned data and return. Start a number of threads reading from the queue of response data to do the time consuming processing. It would be important to ensure that your processing threads were started before you loaded Net:SNMP and all it's dependancies in the main thread, to avoid all the code being replicated into the worker threads. Ie. require not use.

This represents a fairly major restructuring of your existing code, but far less than either of the alternatives above.