After some trial and error, as well as trying every alternative I could find, I've written a new SNMP module that allows for fast asynchronous polling of multiple objects across thousands of nodes.
My tests across 1000 nodes at my employer's network have so far been *very* successful! The usage example completes one round of polling of these nodes in under 30 seconds, with about 25% CPU utilization. If I increase the InFlight parameter to 50, it completes in under 20 seconds with 37% CPU!
At this time, the module is the bare-minimum necessary for my needs, but I believe it has the potential for much, much more. I won't go into the details here of why I've written my own but suffice to say that nothing had the right combination of features, stability, and speed that I needed.
The code makes heavy use of closures, and I'm fairly new to the concept. It depends on the net-snmp libraries, and the SNMP module that comes with them.
IMPORTANT -- in order for this to work, you must be using the SNMP module from net-snmp release version 5.3.2 or the latest version from CVS trunk. The release version 5.4.1 (as well as the versions from CPAN) has a bug in the XS code that leads to a core-dump when running too many asynchronous queries in parallel.
What I want now
I'm posting this to the monastery in hopes that the knowledge of the Monks can help me make further improvemens, and find and fix bugs and any sub-optimal design decisions. This includes:
UPDATE: I've posted a new version of the code here. The code in this node should now be considered obsolete. The new code is slightly faster, more robust, and *much* more scalable. It also has new features.
The old code is still here, but is now hidden with spoiler tags.
My tests across 1000 nodes at my employer's network have so far been *very* successful! The usage example completes one round of polling of these nodes in under 30 seconds, with about 25% CPU utilization. If I increase the InFlight parameter to 50, it completes in under 20 seconds with 37% CPU!
At this time, the module is the bare-minimum necessary for my needs, but I believe it has the potential for much, much more. I won't go into the details here of why I've written my own but suffice to say that nothing had the right combination of features, stability, and speed that I needed.
The code makes heavy use of closures, and I'm fairly new to the concept. It depends on the net-snmp libraries, and the SNMP module that comes with them.
IMPORTANT -- in order for this to work, you must be using the SNMP module from net-snmp release version 5.3.2 or the latest version from CVS trunk. The release version 5.4.1 (as well as the versions from CPAN) has a bug in the XS code that leads to a core-dump when running too many asynchronous queries in parallel.
What I want now
I'm posting this to the monastery in hopes that the knowledge of the Monks can help me make further improvemens, and find and fix bugs and any sub-optimal design decisions. This includes:
- Ideas for module name/name-space
- Speed improvements
- Feature additions
- Bug fixes
- Anything else you'd like to suggest
UPDATE: I've posted a new version of the code here. The code in this node should now be considered obsolete. The new code is slightly faster, more robust, and *much* more scalable. It also has new features.
The old code is still here, but is now hidden with spoiler tags.
<Reveal this spoiler or all in this thread>
I welcome any and all comments and criticism!
Back to
Meditations