Monitoring the JVM with SNMP

Posted by Roger Keays, 21 January 2007, 4:57 PM

Since Java 1.5, Sun's JVM has included an SNMP agent which is quite handy for keeping an eye on your Java apps using your existing monitoring toolset. Here's how to set up OpenNMS to monitor an app server and produce pretty graphs such as the one below, alongside your other SNMP collected metrics like CPU load and memory usage.

1. Install OpenNMS

First you need to install OpenNMS [1] onto the machine which is going to do the monitoring. OpenNMS is not difficult to install if you've had any experience with maven and Tomcat. Their website covers installation fairly accurately, so I'll skip this step and go straight into how to setup the JVM.

2. Enable Java SNMP

Enabling the SNMP agent on the JVM is pretty simple [2], and can be done adding either -Dcom.sun.management.snmp.port=161 or -Dcom.sun.management.config.file=snmp.properties to the java command line. Using the first method gives you an SNMP agent with all the defaults, which includes only listening on the loopback interface. Using the second method allows you to specify a few more properties in the snmp.properties file (or any other file of your choice).

For this task, I wanted SNMP on port 1161 since the app server runs as an underprivileged user and I already had another agent on port 161. I also needed to listen on all interfaces because the machine is being monitored remotely. Finally, since my firewall keeps out unwanted guests, I've also disabled the access-control list. The complete snmp.properties file looks like this:

com.sun.management.snmp.interface=0.0.0.0
com.sun.management.snmp.port=1161
com.sun.management.snmp.acl=false

Test your configuration by trying to query the Java SNMP agent first from the localhost, then from the monitoring machine:

$ snmpwalk -c public -v2c java.example.com:1161 .1.3.6.1.4.1.42.2.145.3.163.1.1.4.1
SNMPv2-SMI::enterprises.42.2.145.3.163.1.1.4.1.0 = STRING: "pid@host"

3. Configure OpenNMS

Next, we need to configure OpenNMS, which is a little more involved. Conceptually, there are four steps:

1. Configure the capabilities daemon to look for the snmp-jvm service on your chosen port (capsd-configuration.xml).
2. Configure the collect daemon to fetch data for this service (collectd-configuration.xml).
3. Configure the OIDs for the data you want to collect (datacollection-config.xml).
4. Configure the snmp graphs to display the collected values (snmp-graph.properties).

capsd configuration

This tells OpenNMS to look for the SNMP agent you have set up. Add the following configuration to capsd-configuration.xml, restart OpenNMS and rescan the host's services using the web interface. You should see a new service 'SNMP-JVM' on the host.

<protocol-plugin protocol="SNMP-JVM" class-name="org.opennms.netmgt.capsd.plugins.SnmpPlugin" 
                 scan="on" user-defined="false">
    <property key="timeout" value="5000" />
    <property key="retry" value="3" />
    <property key="port" value="1161" />
    <property key="vbname" value=".1.3.6.1.4.1.42.2.145.3.163.1.1.4.1" />
</protocol-plugin>

collectd configuration

To tell OpenNMS to actually collect the data from this service, you have to add the following to the collectd-configuration.xml file.

<service name="SNMP-JVM" interval="300000" user-defined="false" status="on">
    <parameter key="collection" value="jvm"/>
    <parameter key="port" value="1161"/>
    <parameter key="retry" value="3"/>
    <parameter key="timeout" value="3000"/>
    <parameter key="oid" value=".1.3.6.1.4.1.42.2.145.3.163.1.1.4.1"/>
</service>

OID configuration

You also need to decide what data you want to monitor by reading the Java MIB [3]. Here are the values I chose to monitor:

Variable	OID
jvmMemoryHeapUsed	1.3.6.1.4.1.42.2.145.3.163.1.1.2.11
jvmMemoryHeapCommitted	1.3.6.1.4.1.42.2.145.3.163.1.1.2.12
jvmMemoryHeapMaxSize	1.3.6.1.4.1.42.2.145.3.163.1.1.2.13
jvmMemoryNonHeapUsed	1.3.6.1.4.1.42.2.145.3.163.1.1.2.21
jvmMemoryNonHeapCommited	1.3.6.1.4.1.42.2.145.3.163.1.1.2.22
jvmMemoryNonHeapMaxSize	1.3.6.1.4.1.42.2.145.3.163.1.1.2.23
jvmThreadCount	1.3.6.1.4.1.42.2.145.3.163.1.1.3.1

This information is added to datacollection-config.xml with the snippet below. An entire <snmp-collection/> needs to be created, because the jvm SNMP agent doesn't include a system table that might be used to distinguish it as a unique system in the default data collection.

<snmp-collection name="jvm" maxVarsPerPdu="10" snmpStorageFlag="primary">
  <rrd step="300">
    <rra>RRA:AVERAGE:0.5:1:8928</rra>
    <rra>RRA:AVERAGE:0.5:12:8784</rra>
    <rra>RRA:MIN:0.5:12:8784</rra>
    <rra>RRA:MAX:0.5:12:8784</rra>
  </rrd>

  <groups>
    <group name="jvm" ifType="all">
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.11" instance="0" alias="jvmHeapUsed" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.12" instance="0" alias="jvmHeapCommitted" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.13" instance="0" alias="jvmHeapMax" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.21" instance="0" alias="jvmNonHeapUsed" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.22" instance="0" alias="jvmNonHeapCommitted" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.2.23" instance="0" alias="jvmNonHeapMax" type="Gauge64" />
      <mibObj oid=".1.3.6.1.4.1.42.2.145.3.163.1.1.3.1" instance="0" alias="jvmThreadCount" type="Gauge64" />
    </group>
  </groups>

  <systems>
    <systemDef name="JVM">
      <sysoidMask></sysoidMask>
      <collect>
        <includeGroup>jvm</includeGroup>
      </collect>
    </systemDef>
  </systems>
</snmp-collection>

Now restart OpenNMS and look for the new RRD data files in share/rrd/**/jvm* to make sure the collection is working. Also check the collectd.log file for error messages.

Creating graphs

Phew! Almost there. If your RRD files are being created all you have to do is edit the snmp-graph.properties config file and reload the graphs page. Here's the configuration I used to create the graphs in this blog:

report.jvm.heap.name=JVM Heap Memory
report.jvm.heap.columns=jvmHeapUsed, jvmHeapCommitted, jvmHeapMax
report.jvm.heap.type=nodeSnmp
report.jvm.heap.command=--title="JVM Heap Memory" \
 DEF:used={rrd1}:jvmHeapUsed:AVERAGE \
 DEF:comm={rrd2}:jvmHeapCommitted:AVERAGE \
 DEF:max={rrd3}:jvmHeapMax:AVERAGE \
 AREA:used#0000ff:"Used     " \
 GPRINT:used:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:used:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:used:MAX:"Max  \\: %5.2lf %s\\n" \
 LINE2:comm#00ff00:"Committed" \
 GPRINT:comm:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:comm:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:comm:MAX:"Max  \\: %5.2lf %s\\n" \
 LINE2:max#ff0000:"Max           " \
 GPRINT:max:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:max:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:max:MAX:"Max  \\: %5.2lf %s\\n"

report.jvm.nonheap.name=JVM Non-Heap Memory
report.jvm.nonheap.columns=jvmNonHeapUsed, jvmNonHeapCommitted, jvmNonHeapMax
report.jvm.nonheap.type=nodeSnmp
report.jvm.nonheap.command=--title="JVM Non-Heap Memory" \
 DEF:used={rrd1}:jvmNonHeapUsed:AVERAGE \
 DEF:comm={rrd2}:jvmNonHeapCommitted:AVERAGE \
 DEF:max={rrd3}:jvmNonHeapMax:AVERAGE \
 AREA:used#0000ff:"Used     " \
 GPRINT:used:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:used:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:used:MAX:"Max  \\: %5.2lf %s\\n" \
 LINE2:comm#00ff00:"Committed" \
 GPRINT:comm:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:comm:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:comm:MAX:"Max  \\: %5.2lf %s\\n" \
 LINE2:max#ff0000:"Max          " \
 GPRINT:max:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:max:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:max:MAX:"Max  \\: %5.2lf %s\\n"

report.jvm.threads.name=JVM Threads
report.jvm.threads.columns=jvmThreadCount
report.jvm.threads.type=nodeSnmp
report.jvm.threads.command=--title="JVM Thread Count" \
 DEF:threads={rrd1}:jvmThreadCount:AVERAGE \
 LINE2:threads#0000ff:"Threads" \
 GPRINT:threads:AVERAGE:" Avg \\: %8.2lf %s" \
 GPRINT:threads:MIN:"Min  \\: %8.2lf %s" \
 GPRINT:threads:MAX:"Max  \\: %8.2lf %s\\n"

Add these report definitions towards the end of the file, and their names (jvm.heap, jvm.nonheap and jvm.threads) to the list at the top of the file to have them display on the graphs page.

JMX Monitoring

OpenNMS also does JMX monitoring, but this is likely to be the subject of another blog post. Unfortunately, AFAICT, it does not yet have support for CompositeTypes which means we can't (yet) collect heap and non-heap usage via the Memory managed bean.

References

[1] http://www.opennms.org
[2] http://java.sun.com/j2se/1.5.0/docs/guide/management/SNMP.html
[3] http://java.sun.com/j2se/1.5.0/docs/guide/management/JVM-MANAGEMENT-MIB.mib

<< EntityManager per session pattern

Back to Blog

Why I chose Netbeans >>

Comment posted by: Roger Keays on 23 February 2007, 4:22 PM

There were a few questions on opennms-discuss about this blog, which I did try to answer, but it seems that the gmane gateway for this list is broken:

<discuss@lists.opennms.org>: mail for lists.opennms.org loops back to myself

Sorry!

Comment posted by: Vince on 8 March 2007, 10:08 AM

Hi, I tried this but i got no luck. Does this really work?

Comment posted by: Roger Keays on 22 March 2007, 9:10 AM

Hey Vince. This should definitely work (I rely on this data heavily!). If you post your problem to the opennms-discuss list I'll be able to reply by email even if I can't post to the list.

Comment posted by: homerlex on 22 March 2007, 11:21 PM

I'm having some trouble with this as well.  A couple things I had to add to get it closer to working:

In snmp-config I had to add something a definition like:

definition version="v2c" port="1161" - and I added a "specific" tag to my server_ip

And in collectd-configuration I had to add a collector service def:

collector service="SNMP-JVM"       class-name="org.opennms.netmgt.collectd.SnmpCollector"/

 At this point I see the SNMP-JVM service for my node in the web interface but no data is being collected.  snmpwalk runs fine, I'm not seeing any errors - I just don't know why data is not being collected.

Comment posted by: Roger Keays on 23 March 2007, 8:16 AM

Hi homerlex. Do you have the same collectd-configuration used in the blog above? You may be missing the oid parameter. Also check that this oid is available when you use snmpwalk. Make sure all your oids in all config files begin with a dot. Finally, have you checked logs/collectd.log for errors?

Comment posted by: vince on 28 March 2007, 5:36 AM

It works after I added "\\n" to the snmp-graph configuration:

GPRINT:max:MAX:"Max : %5.2lf %s \\n"

...

GPRINT:threads:MAX:"Max  : %8.2lf %s \\n"

Comment posted by: Roger Keays on 28 March 2007, 9:26 AM

Hey Vince. Thanks for letting me know about the problem. Somehow all the escape characters got dropped when I first posted the blog. I've put them back in now.

Comment posted by: maframan on 25 August 2007, 4:04 AM

2007-08-24 17:50:01,397 DEBUG [Capsd Rescan Pool-fiber0] IfCollector: 192.168.47.12 testing plugin SNMP-JVM
2007-08-24 17:50:01,431 DEBUG [Capsd Rescan Pool-fiber0] IfCollector: 192.168.47.12 protocol SNMP-JVM supported? false
2007-08-24 17:50:01,431 DEBUG [Capsd Rescan Pool-fiber0] IfCollector: 192.168.47.12 plugin SNMP-JVM completed!
2007-08-24 17:53:40,487 DEBUG [Main] JdbcCapsdDbSyncer: syncServicesTable: checking protocol 'SNMP-JVM'.

 

What is the problem?

Help me!!!!!

Comment posted by: Francesco on 28 August 2007, 2:48 AM

I am edited caps.log and obtain follows listen:

testing plugin SNMP-JVM

protocol SNMP-JVM supported? false

plugin SNMP-JVM completed!

The problem is who the services is not discovered!!!!!!!

Regards

Francesco

Comment posted by: Anuj on 18 October 2007, 5:47 AM

After following the instructions (which were extremely helpful, thanks), I am seeing this in collectd log: 007-10-17 13:12:40,559 DEBUG [CollectdScheduler-50 Pool-fiber0] Collectd: scheduleExistingInterfaces: dbConn = com.mchange.v2.c3p0.impl.NewProxyConnection@1c1ac46, svcName = SNMP-JVM 2007-10-17 13:12:40,560 DEBUG [CollectdScheduler-50 Pool-fiber0] CollectdConfigFactory: interfaceInPackage: Interface 10.24.53.191 passed filter and specific/range for package example1?: true 2007-10-17 13:12:40,570 DEBUG [CollectdScheduler-50 Pool-fiber0] SnmpCollector: initialize: SNMP storage flag: 'all' 2007-10-17 13:12:40,570 DEBUG [CollectdScheduler-50 Pool-fiber0] SnmpCollector: initialize: maxVarsPerPdu=50 2007-10-17 13:12:40,572 DEBUG [CollectdScheduler-50 Pool-fiber0] SnmpCollector: initialize: db retrieval info: nodeid = 1, address = 10.24.53.191, primaryIfIndex = 3, isSnmpPrimary = N 2007-10-17 13:12:40,573 WARN [CollectdScheduler-50 Pool-fiber0] Collectd: scheduleInterface: Unable to schedule 10.24.53.191 for service SNMP-JVM, reason: Interface 10.24.53.191 is not the primary SNMP interface for nodeid 1 2007-10-17 13:13:34,627 DEBUG [CollectdScheduler-50 Pool] Scheduler: run: scheduler exiting, state = STOPPED What does "Interface 10.24.53.191 is not the primary SNMP interface for nodeid 1" mean and how can it be resolved.

Comment posted by: Anuj on 20 October 2007, 4:43 AM

Hello All: I finally figured out the problem with the SNMP trap data collection (mentioned above). Looks like the OpenNMS doesnot recognize a custom community string. hence use public in your agent as the community string and use the same (public) in your snmp-copnfig.xml and everything will be fine (assuming the snmp walk is successful). Thanks

Comment posted by: Seb on 4 February 2008, 8:10 PM

When I run opennms.
I have this in capsd.log :
    2008-01-31 16:25:27,503 ERROR [main] Capsd: Failed to load poller
configuration
    ValidationException: rrd is a required field.;
     - location of error: XPATH: poller-configuration/package
       at org.exolab.castor.xml.FieldVal

Comment posted by: AG on 22 April 2008, 12:53 PM

Has anyone gotten this to work?  If so, could you please post the details?  I am not getting any data collected with the exact settings above. only indicator in the capsd.log is "protocol SNMP-JVM supported? false"

Comment posted by: Roger Keays on 14 June 2008, 2:49 AM

Well, FWIW I can't seem to get anything to work in OpenNMS these days. I switched to MRTG for SNMP visualisations and it took about 5 minutes to set up. Sooo much simpler!

Comment posted by: Oka on 4 July 2008, 4:35 AM

While this information is interesting, I have found its not easy to implement nor does it work for everyone, if you want to just graph your JVM I suggest the following opennms url  www.opennms.org/index.php/JVM_Monitoring_using_SNMP it was the only way I could get it to reliably graph the jvm.... however the GC protion of the code does need some work.

Comment posted by: syed on 13 February 2009, 11:33 PM

I am not aware of this SNMP trap, agent and opneNMS.

Please give me knowledge about this SNMP trap to monitor host's event.

Env:

Windows 2x

OpenNMS and postgresql

Comment posted by: Roger Keays on 10 April 2009, 3:24 AM

I've posted an equivalent configuration using MRTG instead of OpenNMS at

http://www.ilikespam.com/blog/monitoring-the-jvm-with-snmp-mrtg

 

Blog Feed

Website Updates

• Seoul Short Term Accommodation by Roger Keays, yesterday
• RE: width: auto; on Floats i by Roger Keays, 2 weeks ago
• RE: width: auto; on Floats i by Jason, 2 weeks ago
• RE: width: auto; on Floats i by Computer hardware, 3 weeks ago
• RE: width: auto; on Floats i by TheOnlyOne!, 1 month ago
• RE: c:forEach vs ui:repeat i by travesti, 1 month ago
• Smartphone Comparisons and Mini Guide by Roger Keays, 1 month ago
• RE: Packaging Your Facelets by Roger Keays, 1 month ago
• RE: Centering Multiple Line by Gabriel, 1 month ago
• RE: Cheap Second Hand Bikes by haryoto, 1 month ago
• How to Avoid ATM Fees in Bangkok, Thailand by Roger Keays, 2 months ago
• RE: Onward Travel Extortion by Roger Keays, 2 months ago
• Exchange Vietnamese Dong To Thai Baht in Bangkok, Thailand by Roger Keays, 2 months ago
• RE: Javascript Fade Effects by Adam, 2 months ago
• RE: Intel Core 2 Duo vs Inte by sriram, 2 months ago

---

Copyright © 2003 - 2008 Roger Keays.

brisbane web design