Remote Profiling the JVM through an SSH Tunnel

This one took a little time to figure out. I wanted to profile a remote JVM application using VisualVM locally, in a way that didn’t expose any ports to the outside world and has a reasonable level of security. An SSH tunnel to access it would be fine, but I had to figure out whether the tools supported it.

VisualVM is a tool I’ve used at various points in time to find some of the more obvious performance issues in my software. While not as fully featured as other tools out there, it can still be useful when looking for some of the more egregious bottlenecks. Using VisualVM to profile a local JVM application is straightforward enough – start up the application, and it appears in the Local list under the Applications tab. Profiling a remote application is where it gets a little more tricky.

Oracle’s JVM has a range of instrumentation available for remote monitoring, and one option is through the JMX agent. Since Java 6, any application run on the JVM should have the Attach API, which means that to open remote monitoring on a given port you just need to set the appropriate system property on startup, ie:

By default, the protocol connecting a JMX client to a JMX server is Java RMI. Now it used to be that the registry ran on the specified port and a separate, roaming data port was created. You can probably find several rants discussions on older blogs around about various ways to find or pin this down, but since Java 7u4 you can specify the RMI port, even binding it to the same port as the registry, with a system property:

The JMX/RMI server now listens on the one port, though it’s bound to 0.0.0.0 by default and I found no way to specify a property to force it to localhost. While this *should* be fine in a local network or a firewalled server, I do prefer to bind to localhost when I can. To do this you can specify your own agent that sets up the server programmatically, and for this I used https://github.com/gehel/jmx-rmi-agent. This agent runs on startup and creates the JMX service with an explicit URL, and when you set the option to force it to bind to localhost it uses “service:jmx:rmi://localhost:3000/jndi/rmi://localhost:3000/jmxrmi”. You use the -javaagent option to set it as the agent, and it has its own set of system variables so as not to clash with the defaults, but they are fairly straightforward:

Unfortunately it was still externally visible even with the explicit localhost URL, but I did some more reading and found that by creating a custom RMIServerSocketFactory and passing that to the LocateRegistry.createRegistry call, you can bind the socket yourself. Because that’s what we really enjoy having to do. At any rate, my changes are available on the custom-socketfactory branch of my fork, at https://github.com/taufiqkh/jmx-rmi-agent/tree/custom-socketfactory.

All this done, it was binding to 127.0.0.1 like a champ:

Now, some authentication would be nice. It’s bound to localhost and only authorised users should have access to the system, but I do like layers in my security. Fortunately this part was pretty simple. I was happy to use the default monitorRole security role and to specify a password all I needed to do is create a jmx password file with the following contents:

The password file is specified with the system property:

The final command line was the combination of all these options:

System up and running, I now needed to configure the tunnel – this part was fairly straightforward. I was using putty, which has various options for tunnelling. All I needed to do was to set the port it would listen to on my desktop and the port to which it would forward on the remote system, and press Add:

Putty Tunnel

Going back to VisualVM, I now needed to set up the connection. I selected File->Add JMX Connection… and added the URL output by the jmx-rmi-agent, service:jmx:rmi://localhost:3000/jndi/rmi://localhost:3000/jmxrmi. Once I filled in the username and password and selected Do not require SSL for this connection (that’s a bit of configuration I didn’t want to get into at this stage) I pressed OK:

visualvm_add_jmx_connection

This done, I selected the remote system in the Applications tab, and up came the overview. I moved to the Monitor tab for a look at the current activity:

visualvm_remote_connection

Yak shaving done, it’s time for the real work…

This entry was posted in Programming and tagged , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *