Production quality Play apps. - part 3 - Using JMX to monitor and tune Play applications

Play Framework applications use the JVM in the same way as any other Java application. In the previous article, I showed a start script for a Netty based play application that enabled JMX in a secure way, so we can monitor the status of the JVM.

Finding JMX connectivity details

Here's an example of a running Play process.

995      18203     1  1 03:49 ?        00:09:17 java -Dhttp.port=9000 -Dhttp.address=0.0.0.0 -Dcom.sun.management.jmxremote.port=7020 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=/etc/play/play_app/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/etc/play/play_app/jmxremote.access -Dcom.sun.management.jmxremote.authenticate=true -Djava.rmi.server.hostname=10.11.12.13 -XX:+UseConcMarkSweepGC -Xloggc:/opt/play_app/logs/gc.log -XX:-OmitStackTraceInFastThrow -XX:+PrintGCDateStamps -verbose:gc -Xms256M -Xmx256m -server -Dconfig.file=/etc/play/play_app/env-application.conf -Dlogger.file=/etc/play/play_app/env-logger.xml -cp /opt/play_app/lib/* play.core.server.NettyServer /opt/play_app/

From this, you can extract the JMX port (jmxremote.port=7020), and find the JMX username/password file (jmxremote.password) - All three bits of information are necessary to connect.

GC Logging

We can also check the GC log if necessary (Xloggc:/opt/play_app/logs/gc.log)  - The information isn't quite as good as using JMX directly, but is useful for a quick diagnosis of a problem.

Essentially, lines like this show normal GC activity.

2013-02-04T17:06:16.627+0000: 47827.125: [GC 383171K->349037K(1044352K), 0.0158340 secs]
2013-02-04T17:12:34.481+0000: 48204.979: [GC 383149K->349028K(1044352K), 0.0193200 secs]
2013-02-04T17:18:56.583+0000: 48587.081: [GC 383140K->349008K(1044352K), 0.0204980 secs]
2013-02-04T17:25:17.088+0000: 48967.586: [GC 383120K->349015K(1044352K), 0.0058240 secs]

Note the last field - GC time in seconds. This should be nice and low - certainly sub-second.

Out-of-memory problems would lead to a log with repeated Full GC lines - similar to:

2013-01-18T10:53:42.184+0000: 4886909.890: [Full GC 2088630K->2055966K(2088640K), 6.4534550 secs]
2013-01-18T10:53:51.453+0000: 4886916.393: [GC 2080393K(2088640K), 0.0550080 secs]
2013-01-18T10:53:51.576+0000: 4886919.258: [Full GC 2088639K->2053535K(2088640K), 6.2507810 secs]
2013-01-18T10:54:00.613+0000: 4886925.553: [GC 2077526K(2088640K), 0.0533610 secs]
2013-01-18T10:54:00.728+0000: 4886928.562: [Full GC 2088633K->2055761K(2088640K), 6.3442360 secs]
2013-01-18T10:54:09.999+0000: 4886934.939: [GC 2079647K(2088640K), 0.0538060 secs]
2013-01-18T10:54:10.115+0000: 4886937.753: [Full GC 2088638K->2057070K(2088640K), 6.1674520 secs]
2013-01-18T10:54:19.016+0000: 4886943.956: [GC 2081495K(2088640K), 0.0553910 secs]
2013-01-18T10:54:19.118+0000: 4886946.803: [Full GC 2088639K->2054971K(2088640K), 6.1280980 secs]
2013-01-18T10:54:28.024+0000: 4886952.964: [GC 2081821K(2088640K), 0.0521580 secs]
2013-01-18T10:54:28.106+0000: 4886956.119: [Full GC 2088639K->2060868K(2088640K), 6.3114750 secs]
2013-01-18T10:54:37.534+0000: 4886962.475: [GC 2081790K(2088640K), 0.0520260 secs]
2013-01-18T10:54:37.636+0000: 4886965.448: [Full GC 2088639K->2058876K(2088640K), 6.1640440 secs]
2013-01-18T10:54:46.712+0000: 4886971.653: [GC 2080842K(2088640K), 0.0571690 secs]
2013-01-18T10:54:46.807+0000: 4886974.582: [Full GC 2087100K->2059338K(2088640K), 6.2864710 secs]
2013-01-18T10:54:55.971+0000: 4886980.911: [GC 2083885K(2088640K), 0.0530780 secs]
2013-01-18T10:54:56.059+0000: 4886983.794: [Full GC 2088605K->2058919K(2088640K), 6.3044720 secs]

Note that Full GC is taking approx 6 seconds each time here, making the application unusable.

Using Jconsole with the JVM running Play.

Jconsole is an application shipped with the Oracle JVM (and others) that enables us to connect to the JVM remotely via JMX and view internal metrics. Here's a very brief introduction to what it can do.

The preferred way to run it, is to run jconsole on your personal machine, and connect remotely to the application JVM. Start it up, feed it:

the address of the machine running the application, the port it's running on

the username and password for JMX.

Here's Jconsole's overview screen. Important things to note here are CPU usage of the application, and the memory consumption. 

This sawtooth memory consumption graph is ideal - The base of the sawtooth should be flat - indicating memory usage after garbage collection is stable, and not increasing. Continuously increasing base memory usage could indicate a memory leak, code deficiencies, or simply insufficient memory assigned.

Here's the display of the memory and GC statistics. Compare the graph values to the Max values on the display below to show much of the heap has been used.

Keep an eye on the GC time values in the lower value.

The New Generation (ParNew in this case) should tick up gradually, but in a tiny fraction of realtime.

The Old Generation (CMS in this case), should tick up very occassionally - and in tens to hundreds of a second chunks.

Explaining how to do GC tuning and analysis of memory usage is worth a series of articles on it's own, and this is a subject I'll cover at a later date. Until then, here a good article to get you started.

Collecting JMX data

GC logging is a relatively blunt tool, though log analysis tools like GCViewer help. Similarly, you can't really leave jconsole connected 24/7 to see what your application is doing.

The answer is to collect selected JMX metrics with another application, and store them. There are many applications that can do this - some of my favourites are.

Collectd - JMX plugin

OpenNMS - JMX collector

Hyperic

Having high-resolution JMX data at the time a problem occurred can be invaluable in solving problems, as you can trace what was happening immediately before the problem occurred.

Production quality Play apps. - part 2 - Puppet module

In the previous article, I went through a few common options for getting your Play code onto a target system in a controlled manner.

Now, here's the puppet module I use for configuring and running the Play application itself - you can download it from http://www.jmips.co.uk/play_framework/ -  This should be easy to get working with some basic Puppet experience.

Unpack play_framework_apps_with_puppet.tar.gz , and you should see two modules - "example_play_app", and "play_framework".

The "play_framework" module is a puppet defined type. The idea is you call it, as in  example_play_app/manifests/init.pp

  # this defined-type does all the heavy lifting to create a service
  # for play_framework stuff, we just need to feed it a name and uid/gid
  play_framework::play_service { "${module_name}":
    play_uid => '995',
  }

Example Play App module

By default, they play app will take it's name from the module name - so likely you'll want to rename it at some point. After you've renamed the module directory and class name, you need to choose a free UID to run your play application under. The name of the user created will be named after the application. Beware - avoid hypthens in puppet module names! It is possible to use hyphens, but it will complicate things.

After you've chosen a name and UID, take a look at the templates directory for the example play apps.

./templates/env-logger.xml.erb
./templates/start_script.erb
./templates/env-application.conf.erb

The first file is the logger configuration - you should configure your play application to look for this file, and configure it appropriately

The second is the script used to start the application - you should adjust this to your needs.

#!/usr/bin/env sh
exec java -Dhttp.port=9000 \
-Dhttp.address=0.0.0.0 \
-Dcom.sun.management.jmxremote.port=7020 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.password.file=/etc/play/<%= module_name %>/jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=/etc/play/<%= module_name %>/jmxremote.access \
-Dcom.sun.management.jmxremote.authenticate=true \
-Djava.rmi.server.hostname=<%= ipaddress %> \
-XX:+UseConcMarkSweepGC \
-Xloggc:/opt/<%= module_name %>/logs/gc.log \
-XX:-OmitStackTraceInFastThrow \
-XX:+PrintGCDateStamps \
-verbose:gc \
-Xms256M -Xmx256m \
-server  \
-Dconfig.file=/etc/play/<%= module_name %>/env-application.conf \
-Dlogger.file=/etc/play/<%= module_name %>/env-logger.xml \
-cp "/opt/<%= module_name %>/lib/*" play.core.server.NettyServer /opt/<%= module_name %>/

In this application we're using Netty, configured to run on port 9000 and JMX is enabled with a username/password. Java heap size is set to 256Mb, CMS GC is enabled, GC logging is enabled. All of the file locations use the puppet module name.

The third file -  env-application.conf.erb  - contains application specific configuration. Use this to set configuration that varies between environments - ie where a database server is located.

Play Framework module

The defined type module may need some adjusting for your environment, the one I use is configured for Ubuntu, using Upstart to run the Play service.

Again, take a look in the templates directory for this module.

./templates/jmxremote.password.erb
./templates/play_upstart.conf.erb
./templates/jmxremote.access.erb

The JMX files relate to the usernames/passwords and permissions for JMX connectivity. See this document to set this up correctly. My templates use puppet variables to allow for multiple play apps/jmx passwords. Either set these variables in your site manifests via extlookup, or hardcode the JMX config into the modules (not recommended for good security!)

The Upstart file is used to start, stop and monitor the Play application.

description "<%= play_service_name %>"
start on filesystem
stop on runlevel [!2345]
respawn
respawn limit 10 5
umask 022
oom never
env PLAY_SERVICE_HOME="/opt/<%= play_service_name %>"
post-stop script
        rm ${PLAY_SERVICE_HOME}/RUNNING_PID || true
end script
exec start-stop-daemon --start -c <%= play_service_name %> -d ${PLAY_SERVICE_HOME} --exec /etc/play/<%= play_service_name %>/start

Upstart will ensure the application is running, monitor it via it's PID, and restart it if it crashes. You should be able to start/stop/restart the service with "service example_play_app <stop/start/restart>"

Hopefully this should be enough to get your play application started and running. Next time, I'll briefly cover using JMX to monitor and tune the JVM of your Play application.

Production quality Play apps. - part 1 - Automatic Deployment

I've been working with a lot of Play applications recently, and was asked to share how I built the automatic deployment, config management, integration into Linux, etc.

In this post I'll cover the automatic deployment options, and it will be solely focused on how to get binaries onto systems in a controlled and repeatable fashion.

Automatic Deployment 

This all assumes you have a working build of your Play application already.

I'm showing the raw commands to run to get it to work, but this all should be scripted/automated with your automation system of choice. Here's an example shell script, but Mcollective or Saltstack should work too.

Method 1 - Quick and dirty - Deploy straight from Jenkins

This is quite simple, and perfect for a fast deploy from a Build/Test system. I would recommend not using this for production deployments, as it's difficult to guarantee the exact version of what you're deploying!

This assumes you have a working Play app build in Jenkins, and successfully built binaries, usually with SBT's "dist" command.

On your target server to run the Play application.

wget -nv http://<jenkins_server>/job/<play_jenkins_job>/lastSuccessfulBuild/artifact/*zip*/archive.zip -O /tmp/play_deploy/archive-<play_app_name>.zip

This just retrieves the latest Jenkins artifact for the build job, and writes it to a known location on disk

 unzip -q -o /tmp/play_deploy/archive-<play_app_name>.zip  -d /tmp/play_deploy/

Here we unpack the Jenkins ZIP artifact, to reveal the actual Play ZIP file.

cd /tmp/play_deploy/ unzip -q /tmp/play_deploy/archive/dist/<play_zip_name>-*.zip

Here we've unpacked the SBT built ZIP distribution of the Play libs.

sudo /bin/rm -rf /opt/<play_service_name>/lib

Clean up the final destination for the Play app - assuming you've stopped it already!

sudo /bin/mv /tmp/play_deploy/<play_unzipped_dir>-*/* /opt/<play_service_name>/ "

The result of this should be /opt/<play_service_name>/lib filled with the libraries from your Play application. You will likely have to tweak the names slightly, as the outer ZIP wrapper is related to the name of the Jenkins job, and the inside ZIP name is related to the Play project name.

At this point, you're ready for Puppet to take over.

Method 2 - Deploy from a binaries file store.

It's quite common to use a build system to populate a filesystem based binary store. eg. /mnt/play_binaries/<play_project_name>/<version>/<project_name>.zip

Implementing this is outside the scope of this article, but I'd use the Copy Artifact plugin in Jenkins.

To deploy from this, use wget/rsync or similar to copy the file to your target system in the /tmp/play_deploy/ directory, then, as the previous example

cd /tmp/play_deploy/ 

unzip -q /tmp/play_deploy/archive/dist/<play_zip_name>-*.zip 

sudo /bin/rm -rf /opt/<play_service_name>/lib

sudo /bin/mv /tmp/play_deploy/<play_unzipped_dir>-*/* /opt/<play_service_name>/ "

Again, at this point you should have /opt/<play_service_name>/lib populated with the Play application libraries, and you're ready for Puppet to take over the next step.

Method 3 - Deploy from Maven.

This assumes you've published your binaries to Maven, using SBT or similar, as in this example.
Be aware that SBT 0.11 does not properly generate the Maven metadata, the result of which is asking for the "latest" version will not work, but asking for a specific version will. See here for a solution, particularly the sbt-aether-deploy plugin.

If you chose to use Maven for deployment, be sure you know the difference between Release and Snapshot versions, so you're able to guarantee the versions of your code, and all dependencies in each production release.

To deploy, you'll need a custom maven XML file like this one, on the target system. - call it <play_app_name>.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.fake</groupId>
  <artifactId>fake</artifactId>
  <version>1.0</version>
  <packaging>jar</packaging>
  <name>fake</name>
  <url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<excludeTransitive>false</excludeTransitive>
<outputDirectory>/tmp/play_deploy/play_service_name/lib/</outputDirectory>
</properties>
<dependencies>
<dependency>
<groupId>com.mycompany</groupId>
<artifactId>my_play_artifact name</artifactId>
<version>version_to_deploy</version>
<type>jar</type>
</dependency>
</dependencies>
</project>

Assuming you've got your local Maven config set up correctly (ie. your ~/.m2/settings.xml as in this example ) so Maven on your target system can see your internal Maven repo, you can now use Maven to request your libraries.

cd /tmp/play_deploy/ 

mvn -U    dependency:copy-dependencies -f <play_app_name>.xml

This should result in the version you selected in the XML file being deployed to the path you specified in the XML file, along with all dependencies required to run it.

Now, as before, move it to the correct place.

sudo /bin/rm -rf /opt/<play_service_name>/lib
sudo /bin/mv /tmp/play_deploy/<play_service_name>-*/* /opt/<play_service_name>/ "

 Now the code is deployed, you're ready for Puppet to take over.

Method 4 - System Packaging

Systems Admins already have tools to automatically check/deploy/upgrade the software on the systems under their control. RPM and deb are the most popular formats for Linux

Converting Play applications (typically distributed in ZIP format) into these formats is a little more involved, but worth it for larger system estates. I'll describe the process, but the specifics are a little outside a blog post. Perhaps I'll cover it in the future.

Essentially, the idea is as part of your build process, to take the ZIP file, convert it into a RPM/deb, then upload it to your repository. Then it will be available to be installed, using native OS commands, as if you were installing a new version of Apache HTTPD.

This project aims to bring this support natively to SBT, but until then, you're going to have to script this yourself, with the help of some Jenkins plugins like this.


Next time I'll cover the Puppet module I use to automate the configuration and integration of the Play app into the Linux environment.