“Perfbench” update: Tapestry 5 and Grails

I attempted to port the Seam “hotel booking” sample to Grails 1.1.1 as well as Tapestry 5.1, to add to this head-to-head comparison of Apache Wicket and JBoss Seam done earlier this year. You can find the code here: [browse] [SVN]

Here are the relative performance test results (page response time in milliseconds):

Heap-dump comparison for 20 concurrent user / sessions:

I’ll save detailed framework impressions and comparison for a later blog post, and I may need to make updates to the code based on feedback. Here are some overall observations:

• Grails was far more productive than Tapestry 5. This was mainly due to the documentation quality of Grails compared to the scattered and not very well organized Tapestry documentation.
• Ease of writing custom tag-libraries is IMO one of the best things about Grails.
• The two missing rows in the Tapestry performance test results are because I gave up trying to implement form field validation over Ajax when the user “tabs out”. Actually I didn’t implement form-field validation at all because I could not figure out how to integrate Hibernate Validator. Maybe I missed something obvious, do let me know.
• Grails still has some way to go in terms of performance. I am told that significant performance optimizations for GSP will make it into 1.2
• Overall, Wicket is fastest, with Tapestry coming a close second.
• Wicket also takes up the least amount of heap. 31 MB of the Grails heap alone is taken up by instances of the “groovy.lang.ExpandoMetaClass”.
• Session usage of the Seam + JSF combination is significantly higher compared to all the rest, around 760 KB per session.

Thoughts on Tapestry 5

I was very interested in seeing if Tapestry 5 lived up to its promise of significantly better performance and scalability – which is known to come at a cost: a programming model that treats web-pages as static structures that can be “pooled”. One example of what this means is that you can’t rely on constructors anymore so you now have to figure out other ways of re-initializing your server-state (if applicable) after you are handed an instance from the pool. The “magic” of getting a page from the pool or getting hold of request or session scoped variables is all achieved predictably – using annotations.

Another example: due to the way Tapestry does byte-code manipulation – the annotation approach works only for “private” fields and not “protected” ones. I found this out the hard way, encountering an exception when I tried to extend from an abstract base class and re-use code the “old-fashioned” object-oriented way.

There’s a lot more I can go into, but for now, to summarize my experience as a long-time Wicket user trying out Tapestry 5 – the top few differences I found are:

• Having to work around the implications of Tapestry pages being static and “annotation driven” programming like I described above.
• Tapestry does not have the equivalent of Wicket Model-s which give you fine-grained control over the data you need to render (and bind) within a page and how much of it you decide to persist in the session. Model-s are certainly the aspect of Wicket that takes the most getting-used-to, but I have a new-found respect for them now after trying Tapestry.
• I would rate the Ajax infrastructure of Wicket higher: for example Wicket Behavior-s are cleaner compared to Tapestry Mixin-s, and calling / handling Ajax requests (especially when you depend on server-side state) is far easier.
• Tapestry 5 does not have built-in support for web-flow or “conversation scope” so I had to write some code to maintain a map of bookings in the session. BTW I did not have to do this for Grails as I could use WebFlow.

So coming back to the question, does the “Tapestry way” have a distinct advantage and are the quirks of the programming model worth it? The opinion of the Tapestry team on this is pretty clear. Take this quote from the Tapestry home page for example:

In some Tapestry-like frameworks, such as Faces and Wicket, the page structure is more dynamic, at the cost of storing much, much more data in the HttpSession.

Well, looking at the performance comparison results, my personal conclusion is that statements like the above are incorrect. In fact Tapestry seems to take up more heap space than Wicket for the same functionality. Also worth noting are the results of running the Wicket application using the HttpSessionStore instead of the default SecondLevelCacheSessionStore + DiskPageStore:

Because this time, Wicket is faster now even for the “post confirm booking” page. This particular action actually ends up displaying 200 items in an HTML table by the end of the test run for 20 concurrent users, since pagination is not being used.

The benchmark has been designed to be easy for you to run once you check-out the code, refer the end of this blog post for details. The details of the environment I used for the results posted here are as follows:

• java.runtime.version : 1.6.0_16-b01
• java.vm.name : Java HotSpot(TM) Server VM
• java.vendor : Sun Microsystems Inc.
• os.name : Windows XP
• os.version : 5.1
• sun.os.patch.level : Service Pack 3
• Intel Core 2 Duo 3.01GHz
• 3 GB RAM

Feel free to post your results or suggest changes to the code.

Seam / JSF vs Wicket: performance comparison

A while after Seam support for Apache Wicket was announced, I downloaded Seam and took a look at the Wicket example. Then an idea struck – how about doing a performance comparison – I mean, here was the very same application implemented in JSF and Wicket – right? So I decided to write a JMeter script for both the JSF and Wicket versions of the Seam “hotel booking” example and compare results. I started right away but very soon got tired of waiting for app-server re-starts on my trusty laptop (Jetty has really spoiled me) – and it kind of bothered me that the Seam examples were not in Maven 2 layout. Anyway, at some point I decided to re-write an “EJB-free” version of the Seam booking example using just Wicket and JPA. And use Maven and Jetty. And… one thing led to another – and what I ended up doing is this:

• Adapted the Seam JPA example (the one that does not use EJB) as the baseline application. Converted it into a Maven 2 WAR project which can run on Jetty.
• Implemented what I hope is the exact equivalent of the above using only Wicket and JPA, also Maven-ized and Jetty-fied. Decided to also experiment with some of the ideas in this blog post.
• Wrote a JMeter script for both applications taking care to exercise identical functionality.
• Used an Ant script to run the JMeter scripts in batch mode (passing number of concurrent users as a parameter) and start / stop Jetty in sync.
• Automated the entire cycle of running the load test for 1, 5, 10, 15 and 20 concurrent users including some code to parse the JMeter logs and generate a CSV file of tabular results.
• Also included an Ant target that takes a JVM heap dump at the end of the load test – just before the users are logged out and the HTTP sessions killed.

I was able to re-use the official Seam Wicket example to some extent, mainly the HTML files – but the Java side is almost completely re-written. The Seam booking example covers quite a bit of ground from the framework comparison point of view – for example:

• Security – some pages are secured and redirect to a login page
• Templating – some pages inherit from a common layout with a header / footer defined
• Ajax –
  • hotel search results refresh as you type and a “busy” image is shown during the Ajax request
  • form field validations occur as soon as the field loses focus
• Session scope – some state is stored and retrieved in the session
• Page navigation / state –
  • user navigation state transparently managed by the framework
  • user should be able to work in multiple concurrent conversations (browser tabs)
  • the browser back button should work as expected
• JPA – getting access to the entity manager and transaction management
• Forms – binding, validation and displaying error / info feedback to the user
• Hibernate Validator – annotations on the JPA entity classes are re-used for form-validation

Both sides use the exact same entities, persistence.xml and initial HSQLDB import. I’m using the latest Seam 2.1.1-GA and Wicket 1.3.5. Disclaimer: my Wicket code may not be ideal, and I’ve also experimented with a custom RequestCycle for JPA and tried to use inherited models as far as possible. Instructions on how you can download the source and run the scripts on your local machine are at the end of this blog post.

I may do some follow up blog posts on how the code compares between the Seam / JSF and Wicket implementations, and also share some tips on writing JMeter scripts and automation using Jetty, Ant etc. I’m also expecting to have to make corrections and changes to the code based on feedback. For now, I’ll summarize my observations on performance and memory usage.

Performance:
In the JMeter script, except for the login and logout (first two rows and last row), the actions are executed in a loop ten times for each concurrent user. So if the number of concurrent users is 20, the login and logout actions happen 20 times and the rest 20 x 10 times. The numbers below are average page response time in milliseconds.

Wicket appears to be faster by a wide margin. For two pages (“ajax post search” and “post confirm booking”) the results are a bit closer. This can possibly be explained by the fact that these particular actions display the results from a relatively expensive database query. My amateur profiling attempts suggest that the database query is taking most of the time here.

One thing I have to mention: the “cc number” and “cc name” requests are simulations of the Ajax validations of the credit card number and name fields on the booking form. For these particular requests on the JSF side, the entire form is being POST-ed instead of just the value of the form field being validated when the user tabs out (onblur). So the difference here is quite dramatic. I did try adding ajaxSingle=”true” in the JSF view but it did not appear to work (I used HttpFox while building the JMeter scripts). I can re-post the updated results if someone lets me know what changes need to be made to “book.xhtml” to get the Ajax validation to work as expected.

Memory Usage:
The JMeter script can be told to skip the logout page and I wired up one of the Ant targets to take a JVM heap dump / snapshot as soon as the JMeter script completes. So I can compare what the heap looks like just after a load test when all the concurrent HTTP sessions are alive.

I’m very much a NetBeans user but I have to say that the Eclipse Memory Analyzer is far better than what the built-in NetBeans Profiler offers for looking at JVM heap dumps. Here are some side-by-side screenshots of the heap analysis after running the JMeter script for 20 concurrent users.

“Top Consumers” report below showing the classes that dominate memory usage:

The “dominator tree” report below is very useful to see which objects hold on to the most memory directly as well as indirectly. The column headings after “Class name” are Shallow Heap, Retained Heap and Percentage. Shallow Heap means the memory consumed by a single object and Retained Heap is the sum of shallow sizes of all objects that will be garbage collected if the given object is garbage collected.

Looking at the above two reports we can infer that on the Seam / JSF side, the 20 sessions each take up about 800 KB adding up to around 16 MB total. On the Wicket side the 20 sessions add up to around 1.5 MB. On the Wicket side it is the DiskPageStore that appears to hold the most memory and we can see what is going on here after drilling down a little:

The Wicket DiskPageStore uses SoftReference-s to serialized pages so the memory will be reclaimed by the JVM if needed. And the SerializedPageWithSession holds a WeakReference to the actual page instance (MainPage). You can also spot the byte-array which is the result of page serialization. If a serialized page is requested (perhaps the user hit the browser back-button) and the page is no longer in memory because the SoftReference has been GC-ed – it will be restored from the temp file that the DiskPageStore has been saving pages to.

I’m totally impressed by the Eclipse Memory Analyzer. Here we can see the break-up of the contents of the largest HTTP session on both sides. I think we can safely blame JSF for the lion’s share of memory usage on the left:

Finally, a summary of the heap-dump comparison (for 20 users) collated from the various reports:

Instructions:

Prerequisites:

• JDK 1.5 or greater installed
• Apache Ant installed
• JMeter 2.X available unzipped somewhere (better use latest 2.3.2)

Steps:

• Do a Subversion check out of the source from here: http://perfbench.googlecode.com/svn/trunk/perfbench/
• Create a perfbench/build.properties file that points to your JMeter installation. You can look at the comment in perfbench/build.xml for an example.
• Open a command prompt, change directory to perfbench/seam-jpa
• If running for the first time, use the command “ant jmeter-cycle”. The build script would prompt for the number of threads, so enter “1”. It may take time for all the required JAR files to get downloaded. Once you see Jetty start and stop successfully, you should be all set to run the actual benchmark.
• To start the benchmark run “ant jmeter-cycle-full”. This should take 2 – 3 minutes to run a series of tests for 1, 5, 10, 15 and 20 concurrent users. Results will be dumped into perfbench/target. You can look at the *.csv file at the end for the results.
• You can also run a load test which saves a snapshot of the heap dump towards the end by running “ant jmeter-cycle-heapdump”.
• Repeat the previous 3 steps after changing working directory to perfbench/wicket-jpa

Update: perfbench/build.xml starts Jetty with JVM options “-Xms64m -Xmx64m” and you may need to change this if you want to experiment with more concurrent users.