Seam / JSF vs Wicket: performance comparison

A while after Seam support for Apache Wicket was announced, I downloaded Seam and took a look at the Wicket example. Then an idea struck – how about doing a performance comparison – I mean, here was the very same application implemented in JSF and Wicket – right? So I decided to write a JMeter script for both the JSF and Wicket versions of the Seam “hotel booking” example and compare results. I started right away but very soon got tired of waiting for app-server re-starts on my trusty laptop (Jetty has really spoiled me) – and it kind of bothered me that the Seam examples were not in Maven 2 layout. Anyway, at some point I decided to re-write an “EJB-free” version of the Seam booking example using just Wicket and JPA. And use Maven and Jetty. And… one thing led to another – and what I ended up doing is this:

• Adapted the Seam JPA example (the one that does not use EJB) as the baseline application. Converted it into a Maven 2 WAR project which can run on Jetty.
• Implemented what I hope is the exact equivalent of the above using only Wicket and JPA, also Maven-ized and Jetty-fied. Decided to also experiment with some of the ideas in this blog post.
• Wrote a JMeter script for both applications taking care to exercise identical functionality.
• Used an Ant script to run the JMeter scripts in batch mode (passing number of concurrent users as a parameter) and start / stop Jetty in sync.
• Automated the entire cycle of running the load test for 1, 5, 10, 15 and 20 concurrent users including some code to parse the JMeter logs and generate a CSV file of tabular results.
• Also included an Ant target that takes a JVM heap dump at the end of the load test – just before the users are logged out and the HTTP sessions killed.

I was able to re-use the official Seam Wicket example to some extent, mainly the HTML files – but the Java side is almost completely re-written. The Seam booking example covers quite a bit of ground from the framework comparison point of view – for example:

• Security – some pages are secured and redirect to a login page
• Templating – some pages inherit from a common layout with a header / footer defined
• Ajax –
  • hotel search results refresh as you type and a “busy” image is shown during the Ajax request
  • form field validations occur as soon as the field loses focus
• Session scope – some state is stored and retrieved in the session
• Page navigation / state –
  • user navigation state transparently managed by the framework
  • user should be able to work in multiple concurrent conversations (browser tabs)
  • the browser back button should work as expected
• JPA – getting access to the entity manager and transaction management
• Forms – binding, validation and displaying error / info feedback to the user
• Hibernate Validator – annotations on the JPA entity classes are re-used for form-validation

Both sides use the exact same entities, persistence.xml and initial HSQLDB import. I’m using the latest Seam 2.1.1-GA and Wicket 1.3.5. Disclaimer: my Wicket code may not be ideal, and I’ve also experimented with a custom RequestCycle for JPA and tried to use inherited models as far as possible. Instructions on how you can download the source and run the scripts on your local machine are at the end of this blog post.

I may do some follow up blog posts on how the code compares between the Seam / JSF and Wicket implementations, and also share some tips on writing JMeter scripts and automation using Jetty, Ant etc. I’m also expecting to have to make corrections and changes to the code based on feedback. For now, I’ll summarize my observations on performance and memory usage.

Performance:
In the JMeter script, except for the login and logout (first two rows and last row), the actions are executed in a loop ten times for each concurrent user. So if the number of concurrent users is 20, the login and logout actions happen 20 times and the rest 20 x 10 times. The numbers below are average page response time in milliseconds.

Wicket appears to be faster by a wide margin. For two pages (“ajax post search” and “post confirm booking”) the results are a bit closer. This can possibly be explained by the fact that these particular actions display the results from a relatively expensive database query. My amateur profiling attempts suggest that the database query is taking most of the time here.

One thing I have to mention: the “cc number” and “cc name” requests are simulations of the Ajax validations of the credit card number and name fields on the booking form. For these particular requests on the JSF side, the entire form is being POST-ed instead of just the value of the form field being validated when the user tabs out (onblur). So the difference here is quite dramatic. I did try adding ajaxSingle=”true” in the JSF view but it did not appear to work (I used HttpFox while building the JMeter scripts). I can re-post the updated results if someone lets me know what changes need to be made to “book.xhtml” to get the Ajax validation to work as expected.

Memory Usage:
The JMeter script can be told to skip the logout page and I wired up one of the Ant targets to take a JVM heap dump / snapshot as soon as the JMeter script completes. So I can compare what the heap looks like just after a load test when all the concurrent HTTP sessions are alive.

I’m very much a NetBeans user but I have to say that the Eclipse Memory Analyzer is far better than what the built-in NetBeans Profiler offers for looking at JVM heap dumps. Here are some side-by-side screenshots of the heap analysis after running the JMeter script for 20 concurrent users.

“Top Consumers” report below showing the classes that dominate memory usage:

The “dominator tree” report below is very useful to see which objects hold on to the most memory directly as well as indirectly. The column headings after “Class name” are Shallow Heap, Retained Heap and Percentage. Shallow Heap means the memory consumed by a single object and Retained Heap is the sum of shallow sizes of all objects that will be garbage collected if the given object is garbage collected.

Looking at the above two reports we can infer that on the Seam / JSF side, the 20 sessions each take up about 800 KB adding up to around 16 MB total. On the Wicket side the 20 sessions add up to around 1.5 MB. On the Wicket side it is the DiskPageStore that appears to hold the most memory and we can see what is going on here after drilling down a little:

The Wicket DiskPageStore uses SoftReference-s to serialized pages so the memory will be reclaimed by the JVM if needed. And the SerializedPageWithSession holds a WeakReference to the actual page instance (MainPage). You can also spot the byte-array which is the result of page serialization. If a serialized page is requested (perhaps the user hit the browser back-button) and the page is no longer in memory because the SoftReference has been GC-ed – it will be restored from the temp file that the DiskPageStore has been saving pages to.

I’m totally impressed by the Eclipse Memory Analyzer. Here we can see the break-up of the contents of the largest HTTP session on both sides. I think we can safely blame JSF for the lion’s share of memory usage on the left:

Finally, a summary of the heap-dump comparison (for 20 users) collated from the various reports:

Instructions:

Prerequisites:

• JDK 1.5 or greater installed
• Apache Ant installed
• JMeter 2.X available unzipped somewhere (better use latest 2.3.2)

Steps:

• Do a Subversion check out of the source from here: http://perfbench.googlecode.com/svn/trunk/perfbench/
• Create a perfbench/build.properties file that points to your JMeter installation. You can look at the comment in perfbench/build.xml for an example.
• Open a command prompt, change directory to perfbench/seam-jpa
• If running for the first time, use the command “ant jmeter-cycle”. The build script would prompt for the number of threads, so enter “1”. It may take time for all the required JAR files to get downloaded. Once you see Jetty start and stop successfully, you should be all set to run the actual benchmark.
• To start the benchmark run “ant jmeter-cycle-full”. This should take 2 – 3 minutes to run a series of tests for 1, 5, 10, 15 and 20 concurrent users. Results will be dumped into perfbench/target. You can look at the *.csv file at the end for the results.
• You can also run a load test which saves a snapshot of the heap dump towards the end by running “ant jmeter-cycle-heapdump”.
• Repeat the previous 3 steps after changing working directory to perfbench/wicket-jpa

Update: perfbench/build.xml starts Jetty with JVM options “-Xms64m -Xmx64m” and you may need to change this if you want to experiment with more concurrent users.

---

Possibly related posts: (automatically generated)

• Wicket Impressions, moving from Spring MVC / WebFlow
• Facebook flips out