My new open-source project: Karate

After almost 7 years I’m blogging again. It feels great.

And it is good to be back in the thick of open-source action, founding a brand new project, and actively spreading the word and building a community from scratch.  And the best part is the support I’m getting from my employer – Intuit.

For me personally, this is some of the best hacking I have ever done.  There were times when it seemed like things magically came together. After programming for what feels like decades, the 10,000 Hour Rule feels Very Real to me now.

For those following this blog, I’m likely to be blogging over on Medium from now on.

Do have a look at my first blog post over there, and please do help spread the word !

Here’s the link:

Karate: Web Services Testing Made Simple.


“Comparing JVM Web Frameworks” – a response to Matt Raible

For the last few minutes I’ve been trying to post a comment on this blog post and the blog engine there (Roller) throws an exception. Matt links to one of my tweets as an example of the “anger” he has inspired. Here is my response, hopefully at least the blog pingback will register.

Matt, you’ve linked to one of my tweets as an example of the “anger” you have inspired, so I’ll comment.

I have no problem with the web-frameworks comparison presentation you have been making for a while, sure, it is probably highly entertaining in a conference setting, audience participation and all. For the record, I have never agreed with some of your conclusions, having followed your legendary presentation for the last four years or so, but that is another story altogether. Just one example is the fact that you gave Apache Wicket negative marks (was that 2007 or 2008?) because you couldn’t integrate Sitemesh (of all things), and you were trying to do this the night before your presentation. Many in the Wicket community remember this incident well.

What troubles me (and many others in the community) is that you give the impression that your ratings are based on objective information, but that is not true, see this tweet from you: What you are doing is adjusting ratings based on who in the community shouts the loudest. I can’t help saying that this approach comes across as highly arrogant and condescending, you seem to expect framework developers and proponents to rush over and fawn over you to get better ratings, like waiters in a restaurant trying to impress a food-critic for Michelin stars. If you are truly someone who is qualified to evaluate the framework relative strengths and weaknesses, you should be able to come out with your conclusions and stick to them, instead of relying on “community feedback”.

So please, by all means continue to do your web-framework comparison song and dance at conferences and the like, but please stop this charade of serving as an objective source of information on web-framework comparison. You certainly do have a reputation and level of respect in the Java EE community, and I guess there are still many development teams and stakeholders who would blindly follow your advice and recommendations, which I find highly unfortunate (for example, don’t get me started on your recent SOFEA obsession ;).

It looks like you had a great time at the Rich Web Experience, I just hope you don’t add “conference ambience” as a rating factor, because in that case other frameworks don’t stand a chance against Spring MVC and Grails. Just saying.

Anyway, for those wondering if I am anywhere near qualified to comment as Matt Raible, please have a look at some of the work I have done here for example:

“Perfbench” update: Tapestry 5 and Grails

I attempted to port the Seam “hotel booking” sample to Grails 1.1.1 as well as Tapestry 5.1, to add to this head-to-head comparison of Apache Wicket and JBoss Seam done earlier this year. You can find the code here: [browse] [SVN]

Here are the relative performance test results (page response time in milliseconds):


Heap-dump comparison for 20 concurrent user / sessions:


I’ll save detailed framework impressions and comparison for a later blog post, and I may need to make updates to the code based on feedback. Here are some overall observations:

  • Grails was far more productive than Tapestry 5. This was mainly due to the documentation quality of Grails compared to the scattered and not very well organized Tapestry documentation.
  • Ease of writing custom tag-libraries is IMO one of the best things about Grails.
  • The two missing rows in the Tapestry performance test results are because I gave up trying to implement form field validation over Ajax when the user “tabs out”. Actually I didn’t implement form-field validation at all because I could not figure out how to integrate Hibernate Validator. Maybe I missed something obvious, do let me know.
  • Grails still has some way to go in terms of performance. I am told that significant performance optimizations for GSP will make it into 1.2
  • Overall, Wicket is fastest, with Tapestry coming a close second.
  • Wicket also takes up the least amount of heap. 31 MB of the Grails heap alone is taken up by instances of the “groovy.lang.ExpandoMetaClass”.
  • Session usage of the Seam + JSF combination is significantly higher compared to all the rest, around 760 KB per session.

Thoughts on Tapestry 5

I was very interested in seeing if Tapestry 5 lived up to its promise of significantly better performance and scalability – which is known to come at a cost: a programming model that treats web-pages as static structures that can be “pooled”. One example of what this means is that you can’t rely on constructors anymore so you now have to figure out other ways of re-initializing your server-state (if applicable) after you are handed an instance from the pool. The “magic” of getting a page from the pool or getting hold of request or session scoped variables is all achieved predictably – using annotations.

Another example: due to the way Tapestry does byte-code manipulation – the annotation approach works only for “private” fields and not “protected” ones. I found this out the hard way, encountering an exception when I tried to extend from an abstract base class and re-use code the “old-fashioned” object-oriented way.

There’s a lot more I can go into, but for now, to summarize my experience as a long-time Wicket user trying out Tapestry 5 – the top few differences I found are:

  • Having to work around the implications of Tapestry pages being static and “annotation driven” programming like I described above.
  • Tapestry does not have the equivalent of Wicket Model-s which give you fine-grained control over the data you need to render (and bind) within a page and how much of it you decide to persist in the session. Model-s are certainly the aspect of Wicket that takes the most getting-used-to, but I have a new-found respect for them now after trying Tapestry.
  • I would rate the Ajax infrastructure of Wicket higher: for example Wicket Behavior-s are cleaner compared to Tapestry Mixin-s, and calling / handling Ajax requests (especially when you depend on server-side state) is far easier.
  • Tapestry 5 does not have built-in support for web-flow or “conversation scope” so I had to write some code to maintain a map of bookings in the session. BTW I did not have to do this for Grails as I could use WebFlow.

So coming back to the question, does the “Tapestry way” have a distinct advantage and are the quirks of the programming model worth it? The opinion of the Tapestry team on this is pretty clear. Take this quote from the Tapestry home page for example:

In some Tapestry-like frameworks, such as Faces and Wicket, the page structure is more dynamic, at the cost of storing much, much more data in the HttpSession.

Well, looking at the performance comparison results, my personal conclusion is that statements like the above are incorrect. In fact Tapestry seems to take up more heap space than Wicket for the same functionality. Also worth noting are the results of running the Wicket application using the HttpSessionStore instead of the default SecondLevelCacheSessionStore + DiskPageStore:


Because this time, Wicket is faster now even for the “post confirm booking” page. This particular action actually ends up displaying 200 items in an HTML table by the end of the test run for 20 concurrent users, since pagination is not being used.

The benchmark has been designed to be easy for you to run once you check-out the code, refer the end of this blog post for details. The details of the environment I used for the results posted here are as follows:

  • java.runtime.version : 1.6.0_16-b01
  • : Java HotSpot(TM) Server VM
  • java.vendor : Sun Microsystems Inc.
  • : Windows XP
  • os.version : 5.1
  • sun.os.patch.level : Service Pack 3
  • Intel Core 2 Duo 3.01GHz
  • 3 GB RAM

Feel free to post your results or suggest changes to the code.

Announcing Flazr – download RTMP (Flash video) streams

A quick post to announce my new open-source project:

I had done most of the coding early this year and was really feeling bad about sitting on the code and not having time to package and release as open source. Judging by the number of hits I keep getting on this blog entry I made previously on “how to download RTMP streams with Red5” – there appears to be a lot of demand for this.

Flazr is a Java implementation of the RTMP protocol using the excellent Apache Mina project. I think I was able to come up with a far more concise and readable implementation than what the Red5 project uses – which is not that surprising – because the scope of Red5 is much bigger and Flazr focuses only on the client side. I do feel that Flazr will be useful as an additional reference for those interested in understanding the details of the RTMP protocol.

One of the highlights of Flazr (especially from a Java perspective) is the usage of Groovy for scripting. Groovy allows the end-user to script things such as scraping the HTML from a given URL, parsing it and then invoking the RTMP client routine with the right parameters. All this in a platform-independent manner, without the need to compile anything and using a normal text-editor. I expect Flazr’s Groovy approach to be much easier to use (and arguably more powerful) than the PERL-driven approach that projects like “get_iplayer” and “rtmpdump” use.

Flazr has been designed so that end-users can extend the capabilities far beyond what the core supports – using just some plain-text Groovy scripts. This means that end-users won’t need to depend on the project team (*ahem* just me for now :) to pitch in and make changes. I’m quite interested to see how this turns out in practice.

All in all I’m quite positive about Groovy’s useful-ness, the example scripts that come with the Flazr distribution demonstrate usage of the nifty XmlSlurper and implementing a Java interface on-the-fly. End users should be able to even plug in third-party libraries if they so desire and make use of all that the JVM offers.

Why you should use the Maven Ant Tasks instead of Maven or Ivy

Like many others I really like Ant and I intend to keep using it until something better comes along. And I strongly feel that when you try to use *only* Maven for everything, you give up control over a few things.

Are there things that Ant does better than Maven? Oh yes, IMHO. Here are a few examples of typical Java web-app build related stuff that turn out to be “edge cases” in Maven meaning that you have to struggle, read up on obscure plugins or resort to ugly hacks in order to get things to work.

Replacing text in files only for development mode

<target name="jetty-setup-dev" unless="production.mode">
	<echo>making webapp changes for development mode...</echo>       
	<replace file="target/${}/WEB-INF/web.xml" 

The example above shows how I switch to a custom Wicket filter in web.xml that supports hot-class-reloading and debug logging – but only in development mode. The Ant replaceregexp task can give you even more regex driven find-and-replace goodness.

Capturing input from the user

<target name="confirm">
	<input message="Are you sure?" validargs="y,n" addproperty="input"/>
	<condition property="abort">
		<equals arg1="n" arg2="${input}"/>
	<fail if="abort">User aborted.</fail>       

All the best trying to do that in Maven. Oh and using the maven-antrun-plugin doesn’t count – that’s cheating ;)

ZIP a bunch of files from where *you* want into what *you* want

<target name="dist-jtrac-src">
	<mkdir dir="target"/>
	<zip destfile="target/">
		<zipfileset dir="nbproject" prefix="jtrac/nbproject"/>
		<zipfileset dir="etc" prefix="jtrac/etc"/>
		<zipfileset dir="src" prefix="jtrac/src"/>
		<zipfileset dir="." includes="*.*" prefix="jtrac"/>

Maven apologists will promptly point you to the great Maven Assembly Plugin. Sorry, no thanks. I sincerely tried to understand how to use *that* piece of work a few times in the past but gave up in despair.

Execute an arbitrary Java class with control over everything (args, classpath, etc.)

<target name="hbm-export" depends="compile">
	<input message="Hibernate Dialect:" addproperty="dialect"/>
	<java classname="org.hibernate.tool.hbm2ddl.SchemaExport" fork="true">
			<path refid="test.classpath"/>
			<path path="target/classes"/>           
		<jvmarg value="-Dhibernate.dialect=${dialect}"/>
		<arg value="--text"/>
		<arg value="--delimiter=;"/>
		<arg value="--output=target/schema.sql"/>
		<arg value="src/main/resources/*.hbm.xml"/>

This is a target I keep around to forward-engineer the DDL from my Hibernate entities on demand, for any database of the user’s choosing – just enter the dialect at the prompt.

There are times when even Ant may not have all the answers and a technique I use (or misuse) effectively is to use a custom Java class to extend Ant. So I don’t even need to learn how to create custom Ant tasks and mess around with the Ant classpath ensuring that the build remains “xcopy” friendly. Now, the “Maven way” to handle custom stuff is to write a custom plugin. Which is the easiest thing in the world right? Heh – don’t get me started.

There are quite a few more examples I can think of but that’s not the main point of this blog post. Hopefully I have made a case for “why Ant” above. A deal-clincher for me is the fact that all popular IDE-s have excellent support for Ant. NetBeans (which I use) takes it to a whole different level of tight integration – for e.g. you can map Ant targets to toolbars, shortcuts and things like that. Yes, IDE support for Maven is getting there but not as good as Ant IMO.

But am I saying that you shouldn’t use Maven at all? Not really. Let me explain.

For Java projects, the essential one thing that Maven does well and Ant cannot – is dependency management. So IMO the way to go is to use *both* Maven and Ant. The problem is that Maven obviously goes out of the way to *not* use Ant.

Fortunately, there *is* a way to get the information about dependencies from your Maven “pom.xml” project file across to your Ant “build.xml” file. I am referring to the Maven Ant Tasks. The thing is – there is very little documentation out there on how to use this. My conspiracy theory is that the Maven guys don’t want you to know about the existence of the Maven Ant Tasks. This is backed up by the fact that the two options for using Ant mentioned here in the official Maven FAQ cunningly avoid linking to the actual Maven Ant Tasks! (update: more than half a year after this blog post was published, the Maven FAQ was updated with a new entry “How can I use Maven features in an Ant build?” :)

I’ve explored custom plugin based approaches for bridging Maven and Ant with some success, and I’ve gotten some good feedback and even patches for the NetBeans specific integration that I tried for. I wasn’t aware of the Maven Ant Tasks until recently because as I said, the Maven guys have done a good job of making it hard to find. So I hope this post helps improve things on this front.

How to use the Maven Ant Tasks
I’ll use my Seam / JSF vs Wicket comparison project as a reference. First create a standard pom.xml like you would normally as in this example, just focus on dependencies.

This is the magic snippet that goes into build.xml that establishes the link between Maven and Ant:

<target name="mvn-init" unless="compile.classpath" xmlns:artifact="urn:maven-artifact-ant">    
    <typedef resource="org/apache/maven/artifact/ant/antlib.xml" uri="urn:maven-artifact-ant" 
    <condition property="maven.repo.local" value="${maven.repo.local}" else="${user.home}/.m2/repository">
        <isset property="maven.repo.local"/>
    <artifact:localRepository id="local.repository" path="${maven.repo.local}"/> 
    <artifact:pom file="pom.xml" id="maven.project"/>
    <artifact:dependencies pathId="compile.classpath" filesetId="compile.fileset" useScope="compile">
        <pom refid="maven.project"/>
        <localRepository refid="local.repository"/>
    <artifact:dependencies pathId="test.classpath" filesetId="test.fileset" useScope="test">
        <pom refid="maven.project"/>
        <localRepository refid="local.repository"/>
    <artifact:dependencies pathId="runtime.classpath" filesetId="runtime.fileset" useScope="runtime">
        <pom refid="maven.project"/>
        <localRepository refid="local.repository"/>
  • You need just the maven-ant-tasks JAR in “lib” as per line #03, if you are picky about that, you can even consider downloading it automatically as part of the build. You can find details on how to do that in this blog post on how to use the Maven Ant Tasks instead of Ivy by Josh Suereth (who deserves credit for some of the ideas in this post).
  • I prefer bootstrapping the Maven integration within a target instead of globally for the build so that targets that don’t depend on Maven can run faster. Also, I don’t have to “pollute” the entire build.xml with the XML namespace stuff, it is isolated to just within this one target.
  • The unless=”compile.classpath” in line #01 is a nice way to ensure that the bootstrapping happens only once even when Ant executes multiple targets that depend on each other.
  • Lines #04 – 06 allows the user to override the default local repository location by having a “maven.repo.local” property entry set for e.g. in “”. You can even use relative paths here – so all your dreams of having Maven builds that work even when you copy the project folders across to some other machine – can come true ;)
  • The rest intializes classpath and fileset variables for compile, test and runtime scopes – which should be sufficient for most WAR projects.
  • Something that really surprised the heck out of me is that you don’t need Maven installed for all this to work! You are getting all the benefits from a single teeny tiny JAR file. Spread the word, people!

Now you can write your compile target like this, note the ‘classpathref” on line #03:

<target name="compile" depends="mvn-init">
    <mkdir dir="target/classes"/>        
    <javac srcdir="src/main/java" destdir="target/classes" classpathref="compile.classpath"/>
    <copy todir="target/classes">
        <fileset dir="src/main/java" excludes="**/*.java"/>
    <copy todir="target/classes" failonerror="false">
        <fileset dir="src/main/resources"/>

I normally use Jetty for development (like this) so I don’t need to assemble a WAR most of the time. People who haven’t tried this really don’t know what they are missing, huge time saver – not to mention tight IDE integration for things like debugging. But when the time comes to build a WAR this is how you could do it (note the reference to the Maven fileset at line #11):

<macrodef name="war-exploded">
    <attribute name="deployDir" default="target/${}"/>
        <copy todir="@{deployDir}">
            <fileset dir="src/main/webapp"/>
        <copy todir="@{deployDir}/WEB-INF/classes">
            <fileset dir="target/classes"/>
        <copy todir="@{deployDir}/WEB-INF/lib" flatten="true">
            <fileset refid="runtime.fileset"/>         

<target name="war-exploded" depends="compile">

<target name="war" depends="war-exploded">       
    <zip destfile="target/${}.war" basedir="target/${}"/>

Note that I don’t even use the Ant “war” task, I just ZIP ;) Maven evangelists would have you believe that Ant leads to un-maintainable, verbose build scripts that are hard to re-use across projects. I call bullsh*t. Look at the Ant “macrodef” usage above which means that I can reuse the WAR routine and do something like this for Tomcat later:

<target name="tomcat-war-exploded" depends="war-exploded">
    <war-exploded deployDir="${tomcat.home}/webapps/${}"/>

What about Ivy?
Whoa, I never thought that this would turn out to be such a long post, let me end by addressing the question: should projects use Ivy instead?

My opinion is NO. Actually, the trigger for this blog post is that I recently joined a project-team already using Ivy and I’m having a hard time adjusting. Anyway, trying to be objective, here are my reasons:

  • One of the good things about Maven is that it enforces the closest thing we have in the Java world to a standard directory layout structure. I find it hilarious that Ivy falls into the same trap as Maven in that they can’t say good things about Maven just like the Maven guys avoid saying good things about Ant. So you won’t find any recommendations to use a standard project structure in the Ivy “best practices” documentation ;) Ivy projects tend to use ad-hoc directory structures, this is what I have observed.
  • Maven obviously has been around longer than Ivy, this translates into more documentation, more users who are used to working (and struggling ;) with it etc.
  • IDE project descriptor creation support: For example, Ivy has no official way to create your Eclipse project descriptor files for you. This itself would be a reason for me to avoid Ivy. There are some ways to manage your Eclipse classpath like IvyDE which honestly doesn’t have much documentation and something like this and this Google code project which don’t look too stable. If you have a Maven pom.xml and Maven installed, just do “mvn eclipse:eclipse” and you can even add “-DdownloadSources=true” – and you can ensure that even things like the “output folders” like “target/classes” and “target/test-classes” – are standard across your team.
  • Needless to say, using the Maven Ant Tasks approach gives you a smooth migration path to Maven. If you do it right, nothing prevents Maven die-hards on your project team from using only the pom.xml and avoiding Ant completely. This is how I have set up the JTrac build for the last 3 years.
  • Maven is positioned as a “project management and comprehension” tool (whatever that means ;) and there is certainly a good ecosystem of plugins that for e.g. can generate your project web-site, run Checkstyle reports, etc.
  • As someone already using Maven for a while, I found it really unsettling that having to use Ivy for another project was resulting in a ton of JAR duplication on my hard disk. And why on earth did the Ivy guys decide to re-invent the structure in which JAR files are stored in the local repo? So you can’t reuse your Maven repo or copy stuff around.
  • Something that the Ivy guys would hate to admit is that ultimately you end up downloading JARs from the Maven public repositories only. So all that talk of doing a better job than Maven kind of sounds flat. People are quick to blame Maven when resolving of dependencies fails to work as expected but this is usually the fault of whoever setup the metadata in the repositories. There’s not much that Ivy can do to solve that problem.
  • I think Maven multi-project support where you keep common stuff in a parent POM and have projects depend on each other is better than how you would do it in Ivy (although I haven’t explored this fully yet).
  • The only thing that appears to be compelling about Ivy is that doing “exclusions” is far less verbose than how you would have it in Maven. But if the Maven guys just fix this one issue

As an example of the kind of thinking that Ivy encourages which I don’t agree with is this quote from a blog post by Dan Allen (Seam committer and author of “Seam In Action”):

But the most absolutely vital feature of Ivy, and what makes it infinitely more useful than Maven 2, is the fact that you can disable transitive dependencies.

I’m outspoken about my position on transitive dependencies. I see them as both evil and a silly device designed for novices (and people with way too much time on their hands). It makes your build non-reproducible and unstable and in the end causes you more work than the work you were attempting to eliminate by moving to Maven 2. This “feature” really is Maven’s boat anchor. Trust me on this one, it’s trivial to define which libraries your application depends on. There really aren’t that many! And you can put all the worrying aside about exclusions.

All I’ll say is that if you disable transitive dependencies, then what is the point. You might as well write a script that downloads a bunch of JAR files listed in a properties file. And about the statement that “there really aren’t that many!” – I really don’t agree. Take Seam itself as an example of a framework that has so many dependencies (and sub dependencies) and expecting developers to keep track of the sub-dependencies (instead of the project-owners who should be updating the Maven repository metadata correctly) – just seems wrong to me.

Did I miss any reasons for not using Ivy? Do let me know in the comments!

Novell bashing

A couple of links:

Interview with Jeremy Allison (part of the Samba team) on on why he quit Novell after the deal with Microsoft.

The online petition against Novell (initiated by Bruce Perens) – which is a good read that clearly explains the scams and legal nonsense behind software patents. Quote below:

Let’s be truthful about software patents: there can be no non-trivial computer program, either proprietary or Free, that does not use methods that are claimed in software patents currently in force and unlicensed for use in that program. There are simply enough patents, on enough fundamental principles, to make this so. If all software patents were enforced fully, the software industry would grind to a halt.

SourceForge having problems again?

Back after a break after attending SpringOne in Belgium and a spot of travel, a company conference etc.

Yesterday noticed that the support forums for JTrac have disappeared. Yikes! Promptly raised a support request on SourceForge.

Today morning, all the SourceForge summary, stats and admin pages were down for a while. Now it is up but the forums are still not there. Fortunately the project web-site and Subversion access have been working all along.
From the “browse support requests” area on SourceForge, it looks like I am not alone. There are folks raising all kinds of scary problems! “My Account Disappeared!”, ” URGENT! Developers removed from project”, etc.

It would be nice if the guys release a mailer or mention something on their blog or the site news page, but nothing so far. I remember a lot of ranting recently about a CVS outage on SourceForge, what is going on?

[Update 2006-06-30: the forums are back *phew* but other things are down intermittently like the stats page. Subversion is fine which is the most important thing I guess.]