My new open-source project: Karate

After almost 7 years I’m blogging again. It feels great.

And it is good to be back in the thick of open-source action, founding a brand new project, and actively spreading the word and building a community from scratch.  And the best part is the support I’m getting from my employer – Intuit.

For me personally, this is some of the best hacking I have ever done.  There were times when it seemed like things magically came together. After programming for what feels like decades, the 10,000 Hour Rule feels Very Real to me now.

For those following this blog, I’m likely to be blogging over on Medium from now on.

Do have a look at my first blog post over there, and please do help spread the word !

Here’s the link:

Karate: Web Services Testing Made Simple.

 

Wicket Tutorial: YUI AutoComplete using JSON and Ajax

Getting an AutoComplete JavaScript widget to work with a server-side framework involves a few more steps and integration points than what it would take for e.g. a simple date-picker widget. It makes for an interesting example that shows off the strengths of Apache Wicket when it comes to creating custom components – especially when Ajax and integrating third-party JavaScript and CSS is involved.

This tutorial covers the following topics:

  • Creating a re-usable Wicket custom component
  • How to use a Wicket Ajax “Behavior”
  • Integrating a third party JavaScript widget into a Wicket application
  • Packaging CSS and JS resources needed for the custom component
  • How the required CSS and JS can be contributed to the HTML <HEAD> on demand
  • Hooking into the Wicket Ajax life cycle
  • Returning custom JSON data from the Wicket server-side component

The Yahoo! User Interface Library (YUI) AutoComplete control is our target. Our example uses YUI version 2.7.0b which you can download from the YUI home page. We’ll be using Wicket 1.4 but this particular example should work unchanged in Wicket 1.3.X as well.

Note: The Wicket Extensions project includes an AutoComplete component (see it in action here) which should suffice for most of your AutoComplete needs. The YUI version arguably looks and feels richer (animation and all) and has a whole host of customizable options that are worth looking at.

You need Maven 2 installed. Create a new Wicket project by using the Wicket Maven quick-start archetype which you can find here: http://wicket.apache.org/quickstart.html

You can leave the archetype parameters unchanged, and go with the default “com.mycompany” and “myproject”. Run the Maven command to generate the project structure including quick-start Java code. In the resulting “pom.xml” add the following dependency (within the <dependencies> section) so that we can use the Jackson JSON processor library for converting Java data into JSON.

<dependency>
    <groupId>org.codehaus.jackson</groupId>
    <artifactId>jackson-mapper-asl</artifactId>
    <version>1.2.0</version>
</dependency>

Using something like Jackson means that you don’t have to worry about forming JSON by hand and dealing with things such as nesting, escaping characters etc.

Now you can import the project into your IDE. Instructions pertaining to your IDE of choice are available at the Wicket quick-start page itself. Our AutoComplete use case is to load a list of country names matching the user’s input. We’ll use the java.util.Locale class as a quick and easy data source. Create a new Java class in a “com.mycompany.util” package as follows:

public class LocaleUtils {

    private static final Set<String> countries;

    static {
        final Locale[] locales = Locale.getAvailableLocales();
        countries = new TreeSet<String>();
        for(final Locale locale : locales) {
            countries.add(locale.getDisplayCountry());
        }
    }

    public static String[] getCountryNamesMatching(String query) {
        List<String> list = new ArrayList<String>();
        for (final String country : countries) {
            if (country.toUpperCase().startsWith(query.toUpperCase())) {
                list.add(country);
            }
        }
        return list.toArray(new String[list.size()]);
    }

}

And let’s also get this other utility class out of the way before we get to the really interesting stuff. Here’s the listing of “com.mycompany.util.JsonUtils” which uses Jackson to convert Java data into JSON:

public class JsonUtils {

    private static final JsonFactory jf = new JsonFactory();

    public static String marshal(Object o) {
        StringWriter sw = new StringWriter();
        try {
            JsonGenerator gen = jf.createJsonGenerator(sw);
            new ObjectMapper().writeValue(gen, o);
            return sw.toString();
        } catch(Exception e) {
            throw new RuntimeException(e);
        }
    }

}

By looking at the simple “Basic Local Data” YUI AutoComplete sample, we can figure that the following YUI modules are needed as dependencies for the AutoComplete control:

  • yahoo-dom-event
  • animation
  • datasource
  • autocomplete

We need to copy these modules containing JS and CSS resources from the YUI downloaded distribution into our project structure. First create a “com.mycompany.yui” package where our YUI Wicket components will live. Under the “yui” folder on the file system, create a new folder called “res” (resources). Copy the folders listed above from the “yui/build” folder into the “res” folder. You should end up with something like this:

directory-structure

The JavaScript API documentation of the YAHOO.widget.AutoComplete component says that it requires a YAHOO.util.DataSource instance to work. For example, a LocalDataSource can be used when the data is available as an in-memory JavaScript array or JSON object. We will create a new JavaScript component that extends LocalDataSource and override the “makeConnection” method to get the data from a Wicket component over Ajax. Details of how to go about implementing “makeConnection” can be figured out by looking at the base method in “res/datasource/datasource-debug.js” – which is excerpted below for completeness:

// begin excerpt from YUI code
makeConnection : function(oRequest, oCallback, oCaller) {
    var tId = DS._nTransactionId++;
    this.fireEvent("requestEvent", {tId: tId, request: oRequest, callback: oCallback, caller: oCaller});
    var oRawResponse = this.liveData;    
    this.handleResponse(oRequest, oRawResponse, oCallback, oCaller, tId);
    return tId;
},
// end excerpt

In the code taken from YUI above, “oRequest” is the string the user has typed into the input text field. So it looks like after fetching the corresponding data, we need to call the “handleResponse” method to complete the flow, and the data is passed on as the “oRawResponse” method argument.

Here is the code we end up with for “WicketDataSource” which extends “LocalDataSource” the JavaScript “prototype” way and all. We re-use the “YAHOO.widget” namespace to keep things clean and reduce the risk of JavaScript name collisions:

YAHOO.widget.WicketDataSource = function(callbackUrl) {
    this.callbackUrl = callbackUrl;
    this.responseArray = [];
    this.transactionId = 0;
};

YAHOO.widget.WicketDataSource.prototype = new YAHOO.util.LocalDataSource();

YAHOO.widget.WicketDataSource.prototype.makeConnection = function(oRequest, oCallback, oCaller) {
    var tId = this.transactionId++;
    this.fireEvent("requestEvent", {tId: tId, request: oRequest, callback: oCallback, caller: oCaller});
    var _this = this;
    var onWicketSuccessFn = function() {
        _this.handleResponse(oRequest, _this.responseArray, oCallback, oCaller, tId);
    };    
    wicketAjaxGet(this.callbackUrl + '&q=' + oRequest, onWicketSuccessFn);
};

Our implementation (override) of the “makeConnection” method uses the Wicket Ajax engine to GET data from a given URL and we pass a callback function that will call “handleResponse” once the response from the server is received. The value typed by the user is passed as as a query-string parameter “q” which will be read on the server-side.

Update: for more details on the lightweight Ajax engine and JavaScript utilities that come along with Wicket, refer this article by Nino Martinez: Wicket Javascript Internals dissected. The script.aculo.us “Drag & Drop ListEditor” example by Al Maw is another useful resource and you can get the slides and code from this page.

We could have used YUI itself to make the XMLHttpRequest but the nice thing about using the Wicket Ajax engine is that you can spy on all the action using the very nifty Ajax Debug Window in “development mode” like this:

ajax-debug-window

You may be now wondering how “_this.responseArray” gets initialized once the server request completes. The “YAHOO.widget.WicketAutoComplete” (below) is a JavaScript convenience object we define to represent and wrap a single AutoComplete instance (along with the DataSource):

YAHOO.widget.WicketAutoComplete = function(inputId, callbackUrl, containerId) {
    this.dataSource = new YAHOO.widget.WicketDataSource(callbackUrl);
    this.autoComplete = new YAHOO.widget.AutoComplete(inputId, containerId, this.dataSource);
    this.autoComplete.prehighlightClassName = "yui-ac-prehighlight";
    this.autoComplete.useShadow = true;
    this.autoComplete.formatResult = function(oResultData, sQuery, sResultMatch) {
        return oResultData;
    };      
};

So if an instance of “WicketAutoComplete” has been assigned to a JavaScript variable “foo”, the responseArray can be initialized by assigning something to “foo. dataSource. responseArray”. We will take care of this in the Java code of the Wicket component, coming up shortly.

The YUI AutoComplete expects data received from the DataSource to be a JavaScript array of items. The customized “formatResult” method override above just displays each item as-is. You could implement more sophisticated routines such as bold-ing the part that matches the user input, and you can refer the YUI examples for more details. For now, we are keeping it simple, but in the future you may want to consider things like returning a multi-dimensional array or an array of JSON objects etc. from the server.

Our “WicketAutoComplete” JavaScript object constructor takes the following arguments:

  • inputId: HTML id of the input text field (needed by the YUI AutoComplete control)
  • callbackUrl: URL where the Wicket component will listen for Ajax requests, used by our custom WicketDataSource
  • containerId: HTML id of the <DIV> where the autocomplete results will be rendered (needed by the YUI AutoComplete control)

Take the two javascript listings above (“WicketDataSource” and “WicketAutoComplete”) and combine them into a single file called “YuiAutoComplete.js” within the “src/main/java/com/mycompany/yui” folder (or rather the “com.mycompany.yui” package). In the same folder, create “YuiAutoComplete.html” and “YuiAutoComplete.css” as follows:

<wicket:panel>
    <div class="yui-skin-sam">
        <div class="wicket-autocomplete">
            <input wicket:id="text"/>
            <div wicket:id="container"></div>
        </div>
    </div>
</wicket:panel>
.wicket-autocomplete { width: 15em; padding-bottom: 2em; }
.wicket-autocomplete div { font-size: 90% }

The HTML and CSS files are pretty straightforward. Setting the CSS “width” of an enclosing <DIV> is the recommended way of controlling the rendered size of a YUI AutoComplete control. The YUI CSS “skin” reference “yui-skin-sam” can be even kept at the <BODY> tag level (like in the YUI examples) but here we chose to have it self contained within our component.

And finally here is the code of “com.mycompany.yui.YuiAutoComplete” that ties everything together. A description of what is going on appears below the code listing:

public abstract class YuiAutoComplete extends FormComponentPanel {

    private TextField textField;
    private WebMarkupContainer container;

    public YuiAutoComplete(String id, IModel model) {
        super(id);
        textField = new TextField("text", model);
        textField.setOutputMarkupId(true);
        add(textField);
        container = new WebMarkupContainer("container");
        container.setOutputMarkupId(true);
        add(container);
        add(new YuiAutoCompleteBehavior());
    }

    @Override
    public void updateModel() {
        textField.updateModel();
    }

    private String getJsVarName() {
        return "YAHOO.widget." + textField.getMarkupId();
    }

    protected abstract String[] getChoices(String query);

    private class YuiAutoCompleteBehavior extends AbstractDefaultAjaxBehavior {

        @Override
        public void renderHead(IHeaderResponse response) {
            super.renderHead(response);
            response.renderJavascriptReference(new JavascriptResourceReference(YuiAutoComplete.class, "res/yahoo-dom-event/yahoo-dom-event.js"));
            response.renderJavascriptReference(new JavascriptResourceReference(YuiAutoComplete.class, "res/animation/animation-min.js"));
            response.renderJavascriptReference(new JavascriptResourceReference(YuiAutoComplete.class, "res/datasource/datasource-min.js"));
            response.renderJavascriptReference(new JavascriptResourceReference(YuiAutoComplete.class, "res/autocomplete/autocomplete-min.js"));
            response.renderJavascriptReference(new JavascriptResourceReference(YuiAutoComplete.class, "YuiAutoComplete.js"));
            response.renderCSSReference(new CompressedResourceReference(YuiAutoComplete.class, "YuiAutoComplete.css"));
            response.renderCSSReference(new CompressedResourceReference(YuiAutoComplete.class, "res/autocomplete/assets/skins/sam/autocomplete.css"));
            response.renderJavascript("var " + getJsVarName() + ";", getJsVarName());
            response.renderOnDomReadyJavascript(getJsVarName() + " = new YAHOO.widget.WicketAutoComplete('"
                    + textField.getMarkupId() + "', '" + getCallbackUrl() + "', '" + container.getMarkupId() + "');");
        }

        @Override
        protected void respond(AjaxRequestTarget target) {
            String query = getRequest().getParameter("q");
            String[] result = getChoices(query);
            String jsonResult = JsonUtils.marshal(result);
            target.appendJavascript(getJsVarName() + ".dataSource.responseArray = " + jsonResult + ";");
        }

    }

}
  • #01, #18: we extend FormComponentPanel instead of Panel so that we can seamlessly use our YuiAutoComplete component within Form instances
  • #09, #12: we ensure that HTML id-s are rendered and we pass them to the YUI AutoComplete control later
  • #14: automatically add the Ajax wicket behavior (explained below) to our custom component
  • #26: we define a method that subclasses can override to return data corresponding to user input
  • #28: we extend AbstractDefaultAjaxBehavior and we only need to override the “respond(AjaxRequestTarget)” method
  • #33 – 39: we ensure that the required JS and CSS files are contributed to the HTML <HEAD> of any page that contains our custom component. Tip: use “debug” versions of the JS files for e.g. “autocomplete-debug.js” instead of “autocomplete-min.js” to make debugging using FireBug easier. You probably don’t need the “debug” versions for a “production” version of this component. The good thing about loading resources like this from the package / classpath is that you can simply JAR it all up for use by other teams.
  • #40: we output a variable declaration (global to the page) to hold a reference to our “WicketAutoComplete” JavaScript object – which can be referenced later during the Ajax request for passing data to the web page, we derive the name from the HTML id of the input TextField (#23)
  • #41: ensure that an instance of our WicketAutoComplete JavaScript object is instantiated (and assigned to the variable described above) when the page loads, note how the HTML id-s and Ajax “callback” URL are determined
  • #47: get the value of the parameter “q” in the incoming Ajax request
  • #50: ask Wicket to execute some JavaScript when the request completes which will (pass) assign a JSON array value back via our “WicketAutoComplete” object reference

Now the project within your IDE should look like this:

project-structure

Edit the “HomePage.html” and “HomePage.java” files to use an instance of our Wicketized AutoComplete component as follows:

<html>
    <head>  
        <title>YUI Autocomplete Demo</title>
    </head>
    <body>
        Enter Country Name: <span wicket:id="autocomplete"></span>
    </body>
</html>
public class HomePage extends WebPage {

    public HomePage() {        
        add(new YuiAutoComplete("autocomplete", new Model("")) {
            @Override
            protected String[] getChoices(String query) {
                return LocaleUtils.getCountryNamesMatching(query);
            }
        });
    }

}

Note how we have implemented the “getChoices()” method to return a list of country names in this case. That should be it! Start Jetty by using the “com.mycompany.Start” class that should be under the “test” source folder structure. Browse to http://localhost:8080/ and try out your shiny new Wicket + YUI AutoComplete component.

I’ve uploaded the source code here: myproject.pdf – just rename this to a *.zip after downloading. Do comment if you spot any mistakes or stuff that can be improved.