I build up a Clojure stack for web application development.

First Results can be found here: https://github.com/kremers/Clojure-Memcached-Connector

Clojure + Sandbar + Enlive +  spymemcached on membase server

Membase vs. other K,V Stores

• Not Master,Slave! Peer Replication
• Fastest, Supported and Tested Platform
• Also available at Heroku (5mb for free)

Enlive vs Hiccup

• Pure DRY code should be separated from the page markup.
• HTML should be semantic markup. CSS should work for the design.

• Sandbar creates stateful sessions in memory, but we want to scale. Let’s store the sessions in our k,v store. So we need an implementation of ring / ring-core / src / ring / middleware / session / store.clj for memcached

Logging

• Finally: Simple logging (Category, Message) to memcached, quota with flush to file

The first time I had contact with Clojure was in 2009, while writing my final piece of coursework for my  degree on dynamic programming languages in big projects, focusing on Groovy. It was a bit hard for me to get to grips with Clojure, because the learning curve is quiet different to that of Groovy (which is very shallow if you come from a Java background). One of Groovy’s painful aspects appeared during performance tests. Possibly because the language hasn’t got the same academic background as LISP, Haskell or C++.

Why I have not choosen Scala 

I tried Scala because I excepted it to be more academic and exact. A major reason was that I have been impressed by Martin Odersky working on Scala. My first functional attempts at Scala were unsuccessful, due to the lack of JVM TCO. It may be due to my lack of experience that the Scala code seemed to look like “write only” Perl code. Programming Scala felt more like writing in Java, but with a lot more capabilities and primarily more complexity. The language is really hard to understand, even the complexity of its type system  is astounding. I’ve been quite happy using this blog post from Stephen Colebourne to help me. I had some of the same issues after some experiments with Scala. After seeing the answer from Martin Odersky I was a bit disappointed by how he deals with criticism. Blaming people who are investing time to point out weaknesses of a language and its problems to spread FUD.

How I started learning Clojure

Getting into a complete new environment with its own paradigms is hard. Especially if you can not reuse your knowledge from imperative / procedural and object oriented programming languages. Having seen a few real world Clojure applications, I was able to get an understanding of its Java interoparability. With the help of tools like “leiningen” it was easy to build my first applications.

Why I except it to be worth a try

Clojures built-in functionality to deal with the lack of TCO in the JVM (“recur, lazy sequences, …”). I searched a long time in the Scala API for such a feature. It’s not just a wide mix of object orientation with a few functional concepts, which feels like an advantage to me. The community is also adopting good libraries like these for website templates  (Enlive, …).

Where to go from here

I will check out the samples for Heroku and Clojure, since Heroku is one of the most popular cloud providers and all of the infrastructure management is done for you. And now they are properly supporting Clojure. There are also goals to play around with the language such as the “petshop” and examples with database / NoSQL connectivity. Once I have tried writing these things myself, I’ll write another blog entry on it. One of my goals is to ensure that Clojure can be used in bigger web projects. I’m not sure Clojure should be recommended for such business projects yet.

[0] http://clojure.org/

[1] http://stackoverflow.com/questions/310974/what-is-tail-call-optimization

[2] http://blog.joda.org/2011/11/scala-feels-like-ejb-2-and-other.html

[3] http://entwickler.com/itr/news/psecom,id,58955,nodeid,82.html

[4] http://en.wikipedia.org/wiki/Write-only_language

[5] http://de.wikipedia.org/wiki/Fear,_Uncertainty_and_Doubt

[6] https://github.com/technomancy/leiningen

[7] https://github.com/cgrand/enlive/wiki/

[8] http://blog.heroku.com/archives/2011/7/5/clojure_on_heroku/

[9] http://thecomputersarewinning.com/post/clojure-heroku-noir-mongo

The JEE Ecosystem got bigger and bigger due to the increasing market impact of Java. More and more frameworks were integrated into JEE. After JEE5, JSF became also a part of the standard set of frameworks. Most mistakes are made when deciding whether to choose JSF or not. SUN Microsystems defined JSF as “an API for representing UI components and managing their state”. So one of the primary goals was to build server-side user interfaces which react to events (like mouse-clicks) and manage states (e.g. caching form entries after leaving a dialogue). This leads to a Rich Internet Application (RIA) which should behave like a desktop application. Now two paradigms are connected to each other, and they do not really fit. An HTML Website sent over HTTP can not carry a state by default, it can not trigger events in a direct way and it is not possible to interact with the operating system (everything is running in a browser thread which isolates the website). Finally, the idea of JSF is to provide an event-driven architecture like that of desktop applications based on request-response driven distributed model of websites. To deal with the issues, JSF makes use of cookies, javascript, JSF User Sessions and a so called JSF component tree which represents the state of all components. The user session is saved in the RAM of the application server for every user until a timeout is reached. The “view” of the user also contains a component model which is synchronized with the session on the application server to reach the “sicky state” behaviour. If you descide to connect the two worlds (web application and desktop application) you have to live with the consequences of such a model. This may lead to different workarounds (for example working with bookmarks is difficult in such an architecture).

JSF only feels good, if it is used as it’s meant to be. I.e. for Internet Applications which behave like desktop applications. 

Most poeple, companies or software architects choose JSF because it is part of the JEE standard. They assume that it is the industry standard and that it cannot be wrong to use it, regardless of the functional and non-functional requirements. There are several non-technically-related shaky arguments for such a decision. Like “enterprise support, a ‘proven’ solution, included in standard” which sometimes leads to self-protection of decision makers in case of a project failure. This does not minimize the project risk. Quite the opposite, it leads to shift the liability onto the software developers, by forcing them to think in paradigms that don’t fit the customer requirements. Other circumstances can also lead to similar sub-optimal decisions. For example lack of experience, fear of getting in touch with new technology or third-party regulations when choosing a framework. Also, having experience and expertise with a single product or technology can result in the assumption that this is the “silver-bullet”, meaning comparisons between the different frameworks and technologies are skipped. Result: The aim is the ability to deliver something, rather than aiming for quality and maintainability.

Decisions based solely on self-protection are really risky. 

What else is omitted from the rose tinted articles about JSF? One thing is the Post-Back Pattern: JSF is based on the so called “post back architecture”. This is how it works: 1. A page is loaded. 2. All requests are now sent with HTTP-POST. 3. The client recieves the result and processes it with javascript. Steps 2 & 3 are repeated until the client leaves the page. After the client leaves, the session is still available until the JSF session timeout (and longer if resources are bound to the JSF session). This has several disadvantages:

A lot of sub-optimal design decisions have been made in the JSF architecture.

JSF is a really complex framework, because a lot of things are affected under the hood. The complex six step phase-model is an example of how to handle states (see: step diagram). The complexity is not the only issue; cpu-time for processing the steps for every request and the RAM which holds the user sessions for each synch in the component tree must also be considered. Java-Script was a must have in JSF 1.0. After JSF 2 many things changed, for example graceful degradation was implemented to be more fault tolerant to users with java script limitations. Due to JSF’s complexity and because it is so expensive to fulfill simple requirements, frameworks like JBoss Seam are getting more and more popular. The goal of such frameworks is to eliminate complexity at architecture and API levels. If you are forced to work with JSF, JBoss Seam is one of the best solutions to get out of the complexity, though JSF is still running behind the scenes, which affects application speed and scalability.

Compared to other frameworks JSF is complex, CPU and RAM intensive.

Many business applications can be classified as “data capture / store and modify / CRUD” software. High Performance is not an issue here and a large part of the user interface consists of reusable components like input forms & text fields. Component Libraries such as IceFaces and MyFaces come into play. In this context the concept of a “plug and play” component solution works. Such applications can have complex data input tables and validation rules, resulting in large, complex calculations, but the design of the page is simple. On the other side are websites with a lot of design and content. If you are forced to implement such things with JSF you’ll end up with a bloated codebase, which is hard to maintain.

There are domains where JSF adds value.

However other people say that JSF is the silver-bullet and that all problems can be solved with JSF?! You should not trust these people! Most of the stuff that you can find (online articles, java-magazine, …) are from people who clearly recommend JSF. Most of them are biased because they work with JSF every day and their company advises customers to use it. So why lower their own worth on the market? They will not write about the workarounds in their software solutions, the bad experiences they had once JSF was in place and all the scalability problems. A long time ago I bought the “Java Magazine”, which is mostly full of articles from consultants to boost their own value. The quality varies a lot from article to article. The main article was called “Web Frameworks, Struts, JSF and ZK compared”. The author talks about JSF as a more modern framework than Struts. I think the paradigms of event-driven and request-response driven architectures are incomparable. Problems with the component tree or real world experience with big projects were not mentioned. I know from projects where custom JSF user-session recycle services have been developed to prevent memory overflows with only a few hundred concurrent users.

Trust yourself! Evaluate different frameworks. Read about technical details and do not trust the hype a technology is given.

So, what is the next big thing for the future? After the growing market impact of javascript due to application development for Apple IOS and Android, architectures like SOFEA have good chances in the future. The biggest problem is that there is no standard stack defined which brings all of the necessary javascript frameworks together. Until then it is better to stick with the Spring (MVC) stack or Grails.

MphasiS, an HP Company, has also done evaluations of JSF for big, complex projects. The results reflect my experience. Check out this excellent article by Mohan Borwankar. Click here to read the Article of Mohan Borwankar. 

You shold also check out this presentation from reliabledesigns.

Identity templates are arguably one of the most important things you should know about XSLT. Identity templates are really useful if you want to change only a part of the XML input data. The idea is not to create a explicit node copy of each item. The output of the identity transform is the same as the input,  except for CDATA elements and attribute orders. Additional templates are created to define the exceptions for the matched elements which should be modified. This is mandatory if:

1. The Interface / XML structure changes
2. Output enriched with rule results in additional nodes
3. Non-redundant template code is a must-have

Example of a XSLT2 Identity Transform template:

<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="@*|*|processing-instruction()|comment()">
<xsl:copy>
<xsl:apply-templates
select="*|@*|text()|processing-instruction()|comment()"/>
</xsl:copy>
</xsl:template>
</xsl:stylesheet>

From now on, other templates can match nodes which should be modified. For example we wish to remove all nodes with the name “Bill”:

<xsl:template match="Bill"/>

Because the template is empty, the node is removed in the output.

If multiple transformations are applied, XML pipelines are better (XProc, …). Projects under active development include QuiXProc and Tabular.

Here I will show you how XSLT2 can be used in a Adobe LiveCycle / ADEP process. I found solutions where people put a saxon.jar into their application server library path. The whole application on the server uses the alternate XSLT processor, if you put it to the application server classpath. A test with JBoss resulted in errors which occurred during runtime (like the content of a modified process can not be saved). The working alternative is a custom plugin (called DSC). The classloader of the DSC component is isolated from the environment, so you won’t get into support problems (except for the DSC itself). I’ve choosen saxon as the XSLT2 processor (HE edition). The edition can simply be changed by putting replacing the saxon library in the jar file. You can download the DSC here.

I solved the issues regarding the ‘com.sun.xml.ws:jaxws-rt:jar:2.1.4′ with a google search. I’ve found threads from people with the same problem, but they used maven. Maven only feels really good if you are building unspecific things like the well known “petshop”.

1. With Gradle you can escape from all the xml junk.
2. Gradle also supports Apache Ivy.
3. Tool support from Gradle gets better (eclipse toolset, …).
4. Moving from maven to gradle using maven integration, you’ll not get rid of the problems maven causes (for example incorrectly defined pom.xml ressources in the maven repository).
5. Using Gradle results in freedom because it solves major issues such as: The “central repository” way (your problem if you can’t find your dependencies or licensing issues apply to a few jar files). You get rid of the “forced convention” and “forced workflow” style (writing your own scripts and plugins in maven is not recommended).
6. Gradle is a step away from the monolithic way of building software (build results in a system) to a more heterogenous style (build results in a module / service).
7. Gradle can split the project definition into different files (the include mechanism).

In the last few years I have had to deal with complex systems and their issues like the C10K Problem, hard to scale infrastructures, asynchronous transaction handling in heterogeneous environments and above all else – Database performance issues. Simple solutions are available for most bottlenecks, but deciding how and where to place the state in an distributed environment can be really hard.

Google introduced Map Reduce, which offers a kind of standardization. The basic idea of divide and conquer (D&C) has existed for years, but there were no standardized APIs to execute higher level tasks. Google invented its own filesystem (GFS), in which data is stored in a distributed system and in memory for performance benefits. They also implemented an interface for the distributed storage called Bigtable which uses MapReduce for query entries. This is also built upon the Google File System.

Several groups of NoSQL Databases emerged. They followed the principle of Bigtable with eventually consistent and distributed storage. The concept behind an in-memory storage and distributed system is a trade-off between consistency and availability. The aim of this is to enforce strong consistence only where it’s required. Such Projects include Project Voldemort (Persistent),  Hazelcast (Non-Persistent) and Scalaris (Non-Persistent).

When we have to work without SQL, a number of other problems arise; How do we build a search system? How do we execute complex queries? A big subset of the issues can be covered by merge-joins. I really recommend taking a look at: Building Scalable, Complex Apps on App Engine.

What I’m going to cover in this sample implementation:

1. JAX-WS 2.2 Standard conform implementation
2. Spring 3.0.5 Integration due SpringBeanAutowiringSupport
3. Generic JPA 2.0 DAO with Hibernate as persistence provider
4. Gradle 1.x as Build tool
5. MTOM Transport for medium-size attachments
6. Generic SLF4J logging with jcl and jul to log4j
7. JUnit Testing with Spring Integration
8. Generic Class for Services due DRY principles

Download the example here.

Run the example using: gradle jettyRun on the console. You should see something like:
INFO  - <In-Memory Content Repository started...>
INFO  - <Product Service started...>
INFO  - <Product Service WS-* Endpoint started...>

The sample contains a simple domain with a few objects (Category, Product and Item) where Item is an generic byte[] abstraction item to store contents in a JCR compliant repository like Apache Jackrabbit. In the example it’s just in a Memory table to store Binary attachments. The Repository can be changed by implementing Repository<byte[]> with custom functionality.

Test the sample using SoapUI:

Ensure that Enable MTOM and Force MTOM are enabled. If other transports are available, SoapUI only uses MTOM if the attachment exceeds a specified size. Enforce this transport model to test the webservice.

The services are deployed at the following URL: “http://localhost:8080/shoppingEngine/services/product” . Append “?wsdl” to read the WSDL (during “add project” in SoapUI).

A while ago I tried to find an alternative piece of modelling software for projects where M$-Visio is not the project standard and I’m free to choose the modelling software I use.  Four years ago I found YED and I’ve never looked back. The latest versions support BPMN Graphs and can be used to model topologic network plans, a variety of UML Diagrams and much more.

It’s worth a try. Check out the Examples.

Since I’ve done a lot of stuff with Ant and worked in many projects where maven was the weapon of choice, project sizes have  differed massively. As a result I’ve seen a lot of  different configurations, varying from simple standalone applications to complex multi-module projects. Maven looks really suitable for simple projects, or projects where everything should fit into one jar/war file. The problems begin with multi-module Management. Cross-References are not resolved the right way, the build is slow during cyclic references and cache and index reloading causes trouble. If you are using your own repository manager (Nexus, for example), the index and caching problems become much more significant. There are also a few issues with a central settings.xml where version numbers can be stored. Values are not resolved and the behaviour is not reproducible. Scary! The poorly managed primary / third-party repositories are also problematic. They force you to use excludes and old library versions if you don’t use a repository manager. The biggest problem with maven, however, is that it forces you to deal with the given project structure and the build life-cycle. There are a lot workarounds you can use (like running ant tasks in maven), but none of them feel right. Features like build automation and automatic deployment are supported but due the complexity of enterprise systems, you’ll end up with a bundle of ant tasks in the XML structure of the pom.xml. A few people also complain about XML as being the wrong language to write the build file in, but since I’ve written a lot of spring-wiring bean files, xml feels wordy, but it’s not a problem for me. So I tried to migrate the pom.xml of some of my projects to gradle. It worked fine, aside from the bad IDE support from Eclipse/STS. The resulting build file is about fifteen times smaller and looks pretty readable. WAR bundling and the Jetty task to run the application worked fine. The only problem I discovered: The dependency ‘com.sun.xml.ws:jaxws-rt:jar:2.1.4′ does not resolve its necessary dependencies. I fixed this issue by downloading the distribution and put them into my “lib” folder. Maybe I’ll try to fix it and show my experiments.

JPA 2.0 offers a lot of nice features such as support for extended maps, collection mappings, automatic removal of related objects (orphan), configurable pessimistic locking and caching. But in my opinion, there are also downsides, such as the bloating CriteriaQuery which turns a simple JPQL query into a six line monster. The first time I had contact with JPA #2 was while reading IBM Developer networks “Dynamic, typesave queries in JPA 2.0″. Since there are already such great articles like “Don’t repeat the DAO” out there, I was a little confused about the focus of the article from IBM. Typesafety is great, if it’s applied in a generic way! The advantages and drawbacks of so much more code? Due raw type usage / type erasure no runtime checking for the type is possible. Secure / Deterministic refactoring may be to blame. Viewing type problems at runtime shouldn’t be an issue, since the application is being tested properly. Most IDEs offer features for String lookups during refactoring which can help with the refactoring issues. JPQL Queries tend to have a readable structure. The CriteriaQuery API looks like the opposite is the case if the query is a more complex one. In my opinion the price for such a “typesave” Query is too high; while the JPQL Query is based upon SQL and offers readability, the Criteria API looks verbose and bloated. I prefer to write good old JPQL Queries.