07 June 2011

Java EE 6 Server Comparison: JBoss

I never did any serious work with JBoss AS 5 or earlier versions. At the peak of my frustration with Glassfish last summer after the 3.0.1 release when CDI continued to be unusable, I knew that JBoss would be no help as it also included Weld, the main source of trouble. Besides, given the number of problems in released and certified Glassfish, I dreaded an even larger number of problems in an unreleased, uncertified and undocumented JBoss.

When JBoss AS 6.0.0 was finally released late in December 2010, with surprisingly little marketing noise and community echo, I briefly had a look at the release and was disappointed to find out that it was still largely undocumented.


Five months later, there has been no maintenance release, and the documentation is not up-to-date, even the Getting Started Guide still refers to JBoss AS 5.

Anyway, having read Arun Gupta's and Dave Johnson's summaries of their first steps with JBoss, I felt should be able to come to grips with it.

Preparing the server


I copied the MySQL JDBC driver to server/default/lib. To configure my datasource, I copied docs/examples/jca/mysql-xa-ds.xml to server/default/deploy and edited the copy. The most important and non-obvious changes are

<jndi-name>jdbc/myapp</jndi-name>
<use-java-context>false</use-java-context>

All the JBoss datasource examples use a simple JNDI name foo and then include a reference

<jta-data-source>java:/foo</jta-data-source>

in META-INF/persistence.xml. The problem is how to configure a data source with a cross-platform JNDI name that will work on Glassfish, JBoss and Resin alike.

I tried using a resource-ref-name indirection in my web.xml combined with a container specific deployment descriptor, without success. Finally, I found this simple solution with the use-java-context tag in this blog comment.

Running the application


To run my application on JBoss, I copied my WAR to server/default/deploy and then started the server from the command line:

bin/run.sh

Troubleshooting


At first, I all got was a flood of error messages. which I found hard to interpret. For every problem I ran into, it was easy to find similar questions in the JBoss forums or on stackoverflow, but most of the time either without adequate answers, or with solutions referring to JBoss 4 or 5 that did not seem to apply to JBoss 6.

The situation was not helped by the extreme verbosity of the console log messages, and by a sluggish startup. Before starting to deploy my application, or more likely, while scanning the WAR, JBoss takes a deep thought for about 20 seconds with CPU load > 100 %, printing no message whatsoever.

Anyway, most of the problems turned out to be caused by classloader conflicts, which I could solve by excluding a number of libraries from my WAR, but there remained an issue with JNDI lookup exceptions, caused by a known bug in JBoss: Injecting EJBs into the web layer with @Inject results in JNDI lookup exceptions, since JBoss starts the web container before the EJB container by default.

I could not use the producer method workaround mentioned in the JIRA issue, since my web component uses the CDI-to-Spring bridge and is not itself a CDI bean archive. My solution was to create a ServletContextListener class which injects all required session beans with an old-school @EJB annotation. This listener is configured to run before Spring's ContextLoaderListener.

JBoss also complained about an invalid XML namespace definition in my web.xml, and it was right, the namespace was wrong - Glassfish and Resin did not seem to care.

Classloader issues


Unlike Glassfish and Resin, JBoss adopts the Servlet Specification recommendation and does not use a parent-first strategy for class lookups. As a result, there were lots of ClassCastExceptions for org.hibernate.* or javax.* classes included in my WAR. To get rid of these, I excluded all Hibernate, JAXB and Geronimo Spec libraries from my WAR by changing the scope in the Maven POM to provided.

As an alternative, I tried to force JBoss to do a parent-first lookup. The Administration and Configuration Guide does not cover the subject, but following the instructions in an article on JBoss class loading, I added a configuration file WEB-INF/jboss-classloading.xml to my WAR:

<classloading xmlns="urn:jboss:classloading:1.0"
       domain="myapp.war"
       parent-first="true">
</classloading>

This indeed had an effect on classloading, but only produced a new exception

java.lang.NoSuchMethodError: org.objectweb.asm.ClassWriter.(I)V

caused by an older version of asm.jar used by JBoss which is incompatible to the version 3.1 required by my application.

I did not try to upgrade the asm.jar in the JBoss installation for fear of breaking the server, so I ended up excluding the relevant libraries from my WAR, and consequently I had to copy most of these libraries to the appropriate server library directory in Glassfish and Resin to make my WAR portable.

Later, trying to redeploy my application, simply by running

touch server/default/deploy/myapp.war

I ran into OutOfMemoryErrors. A heap dump confirmed my suspicion that some classloaders did not get garbage collected. Taking a closer look with the Eclipse Memory Analyzer, I found a class org.jboss.interceptor.util.InterceptionTypeRegistry holding an indirect reference to an obsolete class loader.

Taking another look at my WAR, I noticed a jboss-interceptor.jar including the javax.interceptor package, a transitive dependency of the CDI-Spring-Bridge. After excluding this library from my WAR, I still had an OutOfMemoryError on redeployment, but I could see in jvisualvm that now at least some classes had been unloaded, so I edited bin/run.conf to increase MaxPermSize to 384MB and the heap size to 1GB. With this modification, I was able to redeploy my application multiple times in a row without any apparent classloader leaks.

So while it is certainly not recommended to include another copy of javax.interceptor in your WAR, this still leaves the question why JBoss loads classes from this package via the WAR classloader and not via its container classloader.

The Web Profile


Unlike Glassfish with its two separate distributions for the Full Profile and the Web Profile, JBoss offers a single distribution including different server configurations, which can be selected by a command line option of the startup script.

JBoss comes with 5 different server configurations. Surprisingly, none of them exactly matches the Web Profile. There is a configuration called jbossweb-standalone which launches a web container but does not include CDI support.

I used the default configuration to launch my application. The default configuration includes the Web Profile but also other services like JMS which are not part of the Web Profile.

See this thread in the JBoss Forum for a more detailed discussion.

Having the JBoss command line options in mind, I once suggested in the Glassfish forum to let users select the desired profile in a similar way. It is now somewhat ironic to find out that JBoss doesn't really let you do that.

Eclipse Integration


Quite some time ago, I made the mistake of installing the entire JBoss Tools set into what must have been Eclipse Galileo, only to find my workspace cluttered almost beyond recognition by tools and menus I had never intended to use.

So this time I took care to only install JBoss AS Tools 2.2.0 from the JBoss Helios Update Site. I was confused (but no longer surprised) by the feature description "Supports JBoss AS 4.x and 5.x". The plugin does support JBoss AS 6 also, at least to some extent.

I was able to launch my application from the workspace using Run on Server. The startup time was between 45-50 s, just the same as from the command line. There was no big difference in the heap size either.

Editing and saving a source file of my top level project should have triggered a redeploy, but didn't. In the Server view, my application still had the status Synchronized. Clicking the Publish button had no effect. Even cleaning the top-level web project did not cause the application to be redeployed.

I also tried reloading a page of my application in the browser, hoping that some hot swap magic might have updated my application without redeploying it.

But no. Stopping and restarting the server seems to be the only way of publishing the current version from Eclipse, so this is at least a critical bug, if not a blocker.

Performance Measurements


  • Empty server startup time: 12 s
  • Empty server memory usage: 100 MB heap, 70 MB PermGen
  • MyApp deployment time: 30 s
  • Server + MyApp restart time: 47 s
  • Server + MyApp memory usage: 236 MB heap, 175 MB PermGen
  • MyApp redeployment time: 30 s
  • Server + MyApp startup time from Eclipse: 47 s
  • Server + MyApp memory usage from Eclipse: 258 MB heap, 181 MB PermGen
  • MyApp redeployment on Eclipse save: failed

Summary


JBoss has taken a long time to finally release a Java EE 6 server with JBoss AS 6.0.0, but quality is worth waiting for, and the server itself appears mature and stable.

This is really the best thing I can say about JBoss. Otherwise, JBoss is too fat, too slow, too verbose, poorly documented and simply not enjoyable to work with.

While Red Hat have pointed out that their business focus is on JBoss AS 7, whereas JBoss AS 6 is "just" a Community release, I'm still somewhat surprised that the JBoss community do not promote their server more actively.

Compared to the first class professional documentation of Glassfish and a whole lot of tutorials and videos on almost each and every aspect of Glassfish, Red Hat and/or the JBoss Community do not appear to have a very active interest in widening their audience, or motivating Glassfish users to give JBoss a try.

For experienced JBoss AS 5 users, upgrading to JBoss AS 6 may be the easiest way to start using Java EE 6 features without losing time on getting familiar with a new server like Glassfish or Resin.

For Java EE newcomers, starting on JBoss AS 6 is probably the hardest route to take, and frustration with the Java EE 6 server may quickly turn into frustration with Java EE 6 as such and may reinforce common prejudice.

The JBoss Eclipse integration is currently not usable for day-to-day development, unless you are ready to shutdown and restart the server instead of simply republishing your application. Even if republishing worked as expected, having to wait 40 seconds for a redeployment is no fun.

All in all, while JBoss may still be the world's number one Java application server in terms of installed instances, its glory seems to have faded. The boss is still around, but the juniors have taken over the business.

Read the whole story


4 comments:

Max said...

I'm interested in seeing your app to know why it wasn't redeploying.

But in any case, try out as7 instead of as6 - your experience will be very different.

Max said...

Ah, if you don't have any descriptors in your project JBoss won't notice it. This sucks agreed.

Solution: use as7 or use clean/full publish or deployed as compressed archive.

MichaƂ Huniewicz said...

Great servers comparison! Thanks for the effort.

JBoss is somehow my favourite JEE server. I'm currently working with version 6 and yes, I know the pain you described, all the trouble one has to go through.

What annoys me most is that even after you get all the configuration right, it still runs out of memory if you redeploy a sufficient amount of times (which is around 5 in my case; I didn't tweak heap settings to be honest).

JBoss 5 had its own problems and was in a way abandoned, most people I know use either 4 or 6. Similar thing may happen to version 6 when 7 is mature.

JBoss still remains my favourite - probably because I know it best, but I may try one of the other two servers.

Craig Ringer said...

I share your pain.

It's reassuring if not nice to see that I'm not alone in my largely miserable and painful experience of developing on Java EE 6 with CDI using Glassfish *or* JBoss.

I've found bugs in Glassfish that I simply cannot believe existed. "Does anybody even use this?" kind of bugs. Like:

GLASSFISH-16917: a giant classloader leak in any app that uses JSF, causing redeploy of medium to large apps to fail in one or two cycles

GLASSFISH-13040: calling methods of a bean that were inherited from a superclass not working

GLASSFISH-1622: asadmin can't set more than one context parameter

and today GLASSFISH-17024: resource reference indirection via web.xml and glassfish-web.xml non-functional in trivial test case.

Then there are things like aren't even considered bugs, like the inability to @Inject into a JPA 2.0 EntityListener (no planned fix in JPA 2.1 unless people start contacting the working group and stressing its importance!). Then there's the need to use a servlet filter and CDI extension like Seam 3 Faces just to get injection into @FacesConverter etc working. These are "how did we actually release the spec with these holes in it" severity issues. What's the point of CDI and dependency injection if it only works in some places some of the time?

CDI was bashed into Java EE 6 too late in the process and it really shows in the lack of integration with the other specs. Even without all the Weld bugs.

I increasingly wonder if anybody actually uses most of the features of these servers. Certainly nobody could've used CDI in production if they wanted to until Weld 1.1.1 and Glassfish 3.1. Even now, I'm having to update to Glassfish 3.1.1 prereleases to work around *more* CDI bugs.

As for JBoss: JBoss AS 6.0.0 has been out for months, and passed release testing. Despite that, it still rejects its own xml schema descriptors for jboss-web.xml. You have to patch it just to use the examples from the documentation without errors!. In JBoss 7, which has just been released, I can't get it to map a datasource name referenced in web.xml via jboss-web.xml .

I guess this is the long way of saying: I share your pain. Java EE is rushing ahead with features and severely neglecting testing, stability, and documentation, leaving things an ugly, buggy mess for those of us who're trying to do real-world work with them.

It also seems like such a huge waste of effort to have three major servers being developed, each of which is incomplete and buggy, instead of consolidating effort.