Sunday 20 December 2009

Exception hell

All Java programmers have certainly encountered their fare share of excessively long, incomprehensible or just plain stupid stacktraces, but this one got me stumped when I saw it:

08:34:55,989 ERROR [jsp:165] java.lang.ClassCastException: com.icesoft.faces.context.ElementController cannot be cast to com.icesoft.faces.context.ElementController
at com.icesoft.faces.context.ElementController.from(ElementController.java:22)
at com.icesoft.faces.context.DOMResponseWriter.enhanceBody(DOMResponseWriter.java:294)
at com.icesoft.faces.context.DOMResponseWriter.enhanceAndFixDocument(DOMResponseWriter.java:239)
at com.icesoft.faces.context.DOMResponseWriter.endDocument(DOMResponseWriter.java:144)
at com.icesoft.faces.facelets.D2DFaceletViewHandler.renderResponse(D2DFaceletViewHandler.java:283)
at com.icesoft.faces.application.D2DViewHandler.renderView(D2DViewHandler.java:161)
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:107)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:268)
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:137)
at com.icesoft.faces.webapp.http.core.JsfLifecycleExecutor.apply(JsfLifecycleExecutor.java:18)
at com.icesoft.faces.webapp.http.core.PageServer$1.respond(PageServer.java:25)
at com.icesoft.faces.webapp.http.servlet.ServletRequestResponse.respondWith(ServletRequestResponse.java:161)
at com.icesoft.faces.webapp.http.servlet.ThreadBlockingAdaptingServlet$ThreadBlockingRequestResponse.respondWith(ThreadBlockingAdaptingServlet.java:36)
at com.icesoft.faces.webapp.http.core.PageServer.service(PageServer.java:30)
at com.icesoft.faces.webapp.http.core.MultiViewServer.service(MultiViewServer.java:56)
at com.icesoft.faces.webapp.http.common.standard.PathDispatcherServer$Matcher.serviceOnMatch(PathDispatcherServer.java:50)
at com.icesoft.faces.webapp.http.common.standard.PathDispatcherServer.service(PathDispatcherServer.java:19)
at com.icesoft.faces.webapp.http.servlet.ThreadBlockingAdaptingServlet.service(ThreadBlockingAdaptingServlet.java:19)
at com.icesoft.faces.webapp.http.servlet.EnvironmentAdaptingServlet.service(EnvironmentAdaptingServlet.java:29)
at com.icesoft.faces.webapp.http.servlet.MainSessionBoundServlet.service(MainSessionBoundServlet.java:106)
Unless my eyesight has really gone, this exception is saying it can't cast ElementController to ElementController? Same name, same package, as per usual it's just a big bag of fail again. Why can't I just get a normal comprehensible exception and stacktrace for once?

This one is bad, but there is one that is much worse, the dreaded NoClassDefFoundError, but that one deserves its own post somewhere after new year.

Monday 7 December 2009

Liferay installation problem: Could not find the main class: . Program will exit.

A few weeks ago I was installing Liferay at a client and ran into a very weird installation problem. I was under the impression that I'd done everything correctly, but when I tried to start Liferay I got a baffling error message almost immediatly:

"Could not find the main class: . Program will exit."

As Eddie Izzard would say: Quod The Fuck! How can a simple Liferay install go this wrong? I followed some simple steps to install it:
  • download Liferay
  • transfer ZIP file to server
  • unzip the downloaded file
  • check installed Java version
  • check DB connection
  • configure DB connection in portal-ext.properties
  • start Liferay
By now I've done this simple procedure more than a few times and it never failed, ... until now. So I was kinda gobsmacked. I started digging around, but couldn't find an obvious cause. Also Googling didn't turn up any leads. So I was on my own and decided to start from the beginning and check every line in the Tomcat start script and quickly found that strange things were happening with the startup classpath, hence the empty main class name. A JAR file seemed to be missing: bootstrap.jar.

How could that be possible. I downloaded a fresh Liferay ZIP that unzipped without problems. Then I remembered a long pause during the download around the same time the datacenter people were fiddling with the firewall. A second fresh download later, this time without hickups, I was able to run a file size check against the old download and what do you think: the first download was 20Mb smaller.

How it was able to unzip with throwing obvious errors I still don't know, but after deleting the previous install, unzipping the new one, Liferay started without any problem.

Email containing cid

Did you ever receive an email containing a line that looks like this:

[cid:163091419@01122009-2514]

This probably means that something, in my case an image, is attached to the mail or embedded in it, but the email client, isn't able to process it correctly. There's an easy way to still visualize it.

Just open the email source and find the correct mail part, this should look something like this:
------=_Part_42520_1491296619.1259746973647
Content-Type: image/png; name="Picture 9.png"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="Picture 9.png"
Once you've found this you'll know what the type and encoding of the missing attachment is. Now you'll just need to select the block of encoded text, drop it in a suitable decoder, save the result as a binary file and hey presto!

Wednesday 25 November 2009

Can't touch this!

Last week I was working on some problem when I got the following output in the console:

touch: cannot touch

So apparently the script I was running was trying to touch a file and was having problems with it. This usually means that the file isn't where it is supposed to be, but just in case I still Googled it and starting from the 3rd result things got funny:


Monday 2 November 2009

Liferay Password Policies with a regular expression sauce

For a Liferay project that I'm working on there are a bunch of security requirements. One of those is that a user password has to conform to a certain set of rules. Because Liferay is pretty customizable, changing the password policy shouldn't be a problem.

Looking around the Control Board already reveals some possibilities hidden in the Portal > Password Policies menu. Here you can tweak the default password policy or even create a new one. There are several things to customize on this page:
  • Changeable/Change required: can the user change his/her password and is a change required after the first login
  • Password Syntax Checking: (dis)allow passwords that are common dictionary words and set a minimum password length
  • Password History: disallow passwords according to a password history of a configurable length
  • Password Expiration: how long is a password valid before it needs to be changed
  • Lockout: how many times can someone try to log in correctly before his/her account gets locked and how does it get unlocked
While this is already a nice feature set, the password syntax checking isn't quite enough to satisfy our password requirements. Checking the source code and the default portal.properties file reveals that it is possible configure a custom passwords.toolkit. This can be one you create yourself and that extends from com.liferay.portal.security.pwd.BasicToolkit:
public abstract class BasicToolkit {

public abstract String generate();

public void validate(String password1, String password2, PasswordPolicy passwordPolicy)
throws PortalException, SystemException {

validate(0, password1, password2, passwordPolicy);
}

public abstract void validate(long userId, String password1, String password2, PasswordPolicy passwordPolicy)
throws PortalException, SystemException;

}

Another possibility is to use the regular expression toolkit that is available in Liferay. This toolkit can be configured as follows:

#
# Input a class name that extends
# com.liferay.portal.security.pwd.BasicToolkit. This class will be called to
# generate and validate passwords.
#
passwords.toolkit=com.liferay.portal.security.pwd.RegExpToolkit

#
# If you choose to use com.liferay.portal.security.pwd.RegExpToolkit as
# your password toolkit, set the regular expression pattern that will be
# used to generate and validate passwords.
#
# Note that \ is replaced with \\ to work in Java.
#
# The pattern ensures that passwords must have at least 4 valid
# characters consisting of digits or letters.
#
passwords.regexptoolkit.pattern=(?=.{4})(?:[a-zA-Z0-9]*)
One useful regular expression (customized a bit more) that checks if a password is at least 6 characters long, contains at least 1 digit, 1 lowercase/uppercase character, 1 symbol from a given list and no whitespace is:

^.*(?=.{6,})(?=.*\d)(?=.*[a-zA-Z])(?=.*[@#$%^&+=])(?!.*\s).*$

I would be using this regular expression if I could only find out how to extend it so it checks if a passwords contains for example at least 3 lowercase/uppercase characters. I tried several variants, but didn't succeed. So for the moment I'm using a custom written BasicToolkit that is using plain old Java code to do the necessary checks.

If a tree falls in the woods does it make a sound? Can you merge a SVN branch if your mother hasn't told you about it?

Change Liferay Chat Portlet Buddy List Strategy

Some time ago I had to check out the possibilities of the Liferay Chat portlet together with a colleague to see if it could be used in our current project or not. Installing it was easily done via the recently introduced Liferay Control Panel. After some quick tests it seemed that for the most part the portlet would be useable if we could find some way to configure who gets to chat to who.

Some clicking around in the control panel and checking out the portlet preferences didn't provide any clues as to how this could be done. So we started looking at the Chat portlet source code and discovered something promising: ChatUtil.getBuddies(long userId).

public class ChatUtil {

public static final int MAX_ENTRIES = 50;

public static final long MAX_POLL_LATENCY = Time.SECOND * 15;

public static final long ONLINE_DELTA = Time.MINUTE;

public static List<Object[]> getBuddies(long userId)
throws SystemException {

long modifiedDate = System.currentTimeMillis() - ONLINE_DELTA;

List<Object[]> buddies = null;

if (PortletPropsValues.BUDDY_LIST_STRATEGY.equals("all")) {
buddies = StatusLocalServiceUtil.getAllStatuses(
userId, modifiedDate, 0, SearchContainer.DEFAULT_DELTA);
}
else if (PortletPropsValues.BUDDY_LIST_STRATEGY.equals("communities")) {
buddies = StatusLocalServiceUtil.getGroupStatuses(
userId, modifiedDate, 0, SearchContainer.DEFAULT_DELTA);
}
else if (PortletPropsValues.BUDDY_LIST_STRATEGY.equals("friends")) {
buddies = StatusLocalServiceUtil.getSocialStatuses(
userId, SocialRelationConstants.TYPE_BI_FRIEND,
modifiedDate, 0, SearchContainer.DEFAULT_DELTA);
}
else if (PortletPropsValues.BUDDY_LIST_STRATEGY.equals(
"communities,friends")) {

List<Object[]> groupBuddies =
StatusLocalServiceUtil.getGroupStatuses(
userId, modifiedDate, 0, SearchContainer.DEFAULT_DELTA);
List<Object[]> socialBuddies =
StatusLocalServiceUtil.getSocialStatuses(
userId, SocialRelationConstants.TYPE_BI_FRIEND,
modifiedDate, 0, SearchContainer.DEFAULT_DELTA);

buddies = new ArrayList<Object[]>(
groupBuddies.size() + socialBuddies.size());

buddies.addAll(groupBuddies);

BuddyComparator buddyComparator = new BuddyComparator(true);

for (Object[] socialBuddy : socialBuddies) {
if (Collections.binarySearch(
groupBuddies, socialBuddy, buddyComparator) < 0) {

buddies.add(socialBuddy);
}
}

Collections.sort(buddies, buddyComparator);
}
else {
buddies = new ArrayList<Object[]>();
}

return buddies;
}

}
This code seems to suggest that there are 4 different ways in which the list of people you can chat to is constructed:
  • all
  • communities
  • friends
  • communities,friends (this looks like something that will have to be refactored someday...)
It seems from the code that all is the default strategy. As almost everything in Liferay can be configured by extending the default portal.properties file, by making and deploying a portal-ext.properties file containing a set of diffs against the original properties file. So we tried to change the strategy to communities by adding 'buddy.list.strategy=communities' to our portal-ext.properties file, using the key as suggested by the code in the PortletPropsValues class:
public class PortletPropsValues {

public static final String BUDDY_LIST_STRATEGY = GetterUtil.getString(
PortletProps.get("buddy.list.strategy"));

}

After restarting the portal and testing, it seemed as if nothing had changed. We still got the see the same people in our respective buddy list. We tried changing the organisations and communities our different users belonged to, but nothing seemed to have the desired effect. Where did we go wrong...? Some Googling pointed us to a forum post that reinforced our feeling that we were indeed looking in the right direction. But why didn't it work then?

While I had to start working on something else, my colleague continued to look into the problem and ultimately found the solution. While we did indeed find the correct configuration key/value pair, we had put it in the wrong file, portal-ext.properties, as the Chat portlet, by means of the portlet.properties file, is configured to override even this file:

include-and-override=portlet-ext.properties

buddy.list.strategy=all
#buddy.list.strategy=communities
#buddy.list.strategy=friends
#buddy.list.strategy=communities,friends
So setting the buddy list strategy in the portal-ext.properties file doesn't have any effect since it will be overridden by the value in portlet.properties because of the include-and-override setting. So when you change the value directly in the portlet everything starts to make a whole lot more sense.

And now I'm gonna have me some object-oriented toast.

Saturday 24 October 2009

My new GPS: to Garmin or not to Garmin

My trusty old GPS device, a Magellan SporTrak Color, that I use for Geocaching, recently started going a bit haywire. After being turned on it would start to go randomly into on-off cycles, making it very difficult to use. I've had this device for about 4 to 5 years now and was pretty satisfied with it. It had all the necessary functions for geocaching such as an electronic compass and map support, but it was starting to show its age a bit, most of all in satellite acquisition speed and accuracy.

So I decided to look for a new device that had to satisfy certain requirements:
  • Electronic compass
  • Map support (commercial and open source if possible)
  • Mac compatible (Magellan has little to no support)
  • USB (my SporTrak only supports serial ports, a USB to serial cable helps, but transfer speed is still slow)
  • Newest generation GPS chip with WAAS/EGNOS support
  • Extendable memory (my SporTrak only had a fixed 32Mb internal memory)
  • Run on standard AA size batteries (I bloody hate battery packs)
  • Color screen (my SporTrak also had this, but it's just a nice to have)
So I started looking around and came across the newer generation Magellan and Garmin devices. After checking out some tests and reviews for the Magellan Trition series and the Garmin Colorado/Oregon/Dakota series, it seemed like these devices are promising, but still have some teething problems. So I decided to go for the tried and tested Garmin GPSMAP 60CSx as it seems to be the preferred device among geocachers according to the usage numbers and reviews on the geocaching site.
The GPSMAP 60CSx fits all the requirements I'd set, but has one small problem: a recommended selling price of 400 euro. So like I usually do when I don't want to pay full price: I started looking around online. After some searching I found a promising eBay shop: SATNAV24. Here I could buy the device I wanted, including an additional 2Gb MicroSD card and get it shipped to me for only 275 euro.

After ordering it and paying for it via PayPal on sunday evening, it was delivered at work on friday, You've got to love those online retailers. With those prices and service I'm wondering why anybody still buys their stuff in regular shops where they never have what you want, treat you as crap, overcharge you or just plainly annoy you (e.g. pushy sales people).

After receiving the new device and testing it out a bit, I've come to the following preliminary conclusions:
  • Startup time and satellite acquisition are really fast
  • Position accuracy is much better than my old SporTrak
  • Mac support in general is good, but MapSource maps still need to be converted on a Windows PC first before they can be used
  • Lots of advanced functionalities (maybe even just a bit too much)
  • Certain tasks, such as entering a waypoint are not as easy, clear and quick as on my old SporTrak
  • The electronic compass is not as simple to use as the one on my old SporTrak
So we'll have to see what the future brings. After some jiggling around with the battery connectors on my old SporTrak it seems to be working a bit better again. The GPSMAP 60CSx will become my new primary GPS device because I don't want to be geocaching somewhere far away from civilisation and have my GPS die on me, but I'll keep the SporTrak as a backup in case I drop the Garmin off a cliff.

Tuesday 13 October 2009

Barbie pink network cable

What's wrong with this picture of the network cable that's currently plugged in to my Mac at a customers' offices?


It's pink! Not a soft pastel pink or even hot pink, bloody Barbie pink. While I do understand the value of color coding network/patch cables so you don't get lost in your own racks, Barbie pink ones are one step too far. And after seeing these cables I think all hope is lost.

Monday 12 October 2009

I do code in my free time

A few days ago I read a blog post by Ted Dziuba: I Don't Code in my Free Time. This post has been reverberating around the blogosphere quite a bit and popped up on a lot of different programming related sites and even on Reddit and I have to say that I disagree.

I'm not a 20-something anymore since this year, but in my opinion age doesn't automatically mean you're right. It's not because you're older that you know better then somebody that's younger. And why couldn't a programmer have a valid opinion about hiring?

My short experience so far tells me that the current hiring procedures for programmers are sadly lacking a lot of oomph. This certainly is the case with a lot of the recruitment companies that wouldn't be able to tell you the difference between an Apache HTTP server and an Apache Tomcat server if their life depended on it. Mostly it's a lot of talk about anything but actual coding skill, but my opinion about how I would hire programmers will be the subject of a future blog post.

While I certainly wouldn't go as far as not hiring someone that doesn't code in his free time, I do think it can be a good additional indicator, especially if it involves a different programming language than is used at your company. It doesn't have to be much and it doesn't have to be coding per se.

Besides coding it could also be reading, going to conferences, posting on forums or maintaining a blog. Everyone of these activities does not only improve your skills, it also gives the interviewer some extra area's to ask questions about that wouldn't come up in an interview otherwise. It is also a way to distinguish yourself from other candidates, be someone unique and show that you care about your craft.

As you can probably tell by now: I do code in my own time. I made and maintain a couple of small websites, I try to blog a bit and I also code some stuff to support one of my other hobbies (I still have time after all the coding, no kids you know): geocaching. Not only do I code for my own, I also code in my own time for work-related stuff, usually when I get some kind of eureka moment at home about how to solve a problem. If I'd waited until I'm back at work the idea would have vanished.

Wednesday 7 October 2009

Hosting a custom maven archetype in Artifactory

Today we needed to set up a an in-house Maven 2 repository. The product we chose for this task is Artifactory from JFrog. Setting up this product will maybe be a separate blog post sometime, but for now the focus of this post will be on getting Artifactory to serve a custom Maven archetype.

After creating your archetype you'll first need to install it to your new Maven repository. This can easily be done using the Artifactory web interface. Once this is done you'll need to create an archetype-catalog.xml file that describes your new archetype using these rules:




your.group.id
your.artifact.id
1.0
http://your.artifactory.server:8081/artifactory/libs-releases/local



Once this is done you'll need to get Artifactory to serve this file. As there is no direct GUI support for this in the web interface, we'll need to use a workaround for this:

curl -u admin:password -f -T /path/to/archetype-catalog.xml -X PUT "http://your.artifactory.server:8081/artifactory/libs-releases-local/archetype-catalog.xml"

Once this is done you'll be able to define a remote archetype catalog with this URL:

http://your.artifactory.server:8081/artifactory/libs-releases-local

You can also easily use it in combination with the m2eclipse plugin:



Atlassian rules

One of my favorite software companies, Atlassian, has just announced that they're reviving their Starter program and that this time it is here to stay. Under the terms of the starter program you can get licenses for up to six products at 10$ a piece. If that isn't a bargain I don't know what is. And they're not shortchanging the buyers, since you'll get:
  • a full featured product
  • 1 year of support and maintenance
  • you'll be able to renew the support each year for 10$
  • an easy to follow install guide, Here be Dragons, with the promise of getting a unique t-shirt when you complete the quest
  • some freebies to complement the products (although a voluntary 10$ donation to Room to Read is encouraged as the 10$ of the other product are also donated in full to this charity - way to go Atlassian!)
The six products to choose from are:
  • Jira: an excellent issue tracker, that I've been using on and off for the last 7 years
  • Confluence: a great wiki/content sharing product, which I also have been using for about 2 years now
  • GreenHopper: an interesting Jira plugin that supports the agile methodology that the company I work for has tested for some time and is now using in production
  • Bamboo: looks like a full-featured continuous integration server that we could use instead of Hudson
  • FishEye: I think our current company CVS server is in dire need for this product. Maybe I should propose this sometime to our CTO
  • Crowd: this is a SSO/IDM product that we're currently evaluating as a less complicated replacement for Sun IDM when not all the bells and whistles of Sun IDM are needed
So even though these licenses are only for 10 users/plans/committers (50 users in the case of Crowd) they are perfect for small teams or even if you're tinkering on your own at home. Confluence, Jira and Bamboo for 10$ a pop are sounding very interesting to use on my own Skunk works.

Wednesday 23 September 2009

Banking fail (+ cell phone rant)

Disclaimer: I'm going to go off on a rant that will at first have nothing to do with banks, but bear with me, I'll get there in the end.

First off, just to make something clear: I hate and despise cell phones. They're the bane of the 21st century. Everybody seems to have one of these things, sometimes even more than one, and they seem to be using it all the time, in every annoying way possible. And don't get me started on the annoying ringtones. Oh god, the ringtones, they make me want to punch someone in the face.

I could keep on ranting about it for pages and pages, but I have to get to the banks sometime before the end of the month.

Even though most of the people I know are aware of the fact that I loathe cell phones, my current employer still wanted me to have a cell phone so I would be reachable. I don't want to be reachable. If you want to contact me, send me an email. I really like the asynchronous nature of email against the synchronous nature of cell phones. So I said - give me a cell phone and pay for a subscription - hoping that that would make them back out. Unfortunately that plan kinda backfired and I now am the proud owner of a company cell phone.

The subscription they provide allows for a certain fixed amount of call minutes for both professional and personal use. Everything above that amount needs to be payed by the employee. Because I tend to forget the cell phone (and if I haven't forgotten it, it probably isn't charged) I've consistently had a monthly amount due of 0 euros. This month was the first time that I actually needed to pay for going over the fixed amount. I had to pay the incredible amount of 43 eurocents!

So I opened up my online banking application, proceeded to fill in all the required fields and pressed sent. And what do I get: the amount you entered is too low! Oh please, are you bloody kidding me? They have a fully ajax-enabled form, but they can't warn me that the amount is too low when I fill in that field, which incidentally is the first field on the bloody form? I don't know why there should even be a threshold value, I'm doing all the work and everything is handled electronically. I guess it would probably be possible to do the transfer if I'd physically go to the bank and ask a teller to do it, but that would be an even bigger waste of time and money for all parties involved.

So now I'm sitting here with a bill for 43 eurocents (paper and postage alone probably costs more) and my bank won't let me pay it because the amount is too low! It's like I'm trapped in a catch-22. I've explained my quandary to the HR manager and she's going to try and find a solution with the telecom provider. I wonder what it will be.

Canon MEAP engrish: I can has fail?

Sometimes it shows that Canon MEAP is made in Japan. So every now and then I get to enjoy some prime examples of engrish. Last week I was experimenting with MEAP IMI, got a stacktrace and encountered something that made me go WTF?

com.canon.meap.imi.OperationFailureException: This bundle has initialize.
    at com.canon.meap.csee.imi.IMIImpl.initialize(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.canon.meap.imi.IMI.initialize(Unknown Source)
    at be.aca.wizard.applet.AcaWizardApplet.start(Unknown Source)
    at com.canon.meap.csee.avs.AppletEventDispatcher.run(Unknown Source)
    at java.lang.Thread.startup(Unknown Source)
   
I'm still trying to figure out what the exception exactly means: already initialized, not correctly initialized, ...

Sunday 20 September 2009

Error message fail!

I just tried to watch the video of Drew Barrymore in Jay Leno's Reasonably-Priced Electric Ford Focus "Race Car" in this post on the Jalopnik site, but that resulted in total fail:


Don't know what the cause is (apparently: null), maybe you can only watch it in the USA or so and not from Spain where I'm currently on holiday, but it sure is something that I should submit to The Daily WTF.

Tuesday 8 September 2009

Java profiling on OSX

Not so long ago I needed to profile a small application to check why a simple WebDAV listing call using Slide took 20 seconds. It turned out to be a network IO problem caused by a blocking readline call on the response stream. I usually tend to use JProfiler for finding out these kind of things, because it is very good at what it does. But because it is a paying application and my company doesn't provide me with a license, I have to keep my eyes open for alternatives (you can't keep working with trial licenses and Mailinator addresses aren't allowed when registering).

So when the Eclipse Test & Performance Tools Platform Project, TPTP in short, appeared on my radar thanks to DZone I decided to try it out. Since I'm using Eclipse as my primary Java IDE this profiling plugin seemed a perfect fit, but little did I know. In Eclipse I opened Help > Install New Software... and added the http://eclipse.org/tptp/updates/ URL as a new software site to download TPTP. After downloading, agreeing to some licenses and an Eclipse restart I was ready to use my newly acquired profiler, or ... maybe not.

I was getting all kinds of different errors, but mostly NullPointerExceptions and some Incompatible Platform messages. After some looking around it was StackOverflow to the rescue: even though TPTP is already at version 4.3 there still isn't a TPTP Agent Controller for OSX. You've gotta be kidding me! How is it possible that they leave such a large group of Java developers, the ones that use OSX, out in the cold. It gets even better, you can't even open the TPTP 4.3 release notes page in Safari without getting a Javascript pop with the message: Sorry but this page isn't currently viewable in Safari. WTF!

So my little excursion away from JProfiler came to a screeching halt and I had to return to my tried and trusted JProfiler once again. One day I'll have to convince the big cheeses to get me a license.

Update 9 september 2009: it looks like I'm not the only one complaining: Green's Opinion

Monday 7 September 2009

Load testing gotcha nr. 1: logging

This will be a first in, hopefully, a series of short but important things to keep in mind when load testing. So without further ado:

Load Testing Gotcha nr. 1

Logging is a good thing, but too much will kill you, your performance and ultimately your server. So use it wisely young Padawan.


Logging and certainly good contextual logging is always nice when things go wrong, but during load testing it should be turned off or as low as possible. The reasoning behind this is that logging, certainly on the more verbose levels will impose a hard to ignore load on your CPU/Memory and certainly on your disk(s).

This extra load will influence the request/response times of your tests. So when your log level is set too high and doesn't reflect your production settings, your load tests won't be representative and can't be reliably used too predict production server behavior under load.

As far as I can tell there are only two reasons to load test using a more verbose log level:
  1. You're actually suspecting your logging of being a performance bottleneck and as such you want to do some load tests at different verbosity levels to study the impact of your logging.
  2. Your application is failing at certain load levels and you need more detailed logging to home in on the cause of the failure.
So this is my (current) view on logging and load testing. I also think that this view isn't only applicable on load testing alone, but should also be kept in the back of your mind during application development. What I want to say with this is that you always have to think carefully about what to log on which logging level.

And one extra free piece of advice: never, ever, use XML as a logging format. I have encountered it once or twice in the wild as a consultant and it will use about 30% more disk space than non-XML logging (possibly even more depending on the length of your tags) and isn't exactly human-readable unless you're name is Neo and you're living in the Matrix.

Another easy WebDAV test server

In the previous week I discovered another easy way to set up a WebDAV test server: Atlassian Confluence. This is a pretty neat wiki product that also exposes its contents via WebDAV. The Devoxx guys even used it to make their whole website.

To set up a new Confluence server you first have to download one of the fully functional 30-day trial editions here. After downloading you just unpack the file to reveal a standalone Tomcat server and then you'll need follow the installation instructions. In short this means:
  • checking your Java installation: version and home directory
  • setting a confluence.home value in /confluence/WEB-INF/classes/confluence-init.properties to a valid path somewhere on your disk
  • make sure you don't have port conflicts
  • start Confluence by starting the standalone Tomcat instance from the bin directory: ./catalina.sh start
  • browse to http://localhost:8080
  • follow the setup wizard you're presented with and remember the username and password you enter
After this you'll have a WebDAV server listening at http://localhost:8080/plugins/servlet/confluence/default. You'll have to authenticate using basic authentication with the credentials you entered in the setup wizard. By default there won't be much data available, but this can be easily solved by creating some spaces in Confluence and adding some pages to the spaces.

Monday 31 August 2009

Listing open ports on OSX

Another day, another thing learned.

I've only been using OSX for about a year (no way in hell I'm ever going back to Winblows) and so I still consider myself quite the OSX virgin. Also my Unix/Linux or Unix-like experience isn't that extensive either. I've only played around a bit with Solaris and Ubuntu and I know just enough to find my way around and support my development work, but that is where it stops.

So for listing open ports I always farted about a bit with netstat and grep. So imagine my surprise when I tried to list the open ports on my Mac in my tried and tested way. Just let me say it wasn't quite what I'd expected. But it seems that there is nothing that a bit of Googling won't solve and so I quickly arrived at the lsof command (which isn't solely available on OSX by the way) and now I use the following command to list open ports:

sudo lsof -i -P | grep -i "listen"

During my Googling I also came across this extremely useful site: commandlinefu.com

Tuesday 25 August 2009

Eclipse debug problem

Yesterday I had some strange debug problem in Eclipse on OSX. Whenever I tried to start a debug session I got the following error screen


Accompanying this error dialog were the following lines in the Eclipse Console view

Error [22] in gethostbyname() call!
err:: Invalid argument
Socket transport failed to init.
Transport dt_socket failed to initialize, rc = -1.
FATAL ERROR in native method: No transports initialized


After some Googling it seems that the cause is an incorrect network configuration. Since I recently had changed something in my hosts file, that was the first place I was going to look, certainly after trying to do ping localhost and not getting a response back.

A quick examination of my /etc/hosts file and adding the line

127.0.0.1 localhost

solved my debugging problems.

Tuesday 11 August 2009

Accessing WebDAV from Java on OSX

I'm currently working on an extension for an existing project. The project consists of a Canon MEAP application that enables users of multifunctionals to scan documents in a quick and simple way to a SMB share or to an email address. The extension that I have to build involves adding a third transfer method, WebDAV, so that scanned documents can be sent to our Atlassian Confluence wiki.

Since the Canon MEAP platform is Java based, I started looking a Java WebDAV libraries. I found 2 good ones: Slide and Jackrabbit. The Slide project has been retired due to lack of active maintenance. This leaves us with Jackrabbit, a JCR (JSR 170) implementation, that also contains a nice WebDAV implementation that is based on Commons HttpClient.

After this I decided to do a small proof of concept to test the the 2 use cases that I needed for my extension: get a file listing and put a file on a WebDAV server. For this I needed a WebDAV capable server off course, normally I could have just used our actual wiki, but after a recent upgrade it seems that something had changed in the WebDAV integration and I couldn't access anything from the normal URL (later got to know that the new/current WebDAV URL of Confluence is https://myserver.com/confluence/plugins/servlet/confluence/default).

So I needed an alternative solution: setting up my own WebDAV server. After looking around a bit I found out that Jackrabbit also has a simple WebDAV enabled server included and does this via a WAR file that you can simply drop into a default Tomcat installation (I used jackrabbit-webapp-1.5.7.war). Another possibility would have been to configure the Apache server that is bundled with OSX to enable WebDAV, but since I've already got Jackrabbit and Tomcat is closer to my Java background I favored that approach.

Since I was already trying a lot of new stuff, I also decided to use that latest Tomcat version and download a 6.0.20 Core version. After doing this I unpacked the downloaded file into a directory, cd'ed to the bin directory of the Tomcat installation and ran ./catalina.sh start and encountered my first problems:
  • -bash: ./catalina.sh: Permission denied: solved by chmod +x catalina.sh
  • The BASEDIR environment variable is not defined correctly: again the cause seems to be some kind of permission problem and was solved by this post.
After this I dropped the download WAR, jackrabbit-webapp-1.5.7.war, into the webapps directory and watched the application deploy. If everything deployed correctly you should now be able to access the web interface at http://localhost:8080/jackrabbit-webapp-1.5.7. The home page displayed fine, but every other menu item I clicked threw errors. After a restart of Tomcat I traced the problem down to a missing JCR jar. This was solved by downloading one and placing it in the lib directory of Tomcat and restarting it. After this you'll need to go to the home page again and click the Create Content Repository button to initialize the WebDAV correctly. Once the initialization is done you'll be able to access your local WebDAV instance at http://localhost:8080/jackrabbit-webapp-1.5.7/repository/default/ (any username/password combination will work by default).

Now we have a working WebDAV server we can concentrate on creating a client that can do a listing of the WebDAV root directory and place a new file in it. Using a set of different sources I was able to put together the following code:


import java.io.FileInputStream;
import java.io.IOException;

import org.apache.commons.httpclient.Credentials;
import org.apache.commons.httpclient.HostConfiguration;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpConnectionManager;
import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager;
import org.apache.commons.httpclient.UsernamePasswordCredentials;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.methods.InputStreamRequestEntity;
import org.apache.commons.httpclient.methods.PutMethod;
import org.apache.commons.httpclient.methods.RequestEntity;
import org.apache.jackrabbit.webdav.DavConstants;
import org.apache.jackrabbit.webdav.DavException;
import org.apache.jackrabbit.webdav.MultiStatusResponse;
import org.apache.jackrabbit.webdav.client.methods.DavMethod;
import org.apache.jackrabbit.webdav.client.methods.PropFindMethod;

public class WebdavTest {

public static void main(String[] args) throws IOException, DavException {
HostConfiguration hostConfig = new HostConfiguration();
hostConfig.setHost("127.0.0.1", 8080);
HttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
HttpClient client = new HttpClient(connectionManager);
Credentials creds = new UsernamePasswordCredentials("userId", "pw");
client.getState().setCredentials(AuthScope.ANY, creds);
client.setHostConfiguration(hostConfig);

PutMethod put = new PutMethod("http://127.0.0.1:8080/jackrabbit-webapp-1.5.7/repository/default/test.txt");
RequestEntity requestEntity = new InputStreamRequestEntity(new FileInputStream("/path/to/file/test.txt"));
put.setRequestEntity(requestEntity);
client.executeMethod(put);

DavMethod pFind = new PropFindMethod("http://127.0.0.1:8080/jackrabbit-webapp-1.5.7/repository/default", DavConstants.PROPFIND_ALL_PROP, DavConstants.DEPTH_1);
client.executeMethod(pFind);

MultiStatusResponse[] responses = pFind.getResponseBodyAsMultiStatus().getResponses();
System.out.println("Folders and files: ");
for (int i = 0; i < responses.length; i++) {
System.out.println(responses[i].getHref());
}
}
}

You'll need the following libraries to make the code compile and run: commons-codec, commons-httpclient, commons-logging, jackrabbit-jcr-commons, jackrabbit-webdav, slf4j-api and slf4j-nop.

Thursday 6 August 2009

JMeter memory problems

Sometimes things that have never failed before do fail. Today this happened while using the tried, tested and trusted JMeter load testing tool. I've used JMeter a lot during the years and even though it is starting to show its age, it is hard to find something better to do the job.

Lately the only thing that made JMeter fail was when an OutOfMemoryError occurred. This was always easily solved by adjusting the heap memory size settings (-Xms and -Xmx) and sometimes even the Permanent Generation size settings (-XX:PermSize and -XX:MaxPermSize) to appropriate values. But today we did this to no avail. While we were only running a load test for 40 threads with 10 loops each, it would still crash and burn even with 2 gigabytes of heap memory.

So we were thinking that maybe the listeners were to blame, so we disabled the Graph Results listener and even the View Results Tree listener, but still kept getting OutOfMemoryError occurrences. So now we were kind of stuck, or maybe not... . I remembered that JMeter can also be run remotely and without a GUI. Could this maybe help? It sure can't hurt to try. So we fired up JMeter in non-GUI mode and what do you think, no more OutOfMemoryError occurrences as far as the eye can see (the jmeter.log file in the bin directory that is).

So now we only had to re-enable the listener, configure them to write their data to some files (so that we could view them in JMeter after the non-GUI tests were done) and run JMeter using the following command: ./jmeter -n -t /path/to/loadtest.jmx

Wednesday 5 August 2009

Checking out Sharepoint webservices

During my work the last few weeks I've been working a lot with webservices and more specific Sharepoint webservices. Eventhough you can hardly call them webservices. They are a pain a real in the ass, the APIs are bad, but the worst thing is the MSDN documentation about them: absolutely useless.

The only thing that makes working with them bearable is soapUI. This excellent, multiplatform application makes life very easy when you need to explore the webservices some product/website/application offers. Just download the standalone version or use one of the various IDE plugins. After this it is as easy as
  1. Create a new project via File > New soapUI Project
  2. Enter a project name
  3. Provide a WSDL, either via a local file or via a URL to a remote service
  4. Keep the Create Requests checkbox ticked
  5. Click OK
After this soapUI will generate a sample request for all the methods present in the WSDL. Now you just have to select the method you want to test, fill in some parameters (soapUI will indicate the optional ones) and click play. That's it.

The only additional functionality that I needed to use was provide some authentication credentials using the Aut button on the bottom of a request window.

One of the more useful Sharepoint webservices that I used was the List Data Retrieval service: http://host:port/_vti_bin/dspsts.asmx?wsdl. This has the ability to return the all the fields of a Sharepoint list together with a definition of the structure of the results. Below an example of the request I used:





1.0
















Monday 3 August 2009

JAX-WS and Maven2

Another day in Maven hell! I still don't get how something as simple as trying to get the JAX-WS tool wsgen to work can be this difficult.

My goal was to create a set of webservices starting from Java code and to then generate the necessary WSDL and XSD files. The first part, creating a domain model, annotating it with JAXB and then generating an XSD was simple enough. The second step kept me busy for the remaining 75% of the day.

I just added a new dependeny to my pom.xml and annotated my service with @WebService. But after triggering a recompile I got an error, the WebService annotation couldn't be found. Checked the pom.xml again, but it contained the correct dependency. So what was going wrong. Afer trying out some suggestions made in various blogs and forum posts, the problem remained. I even got to a point in which the code compiled, but when I executed the wsgen goal it would fail with the message: java.lang.NoClassDefFoundError: javax/jws/WebService. What? How can this be? By this time I was cursing away at Maven again: how can this bloody thing, that is supposed to make my life easier, always fuck up in the most incomprehensible way possible?

Then a little speck of light in the distance: it seems that the JAX-WS libraries are spread over a maven1 and a maven2 repository and that there are problems with some versions and references to other libraries that JAX-WS uses. The solution to all this:
  • empty out your local repository
  • add the java.net maven1 and maven2 repositories to your pom.xml
  • add a dependency to JSR-181 to the wsgen plugin
After doing all this I was able to compile, build and generate everything I wanted (it only took way too much time).

The pom.xml I used:


xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
4.0.0
EDRL
Preprocessor
0.0.1-SNAPSHOT



com.sun.tools.jxc.maven2
maven-jaxb-schemagen-plugin
1.1


process-sources

${project.build.directory}/schemas

be/test/*.java

${basedir}/src/main/java
true


generate





org.codehaus.mojo
jaxws-maven-plugin
1.12



wsgen

generate-sources

be.test.service.TestService
true
true
true






javax.jws
jsr181-api
1.0-MR1







maven-repository.dev.java.net
Java.net Repository for Maven 1
http://download.java.net/maven/1/
legacy


maven2-repository.dev.java.net
Java.net Maven 2 Repository
http://download.java.net/maven/2




maven2-repository.dev.java.net
Java.net Maven 2 Repository
http://download.java.net/maven/2




javax.xml
jaxb-impl
2.1


javax.xml.bind
jaxb-api
2.2


com.sun.xml.ws
jaxws-rt
2.1.3


javax.jws
jsr181-api
1.0-MR1



1.5
1.5




Sunday 2 August 2009

Better validation messages in JSF

This week a collegue of mine wanted to enhance the form validation messages in a JSF application, but he was having problems getting a nice fieldname into the message. The default behavior of JSF is to display something like: j_id_id19: Validation Error: Value is required. This is Ok for a programmer, but an end-user gets kinda confused. There are several solutions for this problem:
  • Overwrite the default JSF messages in Messages.properties to not include the field name, but this will make the problem even worse.
  • Write a custom renderer for the messages, but this is about the same as amputating a leg when you've hit your toe against the bed.
  • Or we could just write a correct HTML form element that has, and this is the important bit, a label attribute, as this is what JSF will use in its default validation message. When this isn't present it will default to the id and when this id auto generated, as it usually is, will show something like j_id_id19.
So my collegue's problem was easily solved by adding label attributes to the fields of his form.

PHP error: Illegal offset type for array

I was working on my parents in law house renting site in Spain, www.casacalida.es, this weekend trying to integrate some weather site data that is provided in XML format, when I ran into a PHP error that I couldn't figure out at first. I'll elaborate some more about the integration of the XML data into their CodeIgniter site later. The error seems to indicate that my multidimensional array keys were wrong somewhere, but I couldn't figure it out at first.

Some searching brought me to this post, that seemed to confirm my suspicion. Only I was using all string keys and not integers, but Maybe I could adopt some of the solution. So I added a string cast for my array key and hey presto my problem was solved.

Getting SSIS to like Sharepoint XMLs

I'm currently working on a project in which we need to get some content from a Sharepoint instance into a SQL server database. The work is being done by another consultant using Microsoft SSIS. The simplest way to get the data from one side to the other would be to do a straight database to database data synchronization. The problem with this plan is that the Sharepoint database schema just plain sucks and that Microsoft doesn't recommend accessing it directly, not just because of the crappy schema, but also because it could change between Sharepoint versions.

So plan B is to use the abundantly available Sharepoint webservices (if you could call them webservices). Since SSIS has support for webservices we tried that and failed again. We found no way to get SSIS to understand the Sharepoint webservices, it just kept complaining about the WSDL and the XSD. So Googling gave us an alternate solution: create some stubs around the webservice and use those in a script task. This got use a bit further but then SSIS started complaining about the XML using multiple namespaces and it also couldn't handle the XHTML content in some tags.

But again we found a workaround. Using the following XSLT to preprocess the XML before using it made SSIS accept it. The XSL removes all namespaces and places the XHTML content of selected tags, by means of a xsl:param in CDATA sections. One last problem was that SSIS doesn't give you the means to provide an XSL with params, so you'll have to put them directly into the content of the xsl:param.


<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:output method="xml" />

<xsl:param name="rtfNodes">TagsName1~TagsName1</xsl:param>

<xsl:template match="*">
<xsl:choose>
<xsl:when test="contains($rtfNodes, local-name())">
<xsl:element name="{local-name()}">
<xsl:apply-templates select="@*" />
<xsl:text disable-output-escaping="yes"><![CDATA[ <![CDATA[ ]]></xsl:text>
<xsl:copy-of select="node()"/>
<xsl:text disable-output-escaping="yes"><![CDATA[]]]]><![CDATA[>]]></xsl:text>
</xsl:element>
</xsl:when>
<xsl:otherwise>
<xsl:element name="{local-name()}">
<xsl:apply-templates select="@*" />
<xsl:apply-templates select="node()" />
</xsl:element>
</xsl:otherwise>
</xsl:choose>
</xsl:template>

<xsl:template match="@*">
<xsl:attribute name="{local-name()}">
<xsl:value-of select="."/>
</xsl:attribute>
</xsl:template>

<xsl:template match="/">
<xsl:copy>
<xsl:apply-templates />
</xsl:copy>
</xsl:template>

<xsl:template match="comment() | processing-instruction() | text()">
<xsl:copy />
</xsl:template>

</xsl:stylesheet>


Thursday 30 July 2009

Liferay Flash Movie Hack

Currently were displaying a company movie in some journal article on our Liferay portal, but the movie isn't located on the Liferay server itself. This was all working nice and dandy, but now the server it is hosted on is being migrated and the hack we used to host the movie on it won't be available on the new server. The hack we used was to place the movie file into a custom folder inside the 'docroot' folder of a Glassfish domain.

Since the server was being migrated and it was already a hack, something had to change, preferably for the better. So we were looking into placing the movie, the movie player and the skin for the movie player, all .flv files into the Liferay document library. Well this is were we ran into the first problems:
  • File extension: .flv and .swf files can't be added to the library by default. Solution: adding dl.file.extensions=.flv,.swf,.mov,.avi,.mp4,.mp3,.m4v,.bmp,.css,.doc,.dot,.gif,.gz,.htm,.html,.jpg,.js,.lar,.odb,.odf,.odg,.odp,.ods,.odt,.pdf,.png,.ppt,.rtf,.swf,.sxc,.sxi,.sxw,.tar,.tiff,.tgz,.txt,.vsd,.xls,.xml,.zip to your portal-ext.properties and restarting the portal. We also tried dl.file.extensions=* and dl.file.extensions=.*, but both didn't work as expected.
  • File size: the movie file turned out to be to big for the document library, but this can be solved by adding dl.file.max.size=0 to your portal-ext.properties and restarting the portal.
So now we had all the necessary files present in the Liferay document library, so you would think that we would only have to switch out the URLs in our movie journal article and hey presto! But, this didn't work. The page would show up, but the movie wouldn't. So after Googling a bit and looking around the Liferay forums, we found a promising post that claims that you have to URLEncode the document URLs and then it should work. We tried this in various ways, by hand and by using a Velocity template to do the hard work in a repeatable way, but still the movie wouldn't show.

Now we were getting desperate, what other possibilities were left? A custom portlet maybe? We tried some more of the stuff mentioned in the Liferay forums, when we saw some reference to Flash content in the following path /html/js/flashutils/player.swf. We check our Liferay installation and in the /webapps/ROOT directory we found a /html directory. We created a new directory, media, beneath it, placed our three Flash files in it and referenced to them from our journal article as follows (other paths are similar):

<param name="movie" value="/html/media/movie.flv" />

We saved the article, browsed to the correct page and now the movie is shown in its full glory. Disclaimer: this is a dirty hack, we don't recommend to use it, certainly not in a structural way, but until we've found a better, working solution, this does solve our problem for now.

Tuesday 28 July 2009

Liferay theme problem

A while ago we were getting some issues reported from users that wew seeing some strange screens when accessing our Liferay portal. Although they saw most of the data we expected them to see, it wasn't presented properly, as if no theme was being applied. Looking in the logs didn't turn up anything useful and since I didn't see it with my own eyes I instructed the user that reported the error to email me a screenshot when it happened again.


So fast forward to today and I get an email with a screenshot. The screen is pretty white generally, but contains all the data, text and images, that I would expect on that given page, but it looks like no theme at all is applied. It check it for myself using Firefox, the user that reported the problem is using IE and you never know, and low and behold, I get the same fucked up page. So immediately I dive into the logs again, but still nothing.


So what could it be, maybe the portal thinks it is being accessed by a mobile device and presents a simpler theme? But that doesn't really make sense since both the user and I are using desktop browsers. Still I check the theme settings on the portal using the 'Manage Pages' part of the dock, but everything seems to be in order. So next step, check Google. And there we find something interesting, a caching problem: LPS-2834. This issue describes exactly what we are experiencing since after a few refreshes in my browser I get the correctly rendered portal page again. So it seems that something gets cached when a mobile device accesses the portal, but that this cached result is also returned when a non mobile browser requests the same content.


So we could apply the patch that is attached to the issue, but we'll probably use this issue as an excuse to do an upgrade to Liferay Portal 5.2.2 as this will solve this bug and some other problems we're having with our current Liferay Portal 5.1.1 on which I may or may not elaborate in the future.