Monday 31 August 2009

Listing open ports on OSX

Another day, another thing learned.

I've only been using OSX for about a year (no way in hell I'm ever going back to Winblows) and so I still consider myself quite the OSX virgin. Also my Unix/Linux or Unix-like experience isn't that extensive either. I've only played around a bit with Solaris and Ubuntu and I know just enough to find my way around and support my development work, but that is where it stops.

So for listing open ports I always farted about a bit with netstat and grep. So imagine my surprise when I tried to list the open ports on my Mac in my tried and tested way. Just let me say it wasn't quite what I'd expected. But it seems that there is nothing that a bit of Googling won't solve and so I quickly arrived at the lsof command (which isn't solely available on OSX by the way) and now I use the following command to list open ports:

sudo lsof -i -P | grep -i "listen"

During my Googling I also came across this extremely useful site: commandlinefu.com

Tuesday 25 August 2009

Eclipse debug problem

Yesterday I had some strange debug problem in Eclipse on OSX. Whenever I tried to start a debug session I got the following error screen


Accompanying this error dialog were the following lines in the Eclipse Console view

Error [22] in gethostbyname() call!
err:: Invalid argument
Socket transport failed to init.
Transport dt_socket failed to initialize, rc = -1.
FATAL ERROR in native method: No transports initialized


After some Googling it seems that the cause is an incorrect network configuration. Since I recently had changed something in my hosts file, that was the first place I was going to look, certainly after trying to do ping localhost and not getting a response back.

A quick examination of my /etc/hosts file and adding the line

127.0.0.1 localhost

solved my debugging problems.

Tuesday 11 August 2009

Accessing WebDAV from Java on OSX

I'm currently working on an extension for an existing project. The project consists of a Canon MEAP application that enables users of multifunctionals to scan documents in a quick and simple way to a SMB share or to an email address. The extension that I have to build involves adding a third transfer method, WebDAV, so that scanned documents can be sent to our Atlassian Confluence wiki.

Since the Canon MEAP platform is Java based, I started looking a Java WebDAV libraries. I found 2 good ones: Slide and Jackrabbit. The Slide project has been retired due to lack of active maintenance. This leaves us with Jackrabbit, a JCR (JSR 170) implementation, that also contains a nice WebDAV implementation that is based on Commons HttpClient.

After this I decided to do a small proof of concept to test the the 2 use cases that I needed for my extension: get a file listing and put a file on a WebDAV server. For this I needed a WebDAV capable server off course, normally I could have just used our actual wiki, but after a recent upgrade it seems that something had changed in the WebDAV integration and I couldn't access anything from the normal URL (later got to know that the new/current WebDAV URL of Confluence is https://myserver.com/confluence/plugins/servlet/confluence/default).

So I needed an alternative solution: setting up my own WebDAV server. After looking around a bit I found out that Jackrabbit also has a simple WebDAV enabled server included and does this via a WAR file that you can simply drop into a default Tomcat installation (I used jackrabbit-webapp-1.5.7.war). Another possibility would have been to configure the Apache server that is bundled with OSX to enable WebDAV, but since I've already got Jackrabbit and Tomcat is closer to my Java background I favored that approach.

Since I was already trying a lot of new stuff, I also decided to use that latest Tomcat version and download a 6.0.20 Core version. After doing this I unpacked the downloaded file into a directory, cd'ed to the bin directory of the Tomcat installation and ran ./catalina.sh start and encountered my first problems:
  • -bash: ./catalina.sh: Permission denied: solved by chmod +x catalina.sh
  • The BASEDIR environment variable is not defined correctly: again the cause seems to be some kind of permission problem and was solved by this post.
After this I dropped the download WAR, jackrabbit-webapp-1.5.7.war, into the webapps directory and watched the application deploy. If everything deployed correctly you should now be able to access the web interface at http://localhost:8080/jackrabbit-webapp-1.5.7. The home page displayed fine, but every other menu item I clicked threw errors. After a restart of Tomcat I traced the problem down to a missing JCR jar. This was solved by downloading one and placing it in the lib directory of Tomcat and restarting it. After this you'll need to go to the home page again and click the Create Content Repository button to initialize the WebDAV correctly. Once the initialization is done you'll be able to access your local WebDAV instance at http://localhost:8080/jackrabbit-webapp-1.5.7/repository/default/ (any username/password combination will work by default).

Now we have a working WebDAV server we can concentrate on creating a client that can do a listing of the WebDAV root directory and place a new file in it. Using a set of different sources I was able to put together the following code:


import java.io.FileInputStream;
import java.io.IOException;

import org.apache.commons.httpclient.Credentials;
import org.apache.commons.httpclient.HostConfiguration;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpConnectionManager;
import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager;
import org.apache.commons.httpclient.UsernamePasswordCredentials;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.methods.InputStreamRequestEntity;
import org.apache.commons.httpclient.methods.PutMethod;
import org.apache.commons.httpclient.methods.RequestEntity;
import org.apache.jackrabbit.webdav.DavConstants;
import org.apache.jackrabbit.webdav.DavException;
import org.apache.jackrabbit.webdav.MultiStatusResponse;
import org.apache.jackrabbit.webdav.client.methods.DavMethod;
import org.apache.jackrabbit.webdav.client.methods.PropFindMethod;

public class WebdavTest {

public static void main(String[] args) throws IOException, DavException {
HostConfiguration hostConfig = new HostConfiguration();
hostConfig.setHost("127.0.0.1", 8080);
HttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
HttpClient client = new HttpClient(connectionManager);
Credentials creds = new UsernamePasswordCredentials("userId", "pw");
client.getState().setCredentials(AuthScope.ANY, creds);
client.setHostConfiguration(hostConfig);

PutMethod put = new PutMethod("http://127.0.0.1:8080/jackrabbit-webapp-1.5.7/repository/default/test.txt");
RequestEntity requestEntity = new InputStreamRequestEntity(new FileInputStream("/path/to/file/test.txt"));
put.setRequestEntity(requestEntity);
client.executeMethod(put);

DavMethod pFind = new PropFindMethod("http://127.0.0.1:8080/jackrabbit-webapp-1.5.7/repository/default", DavConstants.PROPFIND_ALL_PROP, DavConstants.DEPTH_1);
client.executeMethod(pFind);

MultiStatusResponse[] responses = pFind.getResponseBodyAsMultiStatus().getResponses();
System.out.println("Folders and files: ");
for (int i = 0; i < responses.length; i++) {
System.out.println(responses[i].getHref());
}
}
}

You'll need the following libraries to make the code compile and run: commons-codec, commons-httpclient, commons-logging, jackrabbit-jcr-commons, jackrabbit-webdav, slf4j-api and slf4j-nop.

Thursday 6 August 2009

JMeter memory problems

Sometimes things that have never failed before do fail. Today this happened while using the tried, tested and trusted JMeter load testing tool. I've used JMeter a lot during the years and even though it is starting to show its age, it is hard to find something better to do the job.

Lately the only thing that made JMeter fail was when an OutOfMemoryError occurred. This was always easily solved by adjusting the heap memory size settings (-Xms and -Xmx) and sometimes even the Permanent Generation size settings (-XX:PermSize and -XX:MaxPermSize) to appropriate values. But today we did this to no avail. While we were only running a load test for 40 threads with 10 loops each, it would still crash and burn even with 2 gigabytes of heap memory.

So we were thinking that maybe the listeners were to blame, so we disabled the Graph Results listener and even the View Results Tree listener, but still kept getting OutOfMemoryError occurrences. So now we were kind of stuck, or maybe not... . I remembered that JMeter can also be run remotely and without a GUI. Could this maybe help? It sure can't hurt to try. So we fired up JMeter in non-GUI mode and what do you think, no more OutOfMemoryError occurrences as far as the eye can see (the jmeter.log file in the bin directory that is).

So now we only had to re-enable the listener, configure them to write their data to some files (so that we could view them in JMeter after the non-GUI tests were done) and run JMeter using the following command: ./jmeter -n -t /path/to/loadtest.jmx

Wednesday 5 August 2009

Checking out Sharepoint webservices

During my work the last few weeks I've been working a lot with webservices and more specific Sharepoint webservices. Eventhough you can hardly call them webservices. They are a pain a real in the ass, the APIs are bad, but the worst thing is the MSDN documentation about them: absolutely useless.

The only thing that makes working with them bearable is soapUI. This excellent, multiplatform application makes life very easy when you need to explore the webservices some product/website/application offers. Just download the standalone version or use one of the various IDE plugins. After this it is as easy as
  1. Create a new project via File > New soapUI Project
  2. Enter a project name
  3. Provide a WSDL, either via a local file or via a URL to a remote service
  4. Keep the Create Requests checkbox ticked
  5. Click OK
After this soapUI will generate a sample request for all the methods present in the WSDL. Now you just have to select the method you want to test, fill in some parameters (soapUI will indicate the optional ones) and click play. That's it.

The only additional functionality that I needed to use was provide some authentication credentials using the Aut button on the bottom of a request window.

One of the more useful Sharepoint webservices that I used was the List Data Retrieval service: http://host:port/_vti_bin/dspsts.asmx?wsdl. This has the ability to return the all the fields of a Sharepoint list together with a definition of the structure of the results. Below an example of the request I used:





1.0
















Monday 3 August 2009

JAX-WS and Maven2

Another day in Maven hell! I still don't get how something as simple as trying to get the JAX-WS tool wsgen to work can be this difficult.

My goal was to create a set of webservices starting from Java code and to then generate the necessary WSDL and XSD files. The first part, creating a domain model, annotating it with JAXB and then generating an XSD was simple enough. The second step kept me busy for the remaining 75% of the day.

I just added a new dependeny to my pom.xml and annotated my service with @WebService. But after triggering a recompile I got an error, the WebService annotation couldn't be found. Checked the pom.xml again, but it contained the correct dependency. So what was going wrong. Afer trying out some suggestions made in various blogs and forum posts, the problem remained. I even got to a point in which the code compiled, but when I executed the wsgen goal it would fail with the message: java.lang.NoClassDefFoundError: javax/jws/WebService. What? How can this be? By this time I was cursing away at Maven again: how can this bloody thing, that is supposed to make my life easier, always fuck up in the most incomprehensible way possible?

Then a little speck of light in the distance: it seems that the JAX-WS libraries are spread over a maven1 and a maven2 repository and that there are problems with some versions and references to other libraries that JAX-WS uses. The solution to all this:
  • empty out your local repository
  • add the java.net maven1 and maven2 repositories to your pom.xml
  • add a dependency to JSR-181 to the wsgen plugin
After doing all this I was able to compile, build and generate everything I wanted (it only took way too much time).

The pom.xml I used:


xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
4.0.0
EDRL
Preprocessor
0.0.1-SNAPSHOT



com.sun.tools.jxc.maven2
maven-jaxb-schemagen-plugin
1.1


process-sources

${project.build.directory}/schemas

be/test/*.java

${basedir}/src/main/java
true


generate





org.codehaus.mojo
jaxws-maven-plugin
1.12



wsgen

generate-sources

be.test.service.TestService
true
true
true






javax.jws
jsr181-api
1.0-MR1







maven-repository.dev.java.net
Java.net Repository for Maven 1
http://download.java.net/maven/1/
legacy


maven2-repository.dev.java.net
Java.net Maven 2 Repository
http://download.java.net/maven/2




maven2-repository.dev.java.net
Java.net Maven 2 Repository
http://download.java.net/maven/2




javax.xml
jaxb-impl
2.1


javax.xml.bind
jaxb-api
2.2


com.sun.xml.ws
jaxws-rt
2.1.3


javax.jws
jsr181-api
1.0-MR1



1.5
1.5




Sunday 2 August 2009

Better validation messages in JSF

This week a collegue of mine wanted to enhance the form validation messages in a JSF application, but he was having problems getting a nice fieldname into the message. The default behavior of JSF is to display something like: j_id_id19: Validation Error: Value is required. This is Ok for a programmer, but an end-user gets kinda confused. There are several solutions for this problem:
  • Overwrite the default JSF messages in Messages.properties to not include the field name, but this will make the problem even worse.
  • Write a custom renderer for the messages, but this is about the same as amputating a leg when you've hit your toe against the bed.
  • Or we could just write a correct HTML form element that has, and this is the important bit, a label attribute, as this is what JSF will use in its default validation message. When this isn't present it will default to the id and when this id auto generated, as it usually is, will show something like j_id_id19.
So my collegue's problem was easily solved by adding label attributes to the fields of his form.

PHP error: Illegal offset type for array

I was working on my parents in law house renting site in Spain, www.casacalida.es, this weekend trying to integrate some weather site data that is provided in XML format, when I ran into a PHP error that I couldn't figure out at first. I'll elaborate some more about the integration of the XML data into their CodeIgniter site later. The error seems to indicate that my multidimensional array keys were wrong somewhere, but I couldn't figure it out at first.

Some searching brought me to this post, that seemed to confirm my suspicion. Only I was using all string keys and not integers, but Maybe I could adopt some of the solution. So I added a string cast for my array key and hey presto my problem was solved.

Getting SSIS to like Sharepoint XMLs

I'm currently working on a project in which we need to get some content from a Sharepoint instance into a SQL server database. The work is being done by another consultant using Microsoft SSIS. The simplest way to get the data from one side to the other would be to do a straight database to database data synchronization. The problem with this plan is that the Sharepoint database schema just plain sucks and that Microsoft doesn't recommend accessing it directly, not just because of the crappy schema, but also because it could change between Sharepoint versions.

So plan B is to use the abundantly available Sharepoint webservices (if you could call them webservices). Since SSIS has support for webservices we tried that and failed again. We found no way to get SSIS to understand the Sharepoint webservices, it just kept complaining about the WSDL and the XSD. So Googling gave us an alternate solution: create some stubs around the webservice and use those in a script task. This got use a bit further but then SSIS started complaining about the XML using multiple namespaces and it also couldn't handle the XHTML content in some tags.

But again we found a workaround. Using the following XSLT to preprocess the XML before using it made SSIS accept it. The XSL removes all namespaces and places the XHTML content of selected tags, by means of a xsl:param in CDATA sections. One last problem was that SSIS doesn't give you the means to provide an XSL with params, so you'll have to put them directly into the content of the xsl:param.


<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:output method="xml" />

<xsl:param name="rtfNodes">TagsName1~TagsName1</xsl:param>

<xsl:template match="*">
<xsl:choose>
<xsl:when test="contains($rtfNodes, local-name())">
<xsl:element name="{local-name()}">
<xsl:apply-templates select="@*" />
<xsl:text disable-output-escaping="yes"><![CDATA[ <![CDATA[ ]]></xsl:text>
<xsl:copy-of select="node()"/>
<xsl:text disable-output-escaping="yes"><![CDATA[]]]]><![CDATA[>]]></xsl:text>
</xsl:element>
</xsl:when>
<xsl:otherwise>
<xsl:element name="{local-name()}">
<xsl:apply-templates select="@*" />
<xsl:apply-templates select="node()" />
</xsl:element>
</xsl:otherwise>
</xsl:choose>
</xsl:template>

<xsl:template match="@*">
<xsl:attribute name="{local-name()}">
<xsl:value-of select="."/>
</xsl:attribute>
</xsl:template>

<xsl:template match="/">
<xsl:copy>
<xsl:apply-templates />
</xsl:copy>
</xsl:template>

<xsl:template match="comment() | processing-instruction() | text()">
<xsl:copy />
</xsl:template>

</xsl:stylesheet>