SOAP UI multiple hosts

Here’s a quick tip: If you use SOAP UI, you can have multiple environments\URLs setup by using placeholder properties.

On the project, go to properties -> custom properties. From there, you can add a property like “host” for example. You can then have a “qaHost” and “prodHost” properties, which can reference other properties. For example, you might set qaHost=qa.foo.com and prodHost=foo.com. You can then set host=#{#Project#qaHost}. You then set your endpoint URL to http://#{Project#host}/service/foo/bar. Now when you want to test out multiple hosts, you can just toggle the property, instead of needing a project for each, allowing you to keep the same requests and setup for each.

I hope this helps someone.

CXF WS Client With Fluent Builders and More!

Alright, I’ve been using SoapUI to generated my CXF WS clients for a little while now. It’s a pretty good tool, but ultimately if you’re using it to generate CXF, you might be able to skip SoapUI if you’re willing to indulge in some maven goodness. I started down the maven path with this because I wanted to use xjc plugins for the generated jaxb classes. Specifically, I wanted to use the fluent builders.

I ran SoapUI and collected the arguments. I then converted them over to the maven form, giving me this:

<plugins>
<plugin>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-codegen-plugin</artifactId>
<version>${cxf-version}</version>
<executions>
<execution>
<id>generate-sources</id>
<phase>generate-sources</phase>
<configuration>
<wsdlOptions>
<wsdlOption>
<wsdl>${basedir}/src/main/resources/wsdl/client.wsdl</wsdl>
<extraargs>
<extraarg>-p</extraarg>
<extraarg>${project.groupId}.${project.artifactId}.gen</extraarg>
<extraarg>-b</extraarg>
<extraarg>${basedir}/src/main/resources/wsdl/bindings.xml</extraarg>
<extraarg>-client</extraarg>
<extraarg>-exsh</extraarg>
<extraarg>false</extraarg>
<extraarg>-dns</extraarg>
<extraarg>true</extraarg>
<extraarg>-dex</extraarg>
<extraarg>true</extraarg>
<extraarg>-verbose</extraarg>
<extraarg>-xjc-npa</extraarg>
<!-- OPTIONAL: generate toString methods -->
<extraarg>-xjc-Xts</extraarg>
<!-- OPTIONAL: use the fluent API -->
<extraarg>-xjc-Xfluent-api</extraarg>
<!-- OPTIONAL: create value constructors -->
<extraarg>-xjc-Xvalue-constructor</extraarg>
</extraargs>
</wsdlOption>
</wsdlOptions>
</configuration>
<goals>
<goal>wsdl2java</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-xjc-ts</artifactId>
<version>2.2.12</version>
</dependency>
<dependency>
<groupId>org.jvnet.jaxb2_commons</groupId>
<artifactId>jaxb2-value-constructor</artifactId>
<version>3.0</version>
</dependency>
<dependency>
<groupId>org.jvnet.jaxb2_commons</groupId>
<artifactId>jaxb2-fluent-api</artifactId>
<version>3.0</version>
</dependency>
<dependency>
<groupId>org.jvnet.jaxb2_commons</groupId>
<artifactId>jaxb2-basics</artifactId>
<version>0.6.4</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<artifactId>maven-antrun-plugin</artifactId>
<version>1.6</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<copy todir="src/main/java">
<fileset dir="${project.build.directory}/generated-sources/cxf" />
</copy>
</target>
</configuration>
</execution>
</executions>
</plugin>
</plugins>

I know that’s alot of pasta, but what you get is a single command run that will (re)generate your CXF WS client, putting them back in to your java source directory.

Here is my bindings file:

<jaxb:bindings version="2.1"
xmlns:jaxb="http://java.sun.com/xml/ns/jaxb"
xmlns:xjc="http://java.sun.com/xml/ns/jaxb/xjc"
xmlns:xs="http://www.w3.org/2001/XMLSchema">
<jaxb:globalBindings generateElementProperty="false"/>
</jaxb:bindings> 

You’re on your own for your wsdl, just remember to drop it in src/main/resources/wsdl

Connecting to Microsoft Dynamics CRM 4.0 Webservices with Java – Lessons Learned

It all started several years ago. Someone mentioned that our internal CRM instance had a web service that we could use to automate some processes. Thus began an interesting journey, and here are some lessons learned. Much of this is from memory, so I’m sure I’ve missed some things.

The first thing I did was try to generate the client using Axis 1, as that we used at the time (around 2009-2010). I quickly found that because of the authentication, I could not even connect to the remote instance and generate the proxies. After some more trial and error, I figured out that I had to actually download the WSDL and generate my proxies based on the local file. This got me started, but when I actually got the code imported into Eclipse, I could not get it to actually work. I kept getting http 401 errors. This is because of the NTLM authentication CRM uses. I spent more R&D time fighting to get NTLM to work with Axis 1 to no avail. I read somewhere that Axis 2 works with NTLM, so I gave it a shot.

Now, full disclaimer – I’m no Axis expert. However, despite several hours of tinkering, I could not even get Axis 2 to generate anything meaningful. The classes were broken\wouldn’t compile, many things seemed to be missing, etc. So in the end I gave up and moved on to more fruitful work. The project sat for 6 to 9 months.

While this project sat, I started learning about CXF. I did a couple of successful clients with it and got a general feel for how it worked. I experimented and also learned how to use SoapUI. I did a little more research and found that it isn’t that hard to do authentication with CXF. So I picked up the project again and this time I was able to get the code to generate and shortly thereafter I got authentication working…or so I thought.

private void setupSecurity( final CrmServiceSoap crmServiceSoap ) {
//Turn off chunking so that NTLM can occur
Client client = ClientProxy.getClient( crmServiceSoap );
HTTPConduit http = (HTTPConduit)client.getConduit();
HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
httpClientPolicy.setConnectionTimeout( 36000 );
httpClientPolicy.setAllowChunking( false );

http.setClient( httpClientPolicy );

AuthorizationPolicy authorization = new AuthorizationPolicy();
authorization.setUserName( _userName );
authorization.setPassword( _password );

http.setAuthorization( authorization );
}

This worked fine for some months, allowing us to do almost everything that we needed. We were able to create cases , accounts, run queries, etc. The only real glitch was that we had to keep a read only copy of the wsdl on an open web server and our client had to point at it, instead of the one behind the CRM authentication wall.

Fast forward to a couple of weeks ago (September 2012). One thing we hadn’t tried was updating records. No we found a use case for that too. However, this time it didn’t work. We used the same login and the same code that was working before to no avail. As it turns out, the security was actually using the local active directory user. So when we tried to update records that the user did not own, we’d get something like this:

Payload: Server was unable to process request. ---> Exception has been thrown by the target of an invocation. ---> SecLib::AccessCheckEx failed. Returned hr = -2147187962, ObjectID: 949161c6-2333-dd11-8014-001125a94557, OwningUser: de58855c-482c-db11-889e-001125a94557 and CallingUser: b7c87416-e895-db11-96c9-001125a94557

After wiresharking this, I found something interesting… In the http headers, the Authentication value was set to: “Basic xxxxxx-a-bunch-of-data”. Wait…that’s not NTLM?!? As it turns out, NTLM authentication is a bit of a pain to setup. Furthermore, it isn’t really supported in CXF. Supposedly you can do it like so, but I could not get this to work. I’m not 100% sure why, but I think it is because jcifs doesn’t work out of the box with Windows 7. I played with Jespa a bit, even downloading the source code, but I could not get it to work either.

Getting desperate, I figured that I was able to get this to work in SoapUI. And I knew SoapUI is java & open source. Digging through the SoapUI code a bit more, I found that it in fact uses a fairly vanilla http client, rather than a whole web service stack. After a bit of poking, it turns out that the CXF http client is non-extensible\swappable.

So, to make it work I turned to a bit of a hack…I used the DefaultHttpClient as well:

//@formatter:off
private static final String ACCOUNT_UPDATE_SOAP_TEMPLATE =
"    <soap:Envelope xmlns:soap=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:web=\"http://schemas.microsoft.com/crm/2006/WebServices\" xmlns:cor=\"http://schemas.microsoft.com/crm/2006/CoreTypes\">  " +
"       <soap:Body>" +
"            <entity xmlns=\"http://schemas.microsoft.com/crm/2006/WebServices\" xmlns:ns2=\"http://schemas.microsoft.com/crm/2006/Query\" xmlns:ns3=\"http://schemas.microsoft.com/crm/2006/CoreTypes\" xmlns:ns4=\"http://schemas.microsoft.com/crm/2006/Scheduling\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:type=\"account\">" +
"                <accountid>%s</accountid>" +
"                <customfield1>%s</customfield1>" +
"                <customfield2>%s<customfield2>" +
"            </entity>" +
"       </soap:Body>" +
"    </soap:Envelope>";
//@formatter:on

Account account = (Account)entity;

//TODO - hack! CXF's default http client does not seem to work with NTLM authentication
// found this workaround here: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html
DefaultHttpClient httpClient = new DefaultHttpClient();

NTCredentials ntCredentials = new NTCredentials( _userName, _password, "", "mydomain" );
httpClient.getCredentialsProvider().setCredentials( AuthScope.ANY, ntCredentials );

HttpPost request = new HttpPost( "/MSCrmServices/2006/CrmService.asmx" );

String soapMessage =
String.format( ACCOUNT_UPDATE_SOAP_TEMPLATE, account.getAccountid().getValue(), account.getCustomField1(),
account.getCustomField2() );

LOGGER.info( String.format( "Preparing to send soap message: %s", soapMessage ) );

request.setEntity( new StringEntity( soapMessage, ContentType
.create( "application/soap+xml;charset=UTF-8;action=\"http://schemas.microsoft.com/crm/2006/WebServices/Update\"" ) ) );
try {
HttpResponse response = httpClient.execute( new HttpHost( _hostname ), request );
LOGGER.info( response.toString() );
}
catch ( Exception exception ) {
throw new RuntimeException( exception );
}

So the battle is over for now, but the war is not yet won. I hope this helps someone else.

Update #1

After days of working with one of the CXF developers I think we’ve got NTLM authentication working natively with CXF. That last jira link: CXF http client is non-extensible\swappable turned out to be the entry point. One of the dev’s mentioned that the task was in fact complete in the latest\trunk version (2.7.0-SNAPSHOT). The only problem was that the client was buried deep in the API, so I opened up a new ticket to expose the client. Within an afternoon or so it was already done! Lo and behold, I tried it out and after a little bit of mucking around I got it to run, but it was throwing exceptions. After a couple of days more back and forth the finished code has arrived:

Bus bus = BusFactory.getDefaultBus();
bus.setProperty( "use.async.http.conduit", "true" );

Client client = ClientProxy.getClient( proxy );
HTTPConduit http = (HTTPConduit)client.getConduit();
if ( http instanceof AsyncHTTPConduit ) {
AsyncHTTPConduit conduit = (AsyncHTTPConduit)http;
DefaultHttpAsyncClient defaultHttpAsyncClient;
try {
defaultHttpAsyncClient = conduit.getHttpAsyncClient();
}
catch ( IOException exception ) {
throw new RuntimeException( exception );
}
defaultHttpAsyncClient.getCredentialsProvider().setCredentials( AuthScope.ANY,
new NTCredentials( "username", "password", "", "domain" ) );

conduit.getClient().setAllowChunking( false );
conduit.getClient().setAutoRedirect( true );
}

Much thanks to Daniel Kulp, for getting this done. This was a great open source experience!

The Pain of WSSE & WCF…Lessons Learned

Ok, for some reason I volunteered to “expand my knowledge” by taking on this project that would predominately be in .Net. Admittedly, I’ve learned some things, but not all of it has been that fun – this issue in particular.

The vendor that I am\was working with uses WSSE security aka WS-Security for their web services. I’m not sure if this standard is antiquated or Microsoft just doesn’t blend well with it, or maybe I don’t know the .Net framework all that well but this was nothing but a hassle. I’ve even gone so far as to ask on StackOverflow about this and as of 4/4/2012, there are no answers. If you want more technical details, go there and check it out – it might melt your brain a little bit. I know it melted mine.

So what did I learn from all of this?

Well, for starters, Fiddler kicks a$$ 🙂 I’ve used wireshark before this. I’ve even played around with a firefox plugin called tamperdata that allows you to modify http requests fairly easily. So I am no stranger to these types of tools. However, when I first heard about Fiddler from a co-worker\friend I checked it out and didn’t think much of it – partly because I didn’t quite grok the UI right away. But this time was different. I NEEDED to be able to see what was going on. I needed to be able to see those http requests flying out the intertubes in realtime. Once I got used to it, Fiddler really rocked.

What else? Well, this one is fairly obvious… SoapUI. I’ve been progressing my skills with this tool for a while now, but I gotta say, without it, I don’t know what I would have done. It was really great to be able to hand SoapUI projects back and forth between the vendor contact and myself. The whole project! When I had an issue, I could export the project and email it to him – he could load it and test it and we were able to solve quite a few problems this way.

As I mentioned in my StackOverflow question, in the end I got this to work in java (with CXF+WSSJ) pretty quickly, but I am still puzzled and dismayed that I could not get it to work in .Net. I guess I have to chalk this one up to lessons learned.

Simple SOAP Webservice Client with CXF

I’m sure this has probably been documented many times, so this is more for my own documentation than anything, so here goes:

1. Download Apache CXF. Make sure to get the binaries not the source version.
2. Unpack the download and put it somewhere on disk (C:\Program Files, \opt, etc)
3. Add the path to the bin folder to your OS path.
4. Run the wsdl2java script:

wsdl2java -client -fe jaxws21 -d C:\Users\mynamehere\Desktop\generated-webservice-client http://someservername/someservice.svc?wsdl

Here is the documentation to the tool:
http://cxf.apache.org/docs/wsdl-to-java.html

A couple of notes:

1. You don’t really have to have the ‘-client’ param, it just gives a starting point for client code.
2. If you’re using < java 7, you absolutely need the ‘-fe jaxws21’ param, as it will generate code that will not compile for java 5 & 6.

UPDATE #1:

I recently used this and was getting this error:
“WSDLToJava Error: Thrown by JAXB : A class/interface with the same name “*************” is already in use. Use a class customization to resolve this conflict.”

A quick google search landed me here.

By adding this flag: “-autoNameResolution” I got it to work, so here is my updated command:

wsdl2java -autoNameResolution -client -fe jaxws21 -d C:\Users\mynamehere\Desktop\generated-webservice-client http://someservername/someservice.svc?wsdl

UPDATE #2:

After another client generation I opened up the new source files to find that many of the fields in them were

JAXBElement<String>

A quick google search turned up this helpful stackoverflow post.

As the solution to that question suggests, save a file bindings.txt with this in it:

<jaxb:bindings version="2.1"
xmlns:jaxb="http://java.sun.com/xml/ns/jaxb"
xmlns:xjc="http://java.sun.com/xml/ns/jaxb/xjc"
xmlns:xs="http://www.w3.org/2001/XMLSchema">
<jaxb:globalBindings generateElementProperty="false"/>
</jaxb:bindings>

The updated script now should look something like this:

wsdl2java -b path-to-bindings.txt -autoNameResolution -client -fe jaxws21 -d C:\Users\mynamehere\Desktop\generated-webservice-client http://someservername/someservice.svc?wsdl

CXF Server Example in 5 Minutes

Ok, I think I’ve done this before, and most of this is derived from the CXF example found here. However…

1. Whenever I put this stuff down for a while, I always have to spend some time re-figuring-it-out. Therefore, putting it here will hopefully shorten that time next time I have to come back to this.
2. There example is not interface, and therefore service oriented. I think they have another example somewhere that is, but I can’t ever seem to find it.

All of this being said, here it is…

1) You need these dependencies in your pom:

<dependency>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-rt-transports-http</artifactId>
<version>2.5.2</version>
</dependency>
<dependency>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-rt-transports-http-jetty</artifactId>
<version>2.5.2</version>
</dependency>
<dependency>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-rt-frontend-jaxrs</artifactId>
<version>2.5.2</version>
</dependency>
<dependency>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
<version>3.1</version>
</dependency>
<dependency>
<groupId>javax.ws.rs</groupId>
<artifactId>jsr311-api</artifactId>
<version>1.1.1</version>
</dependency>

2) Here are the java source classes:

public interface Echo {
@GET
@Path( "/echo/{input}" )
@Produces( "text/plain" )
public String echo( @PathParam( "input" ) String input );
}

public class EchoService implements Echo {
public String echo( final String input ) {
return "Hello, world! You said: " + input;
}
}

public class Server {
public Server() throws Exception {
JAXRSServerFactoryBean sf = new JAXRSServerFactoryBean();
sf.setResourceClasses( Echo.class );
sf.setResourceProvider( Echo.class, new SingletonResourceProvider( new EchoService() ) );
sf.setAddress( "http://localhost:9000/" );

sf.create();
}

public static void main( final String args[] ) throws Exception {
new Server();
System.out.println( "Server ready..." );

System.in.read();
System.out.println( "Server exiting" );
System.exit( 0 );
}
}

Now all you have to do is run the Server class and open a browser to: http://localhost:9000/echo/bar

And you should see the message: “Hello, world! You said: bar”