Connecting to Microsoft Dynamics CRM 4.0 Webservices with Java – Lessons Learned

It all started several years ago. Someone mentioned that our internal CRM instance had a web service that we could use to automate some processes. Thus began an interesting journey, and here are some lessons learned. Much of this is from memory, so I’m sure I’ve missed some things.

The first thing I did was try to generate the client using Axis 1, as that we used at the time (around 2009-2010). I quickly found that because of the authentication, I could not even connect to the remote instance and generate the proxies. After some more trial and error, I figured out that I had to actually download the WSDL and generate my proxies based on the local file. This got me started, but when I actually got the code imported into Eclipse, I could not get it to actually work. I kept getting http 401 errors. This is because of the NTLM authentication CRM uses. I spent more R&D time fighting to get NTLM to work with Axis 1 to no avail. I read somewhere that Axis 2 works with NTLM, so I gave it a shot.

Now, full disclaimer – I’m no Axis expert. However, despite several hours of tinkering, I could not even get Axis 2 to generate anything meaningful. The classes were broken\wouldn’t compile, many things seemed to be missing, etc. So in the end I gave up and moved on to more fruitful work. The project sat for 6 to 9 months.

While this project sat, I started learning about CXF. I did a couple of successful clients with it and got a general feel for how it worked. I experimented and also learned how to use SoapUI. I did a little more research and found that it isn’t that hard to do authentication with CXF. So I picked up the project again and this time I was able to get the code to generate and shortly thereafter I got authentication working…or so I thought.

private void setupSecurity( final CrmServiceSoap crmServiceSoap ) {
//Turn off chunking so that NTLM can occur
Client client = ClientProxy.getClient( crmServiceSoap );
HTTPConduit http = (HTTPConduit)client.getConduit();
HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
httpClientPolicy.setConnectionTimeout( 36000 );
httpClientPolicy.setAllowChunking( false );

http.setClient( httpClientPolicy );

AuthorizationPolicy authorization = new AuthorizationPolicy();
authorization.setUserName( _userName );
authorization.setPassword( _password );

http.setAuthorization( authorization );
}

This worked fine for some months, allowing us to do almost everything that we needed. We were able to create cases , accounts, run queries, etc. The only real glitch was that we had to keep a read only copy of the wsdl on an open web server and our client had to point at it, instead of the one behind the CRM authentication wall.

Fast forward to a couple of weeks ago (September 2012). One thing we hadn’t tried was updating records. No we found a use case for that too. However, this time it didn’t work. We used the same login and the same code that was working before to no avail. As it turns out, the security was actually using the local active directory user. So when we tried to update records that the user did not own, we’d get something like this:

Payload: Server was unable to process request. ---> Exception has been thrown by the target of an invocation. ---> SecLib::AccessCheckEx failed. Returned hr = -2147187962, ObjectID: 949161c6-2333-dd11-8014-001125a94557, OwningUser: de58855c-482c-db11-889e-001125a94557 and CallingUser: b7c87416-e895-db11-96c9-001125a94557

After wiresharking this, I found something interesting… In the http headers, the Authentication value was set to: “Basic xxxxxx-a-bunch-of-data”. Wait…that’s not NTLM?!? As it turns out, NTLM authentication is a bit of a pain to setup. Furthermore, it isn’t really supported in CXF. Supposedly you can do it like so, but I could not get this to work. I’m not 100% sure why, but I think it is because jcifs doesn’t work out of the box with Windows 7. I played with Jespa a bit, even downloading the source code, but I could not get it to work either.

Getting desperate, I figured that I was able to get this to work in SoapUI. And I knew SoapUI is java & open source. Digging through the SoapUI code a bit more, I found that it in fact uses a fairly vanilla http client, rather than a whole web service stack. After a bit of poking, it turns out that the CXF http client is non-extensible\swappable.

So, to make it work I turned to a bit of a hack…I used the DefaultHttpClient as well:

//@formatter:off
private static final String ACCOUNT_UPDATE_SOAP_TEMPLATE =
"    <soap:Envelope xmlns:soap=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:web=\"http://schemas.microsoft.com/crm/2006/WebServices\" xmlns:cor=\"http://schemas.microsoft.com/crm/2006/CoreTypes\">  " +
"       <soap:Body>" +
"            <entity xmlns=\"http://schemas.microsoft.com/crm/2006/WebServices\" xmlns:ns2=\"http://schemas.microsoft.com/crm/2006/Query\" xmlns:ns3=\"http://schemas.microsoft.com/crm/2006/CoreTypes\" xmlns:ns4=\"http://schemas.microsoft.com/crm/2006/Scheduling\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:type=\"account\">" +
"                <accountid>%s</accountid>" +
"                <customfield1>%s</customfield1>" +
"                <customfield2>%s<customfield2>" +
"            </entity>" +
"       </soap:Body>" +
"    </soap:Envelope>";
//@formatter:on

Account account = (Account)entity;

//TODO - hack! CXF's default http client does not seem to work with NTLM authentication
// found this workaround here: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html
DefaultHttpClient httpClient = new DefaultHttpClient();

NTCredentials ntCredentials = new NTCredentials( _userName, _password, "", "mydomain" );
httpClient.getCredentialsProvider().setCredentials( AuthScope.ANY, ntCredentials );

HttpPost request = new HttpPost( "/MSCrmServices/2006/CrmService.asmx" );

String soapMessage =
String.format( ACCOUNT_UPDATE_SOAP_TEMPLATE, account.getAccountid().getValue(), account.getCustomField1(),
account.getCustomField2() );

LOGGER.info( String.format( "Preparing to send soap message: %s", soapMessage ) );

request.setEntity( new StringEntity( soapMessage, ContentType
.create( "application/soap+xml;charset=UTF-8;action=\"http://schemas.microsoft.com/crm/2006/WebServices/Update\"" ) ) );
try {
HttpResponse response = httpClient.execute( new HttpHost( _hostname ), request );
LOGGER.info( response.toString() );
}
catch ( Exception exception ) {
throw new RuntimeException( exception );
}

So the battle is over for now, but the war is not yet won. I hope this helps someone else.

Update #1

After days of working with one of the CXF developers I think we’ve got NTLM authentication working natively with CXF. That last jira link: CXF http client is non-extensible\swappable turned out to be the entry point. One of the dev’s mentioned that the task was in fact complete in the latest\trunk version (2.7.0-SNAPSHOT). The only problem was that the client was buried deep in the API, so I opened up a new ticket to expose the client. Within an afternoon or so it was already done! Lo and behold, I tried it out and after a little bit of mucking around I got it to run, but it was throwing exceptions. After a couple of days more back and forth the finished code has arrived:

Bus bus = BusFactory.getDefaultBus();
bus.setProperty( "use.async.http.conduit", "true" );

Client client = ClientProxy.getClient( proxy );
HTTPConduit http = (HTTPConduit)client.getConduit();
if ( http instanceof AsyncHTTPConduit ) {
AsyncHTTPConduit conduit = (AsyncHTTPConduit)http;
DefaultHttpAsyncClient defaultHttpAsyncClient;
try {
defaultHttpAsyncClient = conduit.getHttpAsyncClient();
}
catch ( IOException exception ) {
throw new RuntimeException( exception );
}
defaultHttpAsyncClient.getCredentialsProvider().setCredentials( AuthScope.ANY,
new NTCredentials( "username", "password", "", "domain" ) );

conduit.getClient().setAllowChunking( false );
conduit.getClient().setAutoRedirect( true );
}

Much thanks to Daniel Kulp, for getting this done. This was a great open source experience!