Connecting to Microsoft Dynamics CRM 4.0 Webservices with Java – Lessons Learned

It all started several years ago. Someone mentioned that our internal CRM instance had a web service that we could use to automate some processes. Thus began an interesting journey, and here are some lessons learned. Much of this is from memory, so I’m sure I’ve missed some things.

The first thing I did was try to generate the client using Axis 1, as that we used at the time (around 2009-2010). I quickly found that because of the authentication, I could not even connect to the remote instance and generate the proxies. After some more trial and error, I figured out that I had to actually download the WSDL and generate my proxies based on the local file. This got me started, but when I actually got the code imported into Eclipse, I could not get it to actually work. I kept getting http 401 errors. This is because of the NTLM authentication CRM uses. I spent more R&D time fighting to get NTLM to work with Axis 1 to no avail. I read somewhere that Axis 2 works with NTLM, so I gave it a shot.

Now, full disclaimer – I’m no Axis expert. However, despite several hours of tinkering, I could not even get Axis 2 to generate anything meaningful. The classes were broken\wouldn’t compile, many things seemed to be missing, etc. So in the end I gave up and moved on to more fruitful work. The project sat for 6 to 9 months.

While this project sat, I started learning about CXF. I did a couple of successful clients with it and got a general feel for how it worked. I experimented and also learned how to use SoapUI. I did a little more research and found that it isn’t that hard to do authentication with CXF. So I picked up the project again and this time I was able to get the code to generate and shortly thereafter I got authentication working…or so I thought.

private void setupSecurity( final CrmServiceSoap crmServiceSoap ) {
//Turn off chunking so that NTLM can occur
Client client = ClientProxy.getClient( crmServiceSoap );
HTTPConduit http = (HTTPConduit)client.getConduit();
HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
httpClientPolicy.setConnectionTimeout( 36000 );
httpClientPolicy.setAllowChunking( false );

http.setClient( httpClientPolicy );

AuthorizationPolicy authorization = new AuthorizationPolicy();
authorization.setUserName( _userName );
authorization.setPassword( _password );

http.setAuthorization( authorization );
}

This worked fine for some months, allowing us to do almost everything that we needed. We were able to create cases , accounts, run queries, etc. The only real glitch was that we had to keep a read only copy of the wsdl on an open web server and our client had to point at it, instead of the one behind the CRM authentication wall.

Fast forward to a couple of weeks ago (September 2012). One thing we hadn’t tried was updating records. No we found a use case for that too. However, this time it didn’t work. We used the same login and the same code that was working before to no avail. As it turns out, the security was actually using the local active directory user. So when we tried to update records that the user did not own, we’d get something like this:


Payload: Server was unable to process request. ---> Exception has been thrown by the target of an invocation. ---> SecLib::AccessCheckEx failed. Returned hr = -2147187962, ObjectID: 949161c6-2333-dd11-8014-001125a94557, OwningUser: de58855c-482c-db11-889e-001125a94557 and CallingUser: b7c87416-e895-db11-96c9-001125a94557

After wiresharking this, I found something interesting… In the http headers, the Authentication value was set to: “Basic xxxxxx-a-bunch-of-data”. Wait…that’s not NTLM?!? As it turns out, NTLM authentication is a bit of a pain to setup. Furthermore, it isn’t really supported in CXF. Supposedly you can do it like so, but I could not get this to work. I’m not 100% sure why, but I think it is because jcifs doesn’t work out of the box with Windows 7. I played with Jespa a bit, even downloading the source code, but I could not get it to work either.

Getting desperate, I figured that I was able to get this to work in SoapUI. And I knew SoapUI is java & open source. Digging through the SoapUI code a bit more, I found that it in fact uses a fairly vanilla http client, rather than a whole web service stack. After a bit of poking, it turns out that the CXF http client is non-extensible\swappable.

So, to make it work I turned to a bit of a hack…I used the DefaultHttpClient as well:


//@formatter:off
private static final String ACCOUNT_UPDATE_SOAP_TEMPLATE =
"    <soap:Envelope xmlns:soap=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:web=\"http://schemas.microsoft.com/crm/2006/WebServices\" xmlns:cor=\"http://schemas.microsoft.com/crm/2006/CoreTypes\">  " +
"       <soap:Body>" +
"            <entity xmlns=\"http://schemas.microsoft.com/crm/2006/WebServices\" xmlns:ns2=\"http://schemas.microsoft.com/crm/2006/Query\" xmlns:ns3=\"http://schemas.microsoft.com/crm/2006/CoreTypes\" xmlns:ns4=\"http://schemas.microsoft.com/crm/2006/Scheduling\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:type=\"account\">" +
"                <accountid>%s</accountid>" +
"                <customfield1>%s</customfield1>" +
"                <customfield2>%s<customfield2>" +
"            </entity>" +
"       </soap:Body>" +
"    </soap:Envelope>";
//@formatter:on

Account account = (Account)entity;

//TODO - hack! CXF's default http client does not seem to work with NTLM authentication
// found this workaround here: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html
DefaultHttpClient httpClient = new DefaultHttpClient();

NTCredentials ntCredentials = new NTCredentials( _userName, _password, "", "mydomain" );
httpClient.getCredentialsProvider().setCredentials( AuthScope.ANY, ntCredentials );

HttpPost request = new HttpPost( "/MSCrmServices/2006/CrmService.asmx" );

String soapMessage =
String.format( ACCOUNT_UPDATE_SOAP_TEMPLATE, account.getAccountid().getValue(), account.getCustomField1(),
account.getCustomField2() );

LOGGER.info( String.format( "Preparing to send soap message: %s", soapMessage ) );

request.setEntity( new StringEntity( soapMessage, ContentType
.create( "application/soap+xml;charset=UTF-8;action=\"http://schemas.microsoft.com/crm/2006/WebServices/Update\"" ) ) );
try {
HttpResponse response = httpClient.execute( new HttpHost( _hostname ), request );
LOGGER.info( response.toString() );
}
catch ( Exception exception ) {
throw new RuntimeException( exception );
}

So the battle is over for now, but the war is not yet won. I hope this helps someone else.

Update #1

After days of working with one of the CXF developers I think we’ve got NTLM authentication working natively with CXF. That last jira link: CXF http client is non-extensible\swappable turned out to be the entry point. One of the dev’s mentioned that the task was in fact complete in the latest\trunk version (2.7.0-SNAPSHOT). The only problem was that the client was buried deep in the API, so I opened up a new ticket to expose the client. Within an afternoon or so it was already done! Lo and behold, I tried it out and after a little bit of mucking around I got it to run, but it was throwing exceptions. After a couple of days more back and forth the finished code has arrived:

Bus bus = BusFactory.getDefaultBus();
bus.setProperty( "use.async.http.conduit", "true" );

Client client = ClientProxy.getClient( proxy );
HTTPConduit http = (HTTPConduit)client.getConduit();
if ( http instanceof AsyncHTTPConduit ) {
AsyncHTTPConduit conduit = (AsyncHTTPConduit)http;
DefaultHttpAsyncClient defaultHttpAsyncClient;
try {
defaultHttpAsyncClient = conduit.getHttpAsyncClient();
}
catch ( IOException exception ) {
throw new RuntimeException( exception );
}
defaultHttpAsyncClient.getCredentialsProvider().setCredentials( AuthScope.ANY,
new NTCredentials( "username", "password", "", "domain" ) );

conduit.getClient().setAllowChunking( false );
conduit.getClient().setAutoRedirect( true );
}

Much thanks to Daniel Kulp, for getting this done. This was a great open source experience!

About these ads

12 Responses to “Connecting to Microsoft Dynamics CRM 4.0 Webservices with Java – Lessons Learned”

  1. Jagadish KB Says:

    This post was such a boon.. :-) I was struggling with this same issue for the past 1 week… To make NTLM auth work with cxf I read up & almost tried about everything.. how NTLM works, how its supported in Java 6 and above, issue with jcifs and java 7.. had almost searched all topics and finally found this blog.. Thanks a ton for this post! Finally it working.. !
    have a suggestion though it would have been easier if there was link to the entire sample demo source code as there are some classes in the above source which are found even in other packages which fail to compile if included wrongly. So had to do a bit more searchin to find their exact packages where they reside. Nonetheless finally I got a solution and I really appreciate your effort in getting this blog. Thanks again!

    • Shaun Elliott Says:

      You’re welcome. It is not a bad idea to possibly link to the code rather than include it inline. However, the fully functional versions are part of proprietary company code, so I can’t put out all of the code unfortunately.

  2. Hi there,

    So…. one question in the first solution you posted here before the CXF fix, you just were sending the request using http client right or am I misunderstanding something.

    • Shaun Elliott Says:

      Yeah, I was manually sending the soap over http via apache http. See this snippet:

      // found this workaround here: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html
      DefaultHttpClient httpClient = new DefaultHttpClient();

      This is really only necessary if you’re using java < 1.6. If you're using 1.6+, then you can just use CXF.

      Best of luck!

      • Thanks a lot mate that’s what I thought.
        Sorry to keep disturbing BUT, have you try using CXF to hit Dynamics CRM 2011?

        If yes could you please tell me which CXF version were you using?

        Again, thanks a lot.

      • Shaun Elliott Says:

        I’ve used dynamics 4.0.7333.3414, I’m not sure if that is 2011 or not – though, it seems about right. It should work with CXF 2.7+

  3. Holy crap I can’t thank you enough. I’ve been digging around this topic for 2 weeks. Can’t believe I’ve actually got it working based on your approach, hallelujah!!!

    This blog post needs to be ranked at the top of Google’s list!

  4. Isaí Arenas Soto Says:

    Wow, Thanks Shaun.

    I had been working some similar about 1 week. And when I saw your code I could rest.

    Regards

  5. Hello ,

    I am planning on implementing NTLM in fuse 6.0 using Spring DSL.
    Currently the conduit looks like:

    conduit name=”*.http-conduit” xmlns:sec=”http://cxf.apache.org/configuration/security”
    xmlns=”http://cxf.apache.org/transports/http/configuration”>

    domain\\user
    May@2014
    NTLM

    This one.. instead of passing domain and username specified in conduit.. it uses underlying domain name to connect to WCF service.

  6. Hi,

    I want to create a SOAP client using CXF to connect to SharePoint. The authentication scheme is NTLM. I am blocked on a scenario where the logged-in user of a machine, on which the client is being run, has access to SharePoint. But I do not want that user to be used and specify some other user credentials. But as CXF uses in-JDK HttpURLConnection; and HttpURLConnection bypasses the credentials specified, if the logged in user is authenticated. The possible solution is to use AsyncHTTPConduit that uses HttpAsyncClient instead of HttpURLConnection. This is beacuse HTTP components do not bypass specified credentials and logged-in user can be ignored. The code snapshot you have provided shows the usage of DefaultHttpAsyncClient which is deprecated now and CloseableHttpAsyncClient is to be used instead. But CloseableHttpAsyncClient does not provide a way to specify credentials to an already existing CloseableHttpAsyncClient object. I guess that may be the reason I get ‘Authorization loop detected on Conduit’ error, not sure though. Could you please help me with this?

    The other solution that I tried out is to use sun.net.www.protocol.http.ntlm.NTLMAuthenticationCallback, to bypass logged-in user authentication, as mentioned here ‘http://stackoverflow.com/questions/6184881/can-javas-single-sign-on-use-credentials-from-credential-manager-on-windo’. This works when I specify valid credentials, and the code bypasses the logged-in credentials :). But when I specify invalid credentials, I do not get HTTP 401 error, instead I get ‘Could not send message, server reached max retries 20′. I am trying to avoid this solution because it uses java’s internal package and there is no way to determine HTTP 401 error directly.

  7. Pedro Ciarlini Says:

    Please. Let me say Thank you.

    Thank you.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 117 other followers

%d bloggers like this: