Adding a column to a data frame in R

Posted in Uncategorized with tags on October 2, 2014 by Shaun Elliott

Ok, this should be pretty simple. I’ve found R to be both simultaneously elegant and infuriating. This particular instance for example.

I wanted to add a new column to an existing data frame. My intuition led me down the path of lapply. I was wrong. It is far more simple than that.

Given the canonical R data.frame example:

n <- c(2, 3, 5) 
s <- c("aa", "bb", "cc") 
b <- c(TRUE, FALSE, TRUE) 
df <- data.frame(n, s, b) 
To use a function to add a new column, you have several ways. First, define the function, then you can just add it in one of 3 ways:
something <- function(x,y){

df[,4] <- something(df[,1],df[,2])
df["foo"] <- something(df[,1],df[,2])
df$bar <- something(df$n,df$b)

CXF WS Client With Fluent Builders and More!

Posted in Uncategorized with tags , , on December 18, 2013 by Shaun Elliott

Alright, I’ve been using SoapUI to generated my CXF WS clients for a little while now. It’s a pretty good tool, but ultimately if you’re using it to generate CXF, you might be able to skip SoapUI if you’re willing to indulge in some maven goodness. I started down the maven path with this because I wanted to use xjc plugins for the generated jaxb classes. Specifically, I wanted to use the fluent builders.

I ran SoapUI and collected the arguments. I then converted them over to the maven form, giving me this:

<!-- OPTIONAL: generate toString methods -->
<!-- OPTIONAL: use the fluent API -->
<!-- OPTIONAL: create value constructors -->
<copy todir="src/main/java">
<fileset dir="${}/generated-sources/cxf" />

I know that’s alot of pasta, but what you get is a single command run that will (re)generate your CXF WS client, putting them back in to your java source directory.

Here is my bindings file:

<jaxb:bindings version="2.1"
<jaxb:globalBindings generateElementProperty="false"/>

You’re on your own for your wsdl, just remember to drop it in src/main/resources/wsdl

Reading Tab Delimited Files – H2

Posted in Uncategorized with tags , , on April 23, 2013 by Shaun Elliott

Ok, this is probably easy enough to figure out based on the H2 documentation, but here is an example using java’s H2 embedded database, and a little bit of Spring to query tab delimited files:

public class H2Test {
public static void main( final String[] args ) throws Exception {

File h2Directory = new File( System.getProperty( "user.home" ) + "\\h2" );
if ( h2Directory.exists() ) {
FileUtils.cleanDirectory( h2Directory );

// start the TCP Server
Server server = Server.createTcpServer( new String[] {
"~/h2/test" } ).start();

try {
BasicDataSource dataSource = new BasicDataSource();
dataSource.setDriverClassName( "org.h2.Driver" );
dataSource.setUrl( "jdbc:h2:~/h2/test" );
dataSource.setUsername( "sa" );
SimpleJdbcTemplate simpleJdbcTemplate = new SimpleJdbcTemplate( dataSource );

String fileName = "full path and file name here";
System.out.println( simpleJdbcTemplate.queryForList( "SELECT * FROM CSVREAD('" + fileName + "', null,'UTF-8', chr(9));",
new Object[] {} ) );
catch ( Exception e ) {
finally {
// stop the TCP Server

Comparing SQL Logging with Log4j – Default vs. JdbcPlus

Posted in Uncategorized with tags , , , on March 27, 2013 by Shaun Elliott

I was recently playing with logging to jdbc via log4j when I discovered that it isn’t a fully elegant solution. It is fairly raw JDBC, or so it seems. I found this post on stackoverflow that described the same problem I was having. One of the solutions is to use the JDBCPlus appender.

Here is a quick comparison:

log4j.rootLogger=INFO, A1, jdbcPlusJdbcAppender, log4jJdbcAppender

# A1 is set to be a ConsoleAppender.

# A1 uses PatternLayout.
log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n

log4j.appender.jdbcPlusJdbcAppender.sql=INSERT INTO LOG (log_date, log_level, location, message) VALUES ( '@TIMESTAMP@', '@PRIO@', '@CAT@', '@MSG@' )

log4j.appender.log4jJdbcAppender.layout.ConversionPattern=INSERT INTO LOG (log_date, log_level, location, message) VALUES ( '%d{yyyy-MMM-dd HH:mm:ss.SSS}','%p', 'Class: %c', '%m')

A couple of obvious points:

  1. This example is based on the JTDS driver. This means you need JTDS on your path. It also means that if you use another driver, you need to change these settings to match the requirements of your alternate driver.
  2. Both appenders are attached to the root logger. This is just to show both in action at the same time.

Here are the differences that I have noticed:

  1. The default JDBC appender uses a conversion pattern to set up it’s insert statement. The conversion params are just standard values used which you can find here. JDBCPlus, on the other hand uses some magic values that are defined for you. I had to look over their examples carefully to figure out which was which.
  2. The properties themselves are slightly different. If you’re converting from one to the other pay attention to things such as Driver\dbclass, URL\url. It is case sensitive.

Hospitality Service Scheduler Kata

Posted in Uncategorized with tags , , on February 27, 2013 by Shaun Elliott

A while back I was introduced to Uncle Bob’s bowling kata. If you’ve never done it before stop reading and go do it before proceeding here. The bowling kata is an excellent exercise in application design, feature creep and overall good OO skills. In the spirit of the bowling kata, here is a new OO design kata, the “Hospitality Service Scheduler Kata”.


“The hospitality industry is a broad category of fields within the service industry that includes lodging, restaurants, event planning, theme parks, transportation, cruise line, and additional fields within the tourism industry.” – Wikipedia

Within the hospitality industry there are many employees that are required to make a business run. Each of these employees have one or more defined jobs that they are capable of doing, for example: cook, janitor, manager. Additionally, each employee is capable of working a certain amount of shifts per day and per week per local laws and restrictions. Lastly, most hospitality business schedule employees by the week, 1-2 weeks in advance.

Suggested Requirements:

Write a class named “Scheduler” with the following methods:

  • getEmployeeSchedule( employee : Employee, duration : DateSpan )
  • getSchedule( duration : DateSpan )

Things to think about:

  • managing employees (crud operations, persistence agnostic)
  • managing jobs, titles, roles
  • ease of use\user stories

Example Use Cases:

Provide a report per employee for a given duration, it might look something like this:


Monday 9a-3p Server
Tuesday 10a-2p Hostess
Wednesday Off X

Provide a report for a given duration for all employees, it might look something like this:















































Google Calendar Sync – End of Life? How about a work around.

Posted in Uncategorized with tags , , on February 14, 2013 by Shaun Elliott

After installing the OS on my new PC I figured out I was missing something: google calendar sync. If you don’t know what this is, it was an app that Google put out to sync your exchange and gmail calendar events. This was a very handy little app. However, to my dismay, I found that it has entered end of life. I explored other options, but none seemed to be as easy or free. What’s a nerd to do?

Well, I decided to muck around with my old PC and see what I could figure out. It turns out that the app itself can just be zipped up, moved over to another machine and ran…. for the most part. When I ran it on my new box it did prompt me for credentials, but it threw up an error dialog. A quick google search lead me down the direction of investigating the Outlook plugins. I took a look at my Outlook instance on my old dev box and found that you just need to add the plugin explicitly. To do this you just go (in Outlook) to options -> add-ins -> manage (go) -> add. Pick the GoogleCalendarSync.x64 or GoogleCalendarSync dll file you zipped up from your old copy and enable it (the 64 one on 64 bit machines, and the other one for 32). Voila!

After restarting Outlook all was kosher and things were syncing again, yay!

Connecting to Microsoft Dynamics CRM 4.0 Webservices with Java – Lessons Learned

Posted in Uncategorized with tags , , , , , on September 27, 2012 by Shaun Elliott

It all started several years ago. Someone mentioned that our internal CRM instance had a web service that we could use to automate some processes. Thus began an interesting journey, and here are some lessons learned. Much of this is from memory, so I’m sure I’ve missed some things.

The first thing I did was try to generate the client using Axis 1, as that we used at the time (around 2009-2010). I quickly found that because of the authentication, I could not even connect to the remote instance and generate the proxies. After some more trial and error, I figured out that I had to actually download the WSDL and generate my proxies based on the local file. This got me started, but when I actually got the code imported into Eclipse, I could not get it to actually work. I kept getting http 401 errors. This is because of the NTLM authentication CRM uses. I spent more R&D time fighting to get NTLM to work with Axis 1 to no avail. I read somewhere that Axis 2 works with NTLM, so I gave it a shot.

Now, full disclaimer – I’m no Axis expert. However, despite several hours of tinkering, I could not even get Axis 2 to generate anything meaningful. The classes were broken\wouldn’t compile, many things seemed to be missing, etc. So in the end I gave up and moved on to more fruitful work. The project sat for 6 to 9 months.

While this project sat, I started learning about CXF. I did a couple of successful clients with it and got a general feel for how it worked. I experimented and also learned how to use SoapUI. I did a little more research and found that it isn’t that hard to do authentication with CXF. So I picked up the project again and this time I was able to get the code to generate and shortly thereafter I got authentication working…or so I thought.

private void setupSecurity( final CrmServiceSoap crmServiceSoap ) {
//Turn off chunking so that NTLM can occur
Client client = ClientProxy.getClient( crmServiceSoap );
HTTPConduit http = (HTTPConduit)client.getConduit();
HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
httpClientPolicy.setConnectionTimeout( 36000 );
httpClientPolicy.setAllowChunking( false );

http.setClient( httpClientPolicy );

AuthorizationPolicy authorization = new AuthorizationPolicy();
authorization.setUserName( _userName );
authorization.setPassword( _password );

http.setAuthorization( authorization );

This worked fine for some months, allowing us to do almost everything that we needed. We were able to create cases , accounts, run queries, etc. The only real glitch was that we had to keep a read only copy of the wsdl on an open web server and our client had to point at it, instead of the one behind the CRM authentication wall.

Fast forward to a couple of weeks ago (September 2012). One thing we hadn’t tried was updating records. No we found a use case for that too. However, this time it didn’t work. We used the same login and the same code that was working before to no avail. As it turns out, the security was actually using the local active directory user. So when we tried to update records that the user did not own, we’d get something like this:

Payload: Server was unable to process request. ---> Exception has been thrown by the target of an invocation. ---> SecLib::AccessCheckEx failed. Returned hr = -2147187962, ObjectID: 949161c6-2333-dd11-8014-001125a94557, OwningUser: de58855c-482c-db11-889e-001125a94557 and CallingUser: b7c87416-e895-db11-96c9-001125a94557

After wiresharking this, I found something interesting… In the http headers, the Authentication value was set to: “Basic xxxxxx-a-bunch-of-data”. Wait…that’s not NTLM?!? As it turns out, NTLM authentication is a bit of a pain to setup. Furthermore, it isn’t really supported in CXF. Supposedly you can do it like so, but I could not get this to work. I’m not 100% sure why, but I think it is because jcifs doesn’t work out of the box with Windows 7. I played with Jespa a bit, even downloading the source code, but I could not get it to work either.

Getting desperate, I figured that I was able to get this to work in SoapUI. And I knew SoapUI is java & open source. Digging through the SoapUI code a bit more, I found that it in fact uses a fairly vanilla http client, rather than a whole web service stack. After a bit of poking, it turns out that the CXF http client is non-extensible\swappable.

So, to make it work I turned to a bit of a hack…I used the DefaultHttpClient as well:

private static final String ACCOUNT_UPDATE_SOAP_TEMPLATE =
"    <soap:Envelope xmlns:soap=\"\" xmlns:web=\"\" xmlns:cor=\"\">  " +
"       <soap:Body>" +
"            <entity xmlns=\"\" xmlns:ns2=\"\" xmlns:ns3=\"\" xmlns:ns4=\"\" xmlns:xsi=\"\" xsi:type=\"account\">" +
"                <accountid>%s</accountid>" +
"                <customfield1>%s</customfield1>" +
"                <customfield2>%s<customfield2>" +
"            </entity>" +
"       </soap:Body>" +
"    </soap:Envelope>";

Account account = (Account)entity;

//TODO - hack! CXF's default http client does not seem to work with NTLM authentication
// found this workaround here:
DefaultHttpClient httpClient = new DefaultHttpClient();

NTCredentials ntCredentials = new NTCredentials( _userName, _password, "", "mydomain" );
httpClient.getCredentialsProvider().setCredentials( AuthScope.ANY, ntCredentials );

HttpPost request = new HttpPost( "/MSCrmServices/2006/CrmService.asmx" );

String soapMessage =
String.format( ACCOUNT_UPDATE_SOAP_TEMPLATE, account.getAccountid().getValue(), account.getCustomField1(),
account.getCustomField2() ); String.format( "Preparing to send soap message: %s", soapMessage ) );

request.setEntity( new StringEntity( soapMessage, ContentType
.create( "application/soap+xml;charset=UTF-8;action=\"\"" ) ) );
try {
HttpResponse response = httpClient.execute( new HttpHost( _hostname ), request ); response.toString() );
catch ( Exception exception ) {
throw new RuntimeException( exception );

So the battle is over for now, but the war is not yet won. I hope this helps someone else.

Update #1

After days of working with one of the CXF developers I think we’ve got NTLM authentication working natively with CXF. That last jira link: CXF http client is non-extensible\swappable turned out to be the entry point. One of the dev’s mentioned that the task was in fact complete in the latest\trunk version (2.7.0-SNAPSHOT). The only problem was that the client was buried deep in the API, so I opened up a new ticket to expose the client. Within an afternoon or so it was already done! Lo and behold, I tried it out and after a little bit of mucking around I got it to run, but it was throwing exceptions. After a couple of days more back and forth the finished code has arrived:

Bus bus = BusFactory.getDefaultBus();
bus.setProperty( "use.async.http.conduit", "true" );

Client client = ClientProxy.getClient( proxy );
HTTPConduit http = (HTTPConduit)client.getConduit();
if ( http instanceof AsyncHTTPConduit ) {
AsyncHTTPConduit conduit = (AsyncHTTPConduit)http;
DefaultHttpAsyncClient defaultHttpAsyncClient;
try {
defaultHttpAsyncClient = conduit.getHttpAsyncClient();
catch ( IOException exception ) {
throw new RuntimeException( exception );
defaultHttpAsyncClient.getCredentialsProvider().setCredentials( AuthScope.ANY,
new NTCredentials( "username", "password", "", "domain" ) );

conduit.getClient().setAllowChunking( false );
conduit.getClient().setAutoRedirect( true );

Much thanks to Daniel Kulp, for getting this done. This was a great open source experience!


Get every new post delivered to your Inbox.

Join 117 other followers