Our production application stack has a fair amount of external service connections that require the use of certs for SSL security. We used to use a static reference (e.g. /config/certs or c:\config\certs for you Windows people) to locate the certificates, but this was always suboptimal as it made the application less portable. I finally got around to moving from the static reference and landing on Java jar introspection to find the file (using something similar to ClassName.getResource("/cert.pks");).

This conversion worked very well and everything seemed fine until the change got to our continuous integration (or CI) box. That machine, a Centos distribution, kept failing with the error Invalid keystore format when we tried to initialize the keystore in code. I spent hours trying to figure out the difference between my local machine (running OS X, which worked fine) and the CI box. I finally diff’d the cert file in src/main/resources and the resulting file in target/classes. I discovered that the two files were radically different.

Maven has a very cool feature that not only copies values from src/main/resources into target/classes and, ultimately, the resulting Jar file but also can substitute tokens in text files as part of the build (referred to as “filtering”). To enable this, all you have to do is add

<resources>
            <resource>
                <directory>src/main/resources</directory>
                <filtering>true</filtering>
            </resource>
</resources>

to your pom.xml file and you’re off to the races. However, Maven will process every file in the src/main/resources directory as part of performing this substitution. This normally wouldn’t be an issue, but certificate files contain some raw text and Maven dutifully filters them just as it would any resource. This caused a serious issue with the character encoding as Maven was writing out a file with a completely different encoding scheme than the file was created with. Basically, Maven filtering was corrupting our certs.

To fix this, I created a subdirectory off of src/main/resources (I called it binary) and added a manual exclusion in the pom.xml file. The resulting XML looks as follows:

<resources>
            <resource>
                <directory>src/main/resources</directory>
                <filtering>true</filtering>
                <excludes>
                    <exclude>**/binary/*</exclude>
                </excludes>
            </resource>
            <resource>
                <directory>src/main/resources/binary</directory>
            </resource>
</resources>

This worked perfectly. Hopefully, it helps some of you as well!

Grails is pretty awesome. Not only does it use Groovy as its main language, but it also provides nice DSL access to Spring settings.

Securing a website is something that can often be fairly complex and easy to get wrong. Grails provides a plugin for the state-of-the-art security model Spring Security (also called Acegi) (plugin can be found here).

Everything was all well and good until I tried to use two different security models simultaneously: a form-based login for human beings and an http basic authentication login for machines consuming an API.

This turned out to be pretty hard. The plugin requires setting the following in the resources.groovy file to work with basic authentication at all:

beans = {
  authenticationEntryPoint(org.springframework.security.ui.basicauth.BasicProcessingFilterEntryPoint) {
    realmName = 'Grails Realm'
  }
}

(The plugin is supposed to automatically do this by setting the basicProcessingFilter = true flag in the SecurityConfigure.groovy file, but there is a bug as of version 0.5.2 which is, as of now, the latest production release.) This worked for basic authentication, but would no longer redirect users to the form-based login.

Back to the drawing board.

What I discovered was that I could change between using the form-based login or the http headers authentication but not use them both. (I should clarify this by saying that I could shove the headers into the request and it would work, but a challenge response of 401 was not being sent back to the client which technically breaks the http authentication spec.) I found this thread (with contributions from the author of the Acegi plugin itself) but, while it pointed me in a nice direction, it didn’t answer the question fully. What I really needed was a way to use the URL to decide how to authenticate.

I finally discovered that the ExceptionTranslationFilter was not unique for the URLs I was using. In short, if the URL looked like: http://www.test.com/application/api/user/show/1, I wanted to use basic authentication and if the URL was http://www.test.com/application/user/show/1, I wanted to use the form-based authentication. The redirect is controlled by the ExceptionTranslationFilter and there was only one (so only one model would work at a time).

All I ended up having to do is create my own ExceptionTranslationFilter and wire it to the API URL. The final code in resources.groovy looks as follows:

import org.springframework.security.util.FilterChainProxy
import org.codehaus.groovy.grails.plugins.springsecurity.GrailsAccessDeniedHandlerImpl
import org.springframework.security.ui.ExceptionTranslationFilter

// Place your Spring DSL code here
beans = {
  basicAuthenticationEntryPoint(org.springframework.security.ui.basicauth.BasicProcessingFilterEntryPoint) {
    realmName = 'Grails Realm'
  }

  basicExceptionTranslationFilter(ExceptionTranslationFilter) {
    authenticationEntryPoint = ref('basicAuthenticationEntryPoint')
    accessDeniedHandler = ref('accessDeniedHandler')
    portResolver = ref('portResolver')
  }

  springSecurityFilterChain(FilterChainProxy) {
    filterInvocationDefinitionSource = """
         CONVERT_URL_TO_LOWERCASE_BEFORE_COMPARISON
         PATTERN_TYPE_APACHE_ANT
         /xml*/**=authenticationProcessingFilter,basicProcessingFilter,securityContextHolderAwareRequestFilter,anonymousProcessingFilter,basicExceptionTranslationFilter,filterInvocationInterceptor
         /**=httpSessionContextIntegrationFilter,logoutFilter,authenticationProcessingFilter,securityContextHolderAwareRequestFilter,rememberMeProcessingFilter,anonymousProcessingFilter,exceptionTranslationFilter,filterInvocationInterceptor
         """
  }
}

Now, visiting any URL that prefixed by a /xml/ will be a call to the basic authentication headers and every other prefix will be a standard form-based login.

Since I shaved a serious yak figuring this out, I hope it helps someone!

After the church I was involved with fell apart a couple of years ago, a group of us from the music team continued to meet and play together. Occasionally, we’ll get calls from other churches asking us to fill in for their Sunday service and give their worship team a chance to take a break (playing week-in / week-out can get tiresome and often strain volunteer musicians to an emotional breaking point).

One church we recently had the pleasure of subbing for is in the middle of a church split due to the recent vote by the ELCA (Evangelical Lutheran Church in America) to allow for those in “life-long, monogamous, same-gendered” relationships to serve in leadership positions (find out more at http://www.washingtonpost.com/wp-dyn/content/article/2009/08/21/AR2009082103343.html). For those who don’t know, the ELCA is the largest Lutheran denomination in the United States and has classically been a bit more liberal than other Lutheran organizations. The provision does not require churches to accept homosexual ministers but allows for churches that desire integration of same-sex couples into leadership to do so.

To be clear, I’m not Lutheran. However, observing all of this has impacted me fairly heavily, perhaps bringing back some difficult memories of the demise of our faith community in early 2007.

One of the reasons I believe Christianity has been able to survive as long as it has revolves around its ability to adapt to cultural shifts and maintain relevancy. For example, there was a period of time when the church strictly forbade divorce, but it has largely moved away from that stance and has welcomed divorced individuals into fellowship. In fact, studies show the current divorce rate among Christians is very similar, if not identical, to the national average.

It’s hard to imagine a time when a church would split over an issue like divorce, but it did happen (part of the reason for the creation of the Anglican Church) and many people held passionate views one way or the other. I see a mirrored situation currently taking place as Christianity struggles to find a place for homosexuals in both fellowship and leadership.

The interesting thing about this decision by the ELCA is that it did not require member congregations to appoint gay elders and even allowed congregations to reject the practice internally. The fact that church splits are taking place over an issue that will probably have little pragmatic impact on them serves to show just how heated and difficult it is.

I’m not writing to weigh in on one side or the other. I am writing, however, to encourage those on both sides to try and find common ground and avoid splits if possible. Speaking from experience, the destruction of a community has a severely negative impact on those involved and should always be considered a last resort of sorts. There is lots of room for discussion and debate on the matter, but I think it is important that the issue is placed in a historical context. The church has flexed to allow for lifestyles in today’s modern day that it would have never even considered hundreds of years ago. As charged as the fight for or against allowing women into leadership was at the time, it seems rather petty in today’s modern world. The same goes with a host of other issues such as the allowance of tattoos or the orbit of the planets in the solar system.

My opinion is that none of these are worth the destruction and heartbreak that a division has both internally in the community experiencing it and externally to those looking in at the faith. If there is a way to maintain unity, I hope they can find it regardless of the perceived cost!

A few weeks ago, I had the pleasure of getting a promotional copy of Craig Walls’ (from Spring In Action fame) new book and thought I’d post a review of it here on my blog.

Modular Java: Creating Flexible Applications with OSGi and Spring provides a great introduction to those either curious about OSGi or wanting to get more out of their existing OSGi workflow using the Spring Framework.

Modular Java by Craig Walls

In the second portion of the book, Craig throws Spring into the mix and demonstrates how the power of Spring Dependency Injection, autowiring, and the Spring MVC web framework can not only run seamlessly in an OSGi container, but also remove a large portion of the burden that OSGi’s API can put on application development.

Finally, Craig spends some time describing how an actual deployment might look in a production environment using both Tomcat and Jetty and provides optimization tips that make the process as painless as possible.

The book itself is logically organized and Craig’s writing style is approachable and easy to follow. All the example source code is available online, and Craig demonstrates how to install OSGi packages using both Eclipse Equinox and Apache Felix, leaving the final OSGi container decision up to the preference and requirements of the project. The sample application Craig uses to demonstrate the concepts in the book is surprisingly fun and useful, and the book contains some wonderful appendices that function as a great reference for current and future development projects. The book is a relatively quick read but surprisingly complete.

For someone looking to get the most out of OSGi or wanting to find out what all the “buzz” is about, Craig Walls’ book is an outstanding choice.

Or, perhaps more accurately, learning about timezone oddities in MySQL the Hard Way

// the sql var points to the database sql connection info
def startTime = new Date()
sql.eachRow("show tables") { row->
  tables << row['Tables_in_database']
}
sql.eachRow("show columns from " + table) { field->
  if(field['Type']=='timestamp') {
    println "Found timestamp field of ${field['Field']} in ${table}"
    def updateStatement = "update " + table +
    " set ${field['Field']} = " +
    "(select date_add(${field['Field']}, INTERVAL 6 HOUR)) " +
    "where ${field['Field']} is not null"
    println "Updated ${sql.executeUpdate(updateStatement)} " +
    "records from ${table}"
    }
}

Integer totalSeconds = ((new Date()).time - startTime.time)/1000
println "\n\nComplete!  Total time: ${totalSeconds} second(s)"

Not bad. The script simply runs through each of the tables with a timestamp and adds 6 hours (the difference between MDT and UTC) to the timestamps. It’s not perfect (we had another time change to think about), but it was deemed good enough since timestamps with dates before the last “spring forward” change from MST to MDT were beyond the point where their hour of innacuraccy would have a measurable affect on the system (we’re all about pragmatism).

However, I kept getting a very odd error when running the script against test data. Data truncation: Incorrect datetime value “2009-03-08 02:08:00″ After some research, the reason became obvious: there is no 2am hour on the 8th of March! That hour was skipped when daylight savings took affect. Makes sense, but I was dissappointed that MySQL didn’t just figure that out for me. No big deal … I modified the above script to take that into account. We were off.

Or so we thought.

In testing today, we found out that MySQL stores timestamps internally as UTC times and does the timezone extrapolation dynamically. This meant that I didn’t need to go through the above excercise at all … all we had to do was verify the MySQL server was running in the same timezone as the server itself (select @timezone;), update the server’s timezone, and restart mysqld!

Recently, I consumed a WSDL with Apache CXF that required authentication information to be placed into a SOAP Header rather than through standard HTTP authentication. This is all well and good, but I couldn’t figure out how to attach the headers to the CXF method calls. The authentication object had been generated by CXF, but there was no way to attach the object to the other methods.

After some research, I found the answer. Adding -exsh true to WSDL2JAVA updated the method signatures to include the header object as a valid parameter. If you’re cool like me and use the CXF Maven code generator plugin, the XML looks like this:

<execution>
<id>generate-sources-vixxi</id>
  <phase>generate-sources</phase>
    <configuration>
      <sourceRoot>target/generated-sources</sourceRoot>
        <wsdlOptions>
          <wsdlOption>
            <extraargs>
              <extraarg>-exsh</extraarg>
              <extraarg>true</extraarg>
            </extraargs>
          <wsdl>yourWSDLPathHere</wsdl>
        </wsdlOption>
      </wsdlOptions>
    </configuration>
    <goals>
      <goal>wsdl2java</goal>
    </goals>
</execution>

I verified this works with CXF 2.0.9 and 2.1.4. I haven’t tested it on any other versions, but I don’t see any reason why it wouldn’t work on 2.0.9+.

Or, perhaps more fittingly, why I am unhappy with Comcast.

Since I got back to Colorado from Hawaii in 2003, I’ve used, and been relatively happy with, Comcast broadband Internet. There have been issues, but nothing that has been so bad as for me to consider moving to another service.

Unfortunately, that changed last week.

Let me back up and say that I don’t often use my Blog to call out a company for something irritating (that’s usually relegated to Twitter). In this case, however, I feel it a good object lesson in how poor design coupled with horrible customer service can transform a (mostly) satisfied customer and cause them to look into other options.

What took place is pretty simple to understand. My wife and I share the account, and her email address is one of the six email accounts that I can have attached to our service. She decided she wanted to add another address. What she didn’t know, and what the UI didn’t tell her (in BIG RED WARNING LETTERS) was she was actually changing her email account. Without any “are you sure you understand what you are doing and want to do this?” type of dialog, her email address was changed and her previous account stopped receiving mail immediately.

When she realized what happened, she frantically clicked back to get her old address re-activated, but the UI continued to tell her the username was in use even though it had been deactivated.

She called me pretty upset, and I got on the phone with Comcast. They told me that they couldn’t do anything about the issue for a minimum of 30 days. When I asked them why, they told me that when an email address is deactivated, it gets placed into a “holding” status for 30 days to insure another person doesn’t take the alias and start receiving mail intended for another customer. This makes a lot of sense to me. The odd thing is that they couldn’t re-activate it for the same customer!

I told the CSR on the phone that someone should be able to quite easily login to their database and type something like:

update user set status="active" where username = "[Amber's username]";

and that, while I knew he couldn’t do it, someone with a shiny Comcast data-center badge could.

After much back and forth, he finally submitted an “IT” ticket for me. Since then, and his promise that this would be resolved in 72 hours (over a week and a half ago from today), I’ve heard nothing.

There are a couple of lessons here. The first is that it is obvious Comcast doesn’t get how important an email address is. I would have rather Amber lost her phone number than her email address. From our online banking to Facebook, it is extremely difficult to change an email address when you have no access to the old one. Most services send confirmation messages to the old email address. The CSR I spoke with and Comcast’s policies in general around this issue prove that, while they claim to be on the vanguard of Internet technology, they just don’t get it. At all.

Secondly, it is extremely easy to add UI confirmations so do it! There is no excuse for this type of oversight whatsoever.

Our current plan is to switch over to Qwest Fiber as soon as it becomes available in our area. Unfortunately, Comcast has lost our confidence.

The Autocompleter suite of functions in Scriptaculous are pretty nice. While they don’t solve every autocomplete problem straight out, they do provide a good starting point.

However, the answers to problems in Javascript are not always obvious. Despite some amazing support from Firebug, debugging is often difficult and tracing down problems can be rough.

Earlier this week, I ran into a very strange problem with Autocompleter.Local. I was designing a page that would query the server for a list of cities in a state and load that list into the Autocompleter. The catch was that the state could be changed at any time by the user, and the array containing the list of cities available would be updated asynchronously from the server. (While there is an Autocompleter.Ajax for automatic server updates, this assumes the autocompletion is done on the server side which I deemed too slow to be very functional in our case.) The specific issue I was having was that the city name was inserted into the target text field multiple times as the number of hits to the server for updated city lists increased.

I discovered, after looking over code in control.js, that the Autocompleter attaches event observers every time it is newed up. These event observers are difficult to remove manually because they pass in states of the specific object that are often no longer available. I had code that was doing something like:

[get list of city names for a state into cityNameList]
if(!autocompleter)
  autocompleter = new Autocompleter.Local('cityTextfield',
   'cityAutocompleteList', cityNameList);

but the the observers that the Autocompleter was creating were being piled up on each other every time I “reset” the Autocompleter to a new instance.

I discovered that the Autocompleter data is stored in the .options.array field. While it is not documented as part of the API, updating this array effectively updated the Autocompleter without adding new event observers. All I had to do was new up the Autocompleter variable once and then update this array field (directly; using dot notation) every time the data from the server changed.

Often, the Struts 2 tag library doesn’t offer enough functionality to support a nice DHTML type of layout. In those cases, I find myself creating URLs manually to send to the action.

In most cases, this isn’t a bid deal. However, to make it work right, you have to duplicate the format that Struts uses to receive the data. For example, to send a list of items (such as List<String> names;), you would format your URL to look like


&names=johnny&names=tim&names=bill


Maps are a little harder. To create a URL to send to a map, you can use the following format:


&variableName[key]=value


Say you have the following map:

Map<Integer, Integer> userIdOrderIdMap = new HashMap<Integer, Integer>();

To set the map from Struts 2, you’d use the following URL string:


&userIdOrderIdMap[0]=1&userIdOrderIdMap[10]=15

This is the equivalent of:


userIdOrderIdMap.put(0, 1);
userIdOrderIdMap.put(10, 15);


You will probably have to manually call encodeURIComponent() on the URL string to make sure the brackets and any other special characters are escaped properly, but you already knew that, right?

You can also use a dot notation (such as &variableName.key=value), but I find that a bit counterintuitive to the way KVC types are intended to be represented.

Yesterday, my dev team embarked on a significant refactor of our main code base in order to merge in a branch alongside implementing new functionality and changing some existing classes. A trifecta of complexity if you will. For a production quality system, one that we expect to deploy very soon and is the backbone of our company’s infrastructure, you might expect us to have been a bit more careful / timid in making our changes.

However, you’d be wrong and it’s all because of unit tests (that and we are all rock-star engineers, but that’s another post).

I didn’t work with unit tests much before my current position. This was mostly due to legacy platform limitations that did not make automated testing easy. Now, I couldn’t live without them. Confidence in my work is much greater now than it was previously for a good reason: unit tests give engineers power.

I know that the requirements of a particular class are laid out in the classes’ test. Whatever I do to the class, it must not only compile but light up green. This gives me the boldness to make large changes without worrying that the refactor is going to negatively impact (read: break) other parts of the system.

Code these days often exists beyond the classes themselves and in places like configuration files and build scripts, and we’ve created tests that take these pieces of auxiliary setup into account. Doing that extends our confidence beyond our primary classes and into our build process, database connections, and system as a cohesive whole. Very cool.

Unit tests are great for creating interfaces (working with the API before you actually write the implementation is a good way to make sure it doesn’t suck), verifying complex functionality, and system integration. The most empowering thing they offer, however, is confidence. In an agile shop such as ours, this gives us the ability to work as quickly and fluidly as possible and is a huge factor in why our little team has been so successful.

“A compiler tells you your code is well formed, a unit test tells you your code is well behaved.”

– No Fluff Just Stuff, Fall 2008