Early Impressions of Some New Choices

My team and I were recently tasked with a pretty monumental project. We are basically building the services that will power a whole host of extremely high traffic websites that can do everything from paying a bill to streaming video content to an iPad. The complexity of this type of project is very appealing and I’m really happy be a part of something this ambitions.

As part of this endeavor we were given the freedom to choose just about whatever tools and technologies we wanted to build the most scalable, maintainable and bullet-proof system we could. After about a week spikes on different languages, frameworks and tooling we came to the initial conclusion that we would adopt:

  • Git for version control
  • Spring for DI and other stuff (which we since removed, see below)
  • Scala as our primary development language
  • Gradle for our builds
  • MongoDB for persistance

This blog is one of many to report on my impressions of these choices and the tradeoffs they are (or aren’t) causing us to make versus a standard J2EE/Spring Java project that just about everyone on my team has done in the past successfully.

First, Git.

Git. Is. Awesome. Seriously, it is amazing. I’ve been using Git for about two years but, before now, only in very small projects. I thought it was good before, but I’ve really grown to love just how much flexibility and speed it buys me personally not counting how much it buys our team in terms of ease of branching and an easy (now) to understand code history. If you haven’t yet adopted Git as your SCM system, you absolutely should.

Secondly, Spring.
We’ve all used Spring pretty heavily in the past and have a good track record of success with it. However, as we began coding our DI, we came to a very startling conclusion: we probably don’t need it anymore.

Relying on our experience with DI in general and our understanding of what Spring does “under the covers” as it wires objects together, we were able to create a native Scala DI solution that is statically typed, easy to understand (read: no XML or annotations) and, given that it’s simply part of the language, extremely quick to bootstrap. There is no ApplicationContext to wait for or mock for testing. It does its work when the classes are executed (and slightly later if we’re using the lazy modifier which we usually do) and that’s that. While I’m sure we’ll be refining this solution over time, the fact that we own it and it works as well as it does for us makes me suspect that we, as a team, have grown “beyond” the need for Spring. Spring helped us (and me in particular) understand the need for things like DI and AOP and caused me to think of my code in a very decoupled manner. Thanks Spring, but we’ve grown up!

Next, Scala.
I’m not sure what I think of Scala yet. It’s a daily thing. Coming from a lot of Groovy and JavaScript work, I have grown to like the free love devil-may-care type system. Some of the recent code I’ve written in those two languages is the best work I think I’ve ever done. However, I do see the value in a stricter type system and there are many things about Scala I love.

The biggest concern I have is how ridiculously complex Scala can be if you let it. The community has yet to (as far as I can tell) establish a concise set of patterns (something like Effective Java) and many teams (including ours) spend a lot of time trying to figure out what esoteric features to take advantage of and which to ignore. The exciting thing about this is that we are building our own Scala “way” of doing things internally in our team. That’s also the scary part. However, so far, it’s been relatively smooth aside from those times when I think “my god this was so much better in [Groovy|Java]” which are becoming less frequent as I continue to learn the language.

Gradle.
I had high hopes for Gradle when we started the project. We moved from a combination of Gant and Maven and, at first, we were loving the combination of convention and extendability. It really allowed us to customize our build and do some pretty neat stuff with very little effort.

My biggest complaint, which is starting to sour my whole opinion of the tool, is how slow it is. Running gradle tasks in our project on a quad-core Xeon takes 17 seconds! Running a build from scratch takes almost 30 minutes. And, since there isn’t yet great IntelliJ support for it, running a gradle idea (which I do several times a day to refresh the .iml dependencies) takes 4 minutes.

These are all unacceptable timeframes in my opinion. Granted: much of the 30 minutes I mentioned above is spent downloading dependencies (perhaps the majority of it), but the other tasks download nothing and still take far too long.

As gross as it sounds, I’m actually missing Maven. Not necessarily for configuration and dependency management (which were and still are a pain in many cases) or its contrived complexity and “xml hell” but more because it was faster and had first-class IDE support. These things are all missing right now with Gradle and I feel like it might be causing us to lose velocity rather than gain it as the time saved setting up the project does not match the time we all spend waiting for tasks to complete on a daily basis.

We may be doing something wrong and I’m open to doing what it takes to modify our setup, but I feel like what we’ve done is extremely reasonable and a relatively “common” case in terms of project structure and dependencies. To verify that assumption, I ran the same types of commands using another complex project: Gradle itself. I had similar results to those I see in our project. I’m hoping things improve in this regard after a smart colleague of mine attends Gradle training next week.

Finally, MongoDB.
I don’t have much of an opinion on this one as we haven’t integrated to it just yet. I’m interested to see how it compares to CouchDB as that’s where my previous NoSQL experience lies.


Again, I’ll be writing more as we get farther into the project. I feel (mostly) pretty good about the decisions we’ve made and the team I’m on. It’s been, and will continue to be, an awesome ride!

Advertisements

About johnnywey

Welcome to A Regular Expression. This blog is designed to reflect my thoughts on life, music, software design, Apple, faith, philosophy, and whatever else I can think of.

Posted on September 22, 2011, in Git, Gradle, Java, Programming, Scala. Bookmark the permalink. 9 Comments.

  1. Hi Johnny,

    1.) For ‘gradle tasks’ you feel the pretty high startup time penalty of Groovy. You can avoid this by using the Gradle daemon. The daemon will be enabled by default in 1.0. Right now you have to enable it. To experiment with it you can just run your build with –daemon. Of course you will only see the improvement after the second run. You can also enable it permanently via an environment variable.

    2.) Beyond that in our experience Gradle is a fast build system. Definitely not slower than Maven in the projects where we did comparisons. In many situations it is faster specially because of the incremental build. Maven is faster though at the moment for downloading many dependencies. Once they are cached this is no longer an issue of course. You can run Gradle with –profile to see where the time is spend. Just go to build/reports/profile for the report.

    3.) IDEA 11 will have build-in Gradle support. Some of it is already in the IDEA source and can be tried out. See: http://confluence.jetbrains.net/display/IDEADEV/Gradle+integration

    Hans


    Hans Dockter
    Founder, Gradle
    http://www.gradle.org, http://twitter.com/gradleware
    CEO, Gradleware – Gradle Training, Support, Consulting
    http://www.gradleware.com

    • Hi Hans,

      I appreciate the response! We didn’t have much luck with IntelliJ 11 (missing some sort of tooling libraries) but I understand that is a temporary issue.

      I REALLY want Gradle to work for us. I don’t hate Maven per se, but I do really like how easy Gradle is and how much power Groovy provides in the context of a build.

      I’m hoping that we can get over the issues we’re having and make it work for us!

    • I’m 95% sure we’re causing the slowness we’re seeing. Assuming that to be true, we should all document the cause and solution well because I’d hate for others’ first experiences to be so slow.

      On a more positive note: our build files are greatly reduced in size and are MUCH more easily maintained, save a couple small cases I’ll ask the Gradle guys about separately.

  2. I agree with your thoughts on Scala. You can write brilliant code in it; it is type-safe, but it does not get in the way too much; unfortunately, we are indeed missing “efficient Scala” equivalent. I might start something on that topic, muddied with the views of a Spring & Java veteran.

  3. The cure for your Gradle woes is SBT. Way faster.

    • Hi Eric,

      do you have numbers to back your claim? In any case, I have a different opinion on that. Gradle is used in very large enterprise builds (> 10 Million LOC). One of the reasons why they have switched to Gradle is performance. For example Gradle’s reliable incremental build is unique and dramatically reduces the average build time for larger builds. The most important metric for us is the average build time for complex builds.

      I think one should analyze Johnnys performance issues before making such claims. May be something is configured the wrong way. This is very easy with the –profile option.

      Hans

  4. I also agree with your statement on Scala. We used Scala only in a few tiny tool projects, but eventually, I came to replace it with good old java, mainly because:
    – Scala’s binary incompatibility really gives me a bad feeling when it comes to maintainability and dependencies over time
    – “Forcing” our team to learn a new language would’ve just been impossible, in part because of (as you mentioned) the complexity you can achieve with Scala… you know, different skill levels and stuff… but let’s not get into that ;)

    For Spring, it’s funny, we had the same effect in the mentioned small tool projects when we used Scala. Though it wasn’t me who came up with the idea, DI seemed to come in pretty naturally with Scala. However, now that I’m not using it anymore, I’m still glad I have Spring ;)

    When it comes to gradle, I might have to disagree. I’m currently porting the build system of our core framework (pretty much like your project, but in the context of enterprise java applications) from a stone-old ant build to gradle and I’m loving the result. Though I agree that the startup time is pretty bad (haven’t had a chance to check out the daemon mode), the dependency clarity, incremental build, clean build scripts, awesome multi-module support etc. totally pays off imho. I’m far from being done, but as of now, I’d never go back to ant (oh and I’m a convinced maven hater, so that’s never been an option for me ;))

    Good read, I’m looking forward to hearing your thoughts on MongoDB :)

    best regards,
    mike

  5. Johnny,

    Have you considered a post detailing your home-grown scala DI solution? The Spring team would certainly be interested to see what you’ve done.

    And how about the rest of Spring’s functionality, beyond DI? What are you using for a web framework? Lift? How do you find it in comparison to Spring MVC? How about tx management, spring-data-mongodb, etc?

    • Hi Chris,

      A post is definitely coming. I want to make sure we continue to be happy with our solution before showing it off. I’d expect to have something together in the next few weeks.

      We are currently using Lift as our web framework and I am definitely going to write something along the lines of “a Spring MVC / Grails guys take on Lift” in a future post. I’m still wrapping my head around Lift and I don’t want to say too much about it until I feel I have a better understanding of it. Our use case is also pretty special in that the public-facing point of our application is only services. There is no page generation or asset management only JSON / XML.

      As far as the other stuff Spring provides (and, indeed, it is a lot of non-trivial stuff for sure), we are not anti-Spring by any means. If it ends up making sense for us to adopt pieces of the framework for other reasons, none of us are opposed to that. In fact, you may find me writing a post on how arrogant and stupid I was in thinking we could “replace” spring in the coming months! :)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: