Category Archives: Programming

In All Fairness to CloudBees

** EDIT ** Please make sure to check out the comments section for some additional information from a CloudBees engineer. He was kind enough to explain to set up their build environment to support some of the tools I mention below.

Today, due to some frustration I experienced, I tweeted:

In fairness to CloudBees, I should explain further.

Jenkins is, in my opinion, the reference implementation of continuous integration. It’s really great. While it favors Java, it’s not uncommon to see folks use it to run iOS builds, Ruby, Node, you name it. It’s very extendable and has an active developer community. I’ve had significant success using it for the last few years and, earlier this week at my new gig, my team and I setup an environment.

Hosted solutions can be a great idea because they often handle all the stuff about hardware and OS setup I don’t really care about. I am not a system administrator and I didn’t want to provision and manage a VM just for CI if I could help it (I know, being lazy but honest here). CloudBees seemed like a great choice and we had it up and running in literally minutes. It supported Gradle and Grails right out of the box (currently pieces of our software stack), has GitHub integration and other cool features. So far, pretty big win.

However, I’ve grown very used to using Jenkins for things outside of stock Java builds. For example, you can use Jenkins to test JavaScript (see this for an excellent example). We have also been using JRuby and Rake for pieces of our application stack and Jenkins has nice support for that also.

The hosted version, however, didn’t support much outside a stock Java (Ant, Gradle, Maven, Grails) build. I don’t necessarily blame them as trying to keep up with versions of Node, Ruby, PhamtomJS, whatever is probably outside the core competency of CloudBees. They are, first and foremost, a Java hosting platform. However, it sort of sunk us. After trying workarounds for a few hours, my team and I decided to move over to a RackSpace virtual host and setup our own custom solution.

So, to be fair, CloudBees hosted Jenkins (and even their application cloud) is an excellent choice if you are interested in building Java/JVM applications and don’t mind not having support for other, perhaps more fringe, tooling. The system itself ran flawlessly and worked very well for the more common tasks we sent its way. However, it’s just not a long term solution for the things we’re looking to do.

To quote Wheaton’s law:

My tweet was a little uncool and not very fair; I hope this explains it better!

Advertisements

JSONP: How Does it Work?

Interestingly enough, I’ve seen a few candidates in interviews the last few months flail a bit when asked about getting around the same origin policy using JavaScript in a browser. Most know enough to toss out the term “JSONP” and perhaps another solution (e.g. proxy or CORS) but can’t really explain why one might prefer a certain solution over another. In fact, few could even describe how JSONP works aside from some JQuery library they always “just include”.

What is JSONP?

If you spend some time thinking about how JavaScript operates in a browser, you will probably arrive at the realization that the policy does not seem to effect script tags. Specifically, how many of us get JQuery from the Google CDN or some other source that is not the same origin as our current page? (I hope that answer is “most”, by the way.)

That same logic, combined with some DOM knowledge, would probably lead you to experiment with loading other types of JavaScript, aside from a library, using script tags. Since JavaScript has a global scope, if you were to reference a function that could handle that data you could then link it with your application and do something interesting with it.

JSONP is born.

The idea of JSONP is actually pretty simple: toss a script tag into the DOM with a reference to a resource that returns JSON data. Have the server return said JSON with “padding” (the “P” part of JSONP) that executes a function wrapping the incoming data. In order for this to work properly, the server API must also support JSONP. Typically, the function name is named as the callback parameter. For example, let’s pretend you are accessing a service that returns JSON weather information that looks like:

http://www.weather.com/90210

Which returns JSON like:

{"zipCode": "90210","location": "Beverly Hills","high": "85 degrees","low": "55 degrees"}

Adding JSONP support would result in a URL that looks like:

http://www.weather.com/90210?callback=MyCallback

And returns:

MyCallback({"zipCode": "90210","location": "Beverly Hills","high": "85 degrees","low": "55 degrees"});

This data is now exposed to the rest of the application as long as MyCallback is a valid function name in the global scope.

Most JSONP implementations in libraries like JQuery will automatically generate callback functions as well as cleaning up inserted script tags when the callback executes, but the basic idea is actually pretty simple.

Why To Use JSONP?

JSONP is universally supported on just about every browser and a lot of useful APIs have support for it already in place (Yahoo, Twitter, Google, etc.) Aside from the regular XMLHttpRequest and DOM differences, nothing really special needs to be done to make it work across different browsers. Finally, since it has been around for a bit, there are pretty mature JSONP plugins for just about every major library and framework around (JQuery, YUI, Prototype, etc.)

Why NOT To Use JSONP?

Deciding against using JSONP is directly related to how it works. First of all, the only HTTP method you can use is GET since that is the only method script tags support. This immediately eliminates the use of JSONP as an option to interact with nice RESTful APIs that use other HTTP verbs to do fun stuff like CRUD. And while we’re on the subject of GET, note that using anything other than URL parameters to communicate with the server API (e.g. sending some JSON over) is also not possible. (You could encode JSON as a URL parameter, but shame on you for even thinking that.)

Secondly, since you are relying on a script tag, there is no uniform way to catch errors. Some browsers will allow attaching a callback to the script tag (e.g. onError() {}), but it is not universally supported (any guesses on which browsers don’t support it?) and therefore can’t be relied upon without some unfortunate hacks. If the called API is doing syntactically correct things like returning an HTTP 404 response code when a resource cannot be found, there is no way to easily catch that error condition or parse the response body for further information. Instead, what typically takes place are APIs that return HTTP 200 status codes but contain JSON describing an error condition. There are worse things in the world, but I cringe at that approach these days.

Finally, keep in mind that JSONP is really a security vulnerability. It happens to be an extremely useful one, but the same origin policy does provide value in protecting the application and using JSONP is willfully opening up a hole that could potentially become an attack vector.

What Are Some Alternatives?

The easiest alternative is setting up a proxy to an external API through the same origin as the web application. This can be done pretty easily using tooling built into web servers like Apache and Nginx.

If you are able to remove support for IE7 and don’t care much about Opera, the end game is Cross-Origin Resource Sharing (CORS). This solution allows the full array of HTTP verbs and errors (including POSTing JSON) and has a greater level of security than JSONP. It has decent support on IE8+ (with some provisos) and is relatively easy to implement on the server. The downside is that many popular public JSON APIs today don’t yet support it.

Conclusion

CORS >= Proxy > JSONP (most of the time).

Software is Harder than it Looks

Last night, I was thinking about the game Words with Friends for the iPhone. I haven’t played, but I know a lot of people who love it. It’s basically an online version of Scrabble with a twist. I think the maximum time a player can take to finish a turn is a whole week. Games take a while to finish. Most of the people I know who play are participating in a handful of games at once. The genius of this game, and what makes it perfect for mobile devices like the iPhone, is that you can play it on your own time. On the bus and have a few minutes? No problem … that’s the perfect time to play your turn.

On the surface, this game seems super simple. “It’s basically Scrabble but online.” The concept sounds so easy and obvious that the implementation is taken for granted. People don’t understand how complex it is because it just works.

I don’t have any first hand knowledge of the game, but as simple and obvious as it seems, the implementation is likely a lot more difficult than that. Aside from the actual app that you download from the App Store, there is a whole infrastructure running to make sure games and players are kept in sync. Again, I don’t know the folks who built it but I’ll bet they spent a lot of time and effort making it work as well as it does. And I’ll bet they are constantly working on it, making it better and keeping it running smoothly.

Another example is an application I did work on: The TWCableTV application for the iPad. This application also seems simple. The first version just streams live video. That’s it. Since the first release, we’ve added Remote DVR and other features, but the vast majority of the time it’s used to simply watch TV.

What might be surprising is only a small handful (3-4) of people actually worked on the app that gets installed on the iPad. At least 7-10 people worked on the services that talk directly to the iPad, a bunch more to test the app, yet another 2-3 on making sure the television streams were setup and continue to function properly, a whole host of people working in our data centers to keep the lights on, and lots and lots of previous work by other teams across the company creating ways to programmatically retrieve things like a billing data and validate usernames / passwords. Add a few lawyers in the mix to keep us from violating our contractual agreements and the number of folks who had a hand in the application, directly or indirectly, is at least 50 and likely more in the realm of a hundred.

That’s a lot of people for an application that just plays TV.

My point is that ideas for software are comparatively easy to come up with. The implementation of ideas is the hard part. We definitely weren’t the first to think “it would be cool to let people watch live TV on an iPad”. However, we were the first to implement it. And it was hard.

Software is usually like that.

Often, when a person I just met learns I worked on an iPad application, they’ll say something like “I have a great idea for an iPad application that [insert great idea here]” That’s similar to saying “I have a great idea for a skateboard that hovers or a device that converts matter to energy, transports it, and then converts it back to matter”.

The amazing thing about the titans of the computer industry is not that they had great ideas but that they actually implemented the ideas. Steve Jobs didn’t just think of the iPhone. He built it. He hired the talented team of engineers, designers, testers, and even lawyers to make it all work.

It takes work. It takes tenacity. It’s, well … just like everything else great in life I suppose.

It’s hard.