Entries classified under rest

Bill de hÓra's recent piece makes an interesting connection between two waffling technologies - Semantic Web and Web Services:

The case of the DL and ontology world coming to the Semantic Web and worrying over queries that will blow up in the engines is much like the case of the enterprise world coming to the Web worrying over type systems and discovery languages. The likeness is not fleeting - both the Semantic Web and Web Services advocates have been busy building competing technology stacks in the last decade. They have valid points and good technology but the need or demand for such precision in the Web context has been overestimated.

This reminded of an excellent yet rarely cited piece by Cory Doctorow published by O'Reilly in late 2001 entitled The Carpet Baggers Go Home:

The Internet is unpredictable. It's non-goddamned-deterministic.

...

The Internet is full of fantastically useful and frustratingly unavailable services, from the elegant simplicty of Weblogs.com's XML-RPC interface that accepts a URL and a link-title and shoves 'em on top of the stack of recently updated sites, to the unaffiliated public CVS servers that pock the Internet like so much acne. They work well enough, on average, and if they were all to fail suddenly and at once, the Internet would kind of suck until they came back online. But there are enough of these little tools, enough ways of finding and manipulating information, that users can interpret unreliability as damage and route around it, finding alternate means of communicating and being communicated at.

...

The next generation of Internet entrepreneurs will be people who understand this. They'll be working to provide unreliable services that work in concert with other unreliable services to provide a service that works on average, but not predictably at any given moment. They'll challenge the received wisdom that customers are hothouse flowers, expensive to acquire and prone to wilting at the first sign of trouble. These entrepreneurs will build services that are so compelling that they'll be indispensable, worth using even if the service flakes out when you want it the most.

And Clay Shirky:

Much of the proposed value of the Semantic Web is coming, but it is not coming because of the Semantic Web. The amount of meta-data we generate is increasing dramatically, and it is being exposed for consumption by machines as well as, or instead of, people. But it is being designed a bit at a time, out of self-interest and without regard for global ontology. It is also being adopted piecemeal, and it will bring with it with all the incompatibilities and complexities that implies. There are significant disadvantages to this process relative to the shining vision of the Semantic Web, but the big advantage of this bottom-up design and adoption is that it is actually working now.

And even Richard P. Gabriel:

From 1984 until 1994 I had a Lisp company called "Lucid, Inc." In 1989 it was clear that the Lisp business was not going well, partly because the AI companies were floundering and partly because those AI companies were starting to blame Lisp and its implementations for the failures of AI. One day in Spring 1989, I was sitting out on the Lucid porch with some of the hackers, and someone asked me why I thought people believed C and Unix were better than Lisp. I jokingly answered, "because, well, worse is better." We laughed over it for a while as I tried to make up an argument for why something clearly lousy could be good.

There's been a lot of good discussion around Udell's End HTTP abuse article. We need to get this figured out because it's almost embarrassing to be an advocate of standard approaches to building web applications when something as fundamental as the correct use of HTTP GET is butchered so often. Unfortunately, misuse of HTTP GET is just the tip of the iceberg. There's a whole slew of HTTP abuse going on out there (often in the form of neglect) that can be laid at the footstep of two parties: frameworks and evangelists. The frameworks suck and the evangelists aren't trying hard enough (I consider myself in both camps, btw).

The small community that is coalescing around REST/HTTP should prioritize the following objectives above anything else at this point:

1. Raise awareness of what HTTP is capable of.
2. Fix the tools.

HTTP Kicks Ass

We need to raise awareness of what HTTP is really about. If you're reading and understanding this, then you've likely had the Ah-ha moment based on the realization that HTTP isn't just a beat up old protocol for transferring files from a web server to browsers (if you haven't, read this); but this understanding is not common. The large majority of smart technical people believe that HTTP is legacy technology: an old protocol, maybe a step above gopher, that has somehow hung around through the years. Something to be dealt with, not taken advantage of. We need to show how limiting this mind-set is.

Many of us were first introduced to the true capabilities of HTTP via the REST architectural style. If you were like me, Mark or Paul (or both at the same time) forcefully induced the REST epiphany on you against your will and then you went and re-read RFC 2616 while slapping yourself on the forehead repeatedly. The correlation between the architectural concepts described in Roy's thesis and the implementation semantics described in the RFC were clear as a bell. HTTP was no longer a simple mechanism for exposing a directory of files and executable processes to a browser, it was the essence of web architecture.

Here's the thing though: most people don't read RFC's! They hate RFCs. Reading RFCs ranks close to doing taxes for most people. The only thing worse than reading RFCs is reading PhD thesis'.

The evangelist needs to reach these people somehow and I really don't think it's going to be through describing the architectural concepts of REST so much as it will be through describing the here's-what-you-can-do-right-now-in-the-real-world benefits of HTTP. It's a fine line, I know, but I think it's important.

How do we give people the Ah-ha! without requiring that they read a thesis and an RFC?

A Three Legged Dog

I look at what Zeldman, Meyer, and others are accomplishing with the Designing with Web Standards movement and it seems a good model. They emphasis the correct use of standard CSS, (X)HTML, and DOM scripting (the three legged stool) to achieve enormous benefits over proprietary web design techniques. They have books and a cluster of weblogs that show designers how to reap the rewards of this system. It's been a smashing success and is only gaining traction.

Can we take a page from their book? In my opinion, we're just as entitled to the phrase Designing with Web Standards as they are. We have a three legged stool too: HTTP, URIs, and XML. Except our stool is more like a three legged dog - you can get around but it is not quite optimal. Why is this?

My feeling is that we haven't done a good enough job of showing examples of what the correct use of HTTP, URIs, and XML looks like in the real world. Joe Gregorio's excellent column aside, there just is not a lot of here's-how-you-get-shit-done-with-http type content available on the web, led alone books or magazine articles. We're amazingly talented at pointing out when something is being done wrong (e.g. WS-*, non-safe/idempotent GET, incorrect charset, etc.) but we suck at showing how to do it right in the first place.

(Here's some more pictures of three legged dogs because three legged dogs are the shit.)

To Hell With Bad Web Frameworks

Why are we having such a hard time showing correct use of HTTP and URIs? Because our tools suck. How are we suppose to show how to use HTTP and URIs properly when the tools and frameworks actively discourage practicing standards? Our incessant ranting on correct use of web technology is filed by many into the not living in the real world drawer. We're assholes.

In many ways this is the same battle that the Designing with Web Standards crowd has been fighting with their tools - the browsers. How can you preach standard use of (X)HTML, DOM, and CSS when the browsers have such poor support for them? Those guys drew a line and said To Hell With Bad Browsers and it's paying off. I think we need to take on a similar attitude and start expecting more from our web frameworks.

Every web framework I've ever worked with (Apache, CGI, Java Servlets, Quixote, Webware, Ruby on Rails, PHP, ASP.NET, CherryPy) were extremely limited in their support for the full set of capabilities provided by HTTP.

For instance, which frameworks ...

• ... help implement content negotiation properly?

• ... provide facilities for implementing a smart caching strategy for dynamic content? (proper use of If-Modified-Since, Expires, Cache-Control, etc.)

• ... make dealing with media types easy?

• ... make dealing with character encodings easy?

• ... encourage the use of standard HTTP authentication schemes?

• ... have a sane mechanism for attaching different behavior to different verbs for a single resource?

• ... help ensure that URIs stay cool?

• ... make dealing with transfer encodings (gzip, compress, etc.) easy?

• ... help you use response status codes properly? (e.g. Nearly all dynamic content returns either a 200 or 500).

Sure, some frameworks have tricks for portions of the list but there should be documented, well-thought-out mechanisms for implementing these facets of HTTP. Let's face it, if you want to do something outside of exposing well-known static representation types from disk for GET, or process application/x-www-urlencoded data via POST, you're off the radar for most web frameworks. I'm not saying that other things aren't possible, I'm saying they aren't supported well.

Shutting this one down

I've rambled for to long already. I have more to say on this but I'll try to keep it to short and focused posts over the next week or so.

To sum up, we need a good implementation of HTTP/1.1 that provides a real framework for building standards based web applications. We then need to advocate and illustrate the correct use of HTTP/URIs/XML as a killer technology that has been hiding right under our noses by showing the benefits of using the system correctly. Until we get this stuff straightened out, expecting people to use GET properly is unrealistic.

...

Sidebar: I just noticed that Leigh Dodds beat me to it:

[Udell] then continues by exploring the ease of using GET versus POST on the client-side. I think the fault actually lies on the server-side. Specifically, with existing web applications frameworks.

Word.

Patrick Logan wonders what I'm referring to when I use the word success in this paragraph from What WS-* Got Wrong:

From the beginning, WS-* has been approached incorrectly. The approach that we're advocating is to first embrace and understand existing web architecture and then to gradually enhance (constrain) it to meet the needs of new requirements. We advocate this not because we think it's a better way to be successful, we think it's the only way to be successful.

That's some claim! Is "success" even defined in this case? -Patrick

Success is the widespread adoption of some form of interoperable communications between coordinated and uncoordinated machine agents over large network bodies.

What I was trying to say here is that my beef with WS-* has less to do with architectural and technical issues and more to do with pragmatism and practicality. I'm fully aware that these technologies could be adopted and would work more or less as specified. My problem is that WS-* won't be adopted in any significant way because it does not exhibit the traits necessary for a technology to be adopted on a large scale.

If someone personally coordinated 200 people in various organizations, I'm sure they could implement solutions on top of WS-* given a long enough time-frame. If 200 people are working separately, and are left to their own devices, in a more organic, web-like environment, WS-* has much less chance of being used successfully.

I see REST vs WS-* as a simple case of worse is better (except in many ways I think REST is the more technically correct so bonus points there). It does exhibit the traits required for technology to be widely adopted and what's more, it already has been widely adopted.

Mark Baker brings up an important aspect of the WS-* vs. REST situation that I've been hoping to speak about myself:

Yes, absolutey, we'll need the capabilities that the WS-* stack provides (for the most part). But do we need the WS-* stack itself to provide those capabilities? I'm certainly all for trying to reuse existing specs where that makes sense. The issue though, is that with most of them, they explicitly or implicitly require disregarding a key constraints of REST, disrespecting Web architecture, or both. WSDL's probably the poster child for this, as its raison d'etre is primarily to encourage rejection of REST's uniform interface.

Right. The problem with WS-* isn't that its trying to solve the wrong problems or that nothing valuable has been produced. The problem is the general disregard for existing web architecture. WS-* tried to take a revolutionary approach to something that required only incremental improvement. HTTP and URIs were treated as legacy technology that would provide a quick and dirty mechanism for bootstrapping this special next generation of web technology.

The REST advocates' criticism of WS-* generally comes off as an attack on technology or architecture but what really pisses the REST people off is the WS-* approach. From the beginning, WS-* has been approached incorrectly. The approach that we're advocating is to first embrace and understand existing web architecture and then to gradually enhance (constrain) it to meet the needs of new requirements. We advocate this not because we think it's a better way to be successful, we think it's the only way to be successful.

An example of the approach we're advocating is Bill de hÓra's recent proposal for reliable messaging over HTTP, (HTTPLR). The difference between the HTTPLR approach and the WS-ReliableMessaging approach is that Bill's proposal builds on simple proven web architecture without requiring additional apparatus.

Anyway, what we need to learn from this is that most successful technologies aren't successful on accident. HTTP and URIs were generally understood only at a very technical / bits-on-the-wire level -- the RFCs existed but Roy Fielding's architectural description came much later and I think it hurt us. We didn't understand how HTTP and URIs formed an overall architecture and how vital that architecture was to the success of the web. If the larger technical community had had an architectural understanding of the web, we might not be in such a mess.

What many of us are now wondering (and have been wondering for some time) is how much longer WS-* is going to continue down a path of incompatibility with web architecture and whether the work that's in progress or that has been completed by WS-* can be reconciled to work under the constraints of web architecture. The feeling I get is that a lot of the REST advocates feel so spited from years of being ignored that they would rather sit back and watch the WS-Building burn to the ground and build from scratch, which is unfortunate for everyone really.

I think Jonathon Schwartz (President/COO, Sun Microsystems) just joined the loyal WS opposition:

5. Web services may collapse under its own weight.

No one at the conference said this. Those are my words. I'm beginning to feel that all the disparate web service specs and fragmented standards activities are way out of control. Want proof? Ask one of your IT folks to define web services. Ask two others. They won't match. We asked folks around the room - it was pretty grim. It's either got to be simplified, or radically rethought.

As you know, I also believe simplicity and volume always win - and that today's web services initiatives are in danger of vastly overcomplicating a very simple (really simple) solution.

What's been apparent to those in the trenches for the past year or two is finally starting to find its way up the chain of command.

So there's buzz around some new quasi-Open Source MQSeries¹ clone named AMQ. Supposedly, this thing is going to be used for Web Services and Service Oriented Architectures (SOA) in the financial industry initially but then grow horizontally. The idea being that message queuing provides a lot of the functionality not directly handed to you by HTTP, like reliable delivery, publish/subscribe, multiple subscribers, etc.

The article was kind of weird though because some of it doesn't make any sense:

Not only has AMQ drawn the participation of several banks, but also companies such as Red Hat Inc., Novell Inc. and Sun Microsystems Inc. are considering building AMQ into the kernels of Red Hat Enterprise Linux, SuSE Linux and Solaris, respectively, Davies said.

Wtf? No one is building message queuing functionality into the kernel, bro. That's user-land stuff.

Although Davies said he is an avowed Java aficionado and a supporter of the Java Message Service API, he said JMS "fails on the non-Java side as a transport mechanism."

That's like saying JDBC sucks at indexing my tables. JMS is a standard Java interface that various message queuing implementations implement so that coders don't have to write to vendor specific APIs. There's no transport mechanism involved. If AMQ is truly a MQSeries clone, it will have a JMS implementation wrapped around it within a month. Maybe the original intent was to say that no existing F/OSS MQ implementations have open wire formats? Or something?

I think what this article may have actually been trying to say (although we may never know because it is so full of hallow buzzwords) is something like the following:

1. We, the financial industry, hate proprietary shit.

2. We are tired of waiting around for you WS and SOA folks to actually produce something we can use.

3. We just want a non-proprietary MQSeries that we can use over the internet.

4. There are no open protocols for message queuing and, like we said, we need message queuing. Don't even start with that abstracting non-functional concerns crap. We need message queuing, period.

5. We have developed an open protocol for message queuing. We've done this, somewhat mysteriously, behind closed doors.

6. We've implemented this open protocol in C and C++ and we're going to make these implementations available under some kind of open license.

7. We're going to call this Web Services and SOA because that's what my boss' boss said he wanted and since none of you people can tell me what any of this stuff is, I don't think you can say it's not what we've built.

It seems reasonable to believe that all we're looking at here is an internet friendly protocol and app for chucking messages around in a secure manner which, to be honest, is probably all that's needed. Except it seems to be having an identity crises because it's talking about Web Services and we all know by now that Web = HTTP/URIs and AMQ will likely use neither.

REST is already eating one half of the WS-Sandwich on the web with WS-* retreating to Enterprise Level tasks. I don't think it's impossible to imagine a not-to-distant future where ad-hoc message formats pop up and start flying over this AMQ thing to eventually eat the other half of the WS-Sandwich.

WS-Crust anyone?

Footnotes

^1. I don't care that they rebranded it WebSphere MQ. It's MQSeries... Shut-up.

Yahoo! launched a Web Services interface today. I'm gushing with joy over their decision to follow proven web architecture and select a simple REST model over a more complex WS-* model. Let's take a look at an item pulled from their FAQ:

Q: Does Yahoo! plan to support SOAP?

Not at this time. We may provide SOAP interfaces in the future, if there is significant demand. We believe REST has a lower barrier to entry, is easier to use than SOAP, and is entirely sufficient for these services.

Get used to that answer. You won't ever hear it from the evil tool vendors but you can be damn sure you'll be hearing it from the smart service providers that really are looking to court developers.

Also, it was extremely cool to serendipitously stumble across mine own How I explained REST to my wife article after passing from Yahoo!'s Constructing REST Requests page to Wikipedia's REST page.

Bill de hÓra describes the integrator's dilemma (also known as, WS-* vs. REST).

If dirt simple equates to good growth and better profits, then the missed opportunity arises when simple and simplistic are conflated. When the WS contingent are looking at the REST and syndication crowd and saying more or less, 'here's a nickel kid', they may want to stop and take a second look at what the kid is doing with that nickel.

I thought that was great. I suggest reading the whole piece. Bill's super smart.

The WS-* vs. REST thing is starting to heat up again due largely to James Governer's SOAP is boring, wake up Big Vendors or get niched piece last week. This led to the discussion at the beginning of Bill's post on whether Microsoft is ignoring developer demand for REST tools. I have a sinister theory on this (but not MS in particular) that I've been hanging on to for a while so let's just have at it. Bill has provided the voice of reason and I don't have anything to add so I'll just follow up with some good old fashioned religious rambling.

These are blogs, right? The last time I read the blogger's handbook, we were encouraged to make unsubstantiated claims about the intentions of others (especially big companies and industry cartels) without any real evidence, so here we go...

Some days the Powerbook gets more attention than the wife and so she snoops over my shoulder and starts asking a bunch of questions about whatever is on the screen. She doesn't really care, this is just the cue to shift my attention over to her. I usually do just that and say something like, Oh, this is some interesting stuff but nothing you would care about.

But on this day I decided that I would play along a bit and see how far I could take her into my world before she ran away screaming in terror.