Entries classified under web

IBM says LAMP users need to grow up.

Woodshed!

Let's do it:

According to Daniel Sabbah, general manager of IBM's Rational division, LAMP -- the popular Web development stack -- works well for basic applications but lacks the ability to scale.

Nope. We call bullshit. After wasting years of our lives trying to implement physical three tier architectures that "scale" and failing miserably time after time, we're going with something that actually works.

If you look at the history of LAMP development, they're really primative tools ... the so-called good enough model. The type of businesses being created around those particular business models are essentially going to have to grow up at some point.

No. The LAMP stack is a properly constructed piece of software. Features are added when an actual person has an actual need that arises in the actual field, not when some group of highly qualified architecture astronauts and marketing splash-seekers get together to compete for who can come up with the most grown-up piece of useless new crap to throw in the product.

The LAMP model works because it was built to work for and by people building real stuff. The big vendor / big tools model failed because it was built to work for Gartner, Forrester, and Upper Management whose idea of "work" turned out to be completely wrong.

Now you're saying that the primitive yet successful LAMP model should adopt the traits of the sophisticated yet failing big vendor model.

I believe that in the same way that some of those simple solutions are good enough to start with, eventually, they are going to have to come up against scalability, Sabbah said during a press conference at the IBM Rational User Conference in Las Vegas.

We can't scale? Really? Are you insane?

Alright, that last jab may have been a bit unfair. I think what Sabbah is really talking about is PHP. I can't be sure but none of Yahoo!, Amazon, Ebay, or Google seem to be using PHP widely on their public sites [1]. But then again, they aren't using Websphere/J2EE, .NET, or other scalable physical three tier architectures either.

While we're talking about architectures, I'd like to jump into a brief commentary on what's really at the root of the debate here.

There arewere two widely accepted but competing general web systems architectures: the Physical Three Tier Architecture and the Logical Three Tier Architecture. IBM (and all the other big tool vendors) have been championing one of them and LAMP is a good framework for the other (although you'll rarely hear anyone admit that LAMP provides an overall architecture).

The Physical Three Tier Architecture

Many large enterprise web applications tried really hard to implement a Physical Three Tier Architecture, or they did in the beginning. The idea is that you have a physical presentation tier (usually JSP, ASP, or some other *SP) that talks to a physical app tier via some form of remote method invocation (usually EJB/RMI, CORBA, DCOM) that talks to a physical database tier (usually Oracle, DB2, MS-SQL Server). The proposed benefits of this approach is that you can scale out (i.e. add more boxes) to any of the physical tiers as needed.

Great, right? Well, no. It turns out this is a horrible, horrible, horrible way of building large applications and no one has ever actually implemented it successful. If anyone has implemented it successfully, they immediately shat their pants when they realized how much surface area and moving parts they would then be keeping an eye on.

The main problem with this architecture is the physical app box in the middle. We call it the remote object circle of hell. This is where the tool vendors solve all kinds of interesting what if type problems using extremely sophisticated techniques, which introduce one thousand actual real world problems, which the tool vendors happily solve, which introduces one thousand more real problems, ad infinitum...

It's hard to develop, deploy, test, maintain, evolve; it eats souls, kills kittens, and hates freedom and democracy.

Over the past two years, every enterprise developer on the planet has been scurrying to move away from this architecture. This can be witnessed most clearly in the Java community by observing the absolute failure of EJB and the rise of lightweight frameworks like Hibernate, Spring, Webwork, Struts, etc. This has been a bottom up movement by pissed off developers in retaliation to the crap that was pushed on them by the sophisticated tool vendors in the early century.

Which brings us nicely to an architecture that actually works some times and loves freedom.

The Logical Three Tier Architecture

More specifically, the Shared Nothing variant of the Logical Three Tier Architecture says that the simplest and best way to build large web based systems that scale (and this includes enterprise systems goddamit) is to first bring the presentation and app code together into a single physical tier. This avoids remote object hell because the presentation code and the business logic / domain code are close to each other.

But the most important aspect of this approach is that you want to push all state down to the database. Each request that comes into the presentation + app tier results in loading state for a set of objects from the database, operating on them, pushing their state back down into the database (if needed), writing the response, and then getting the hell out of there (i.e. releasing all references to objects loaded for this request, leaving them for gc).

That's the rule.

So the physical database tier and the physical presentation + app tier make up our logical three tier architecture but I'd like to talk about one other latch-on piece of this setup because it's interesting to contrast it with how the Physical Three Tier purists deal with the same problem.

Fine Grained Caching

Some mechanism for caching becomes really important when you decide that you are spending too much money on hardware (note that both of these architectures will scale up and out, on each physical tier independantly, for as far and wide as you can pay for hardware). Adding some form of caching reduces the amount of hardware needed dramatically because you've reduced utilization somewhere.

In the physical three tier architecture, there is generally a lot of sophisticated mechanisms for caching and sharing object state at a very granular level in the app tier to reduce utilization on the the database and increase response time. This is cool and all but it increases utilization on the app tier dramatically because so much time is now spent managing this new state.

The introduction of state (even just a little state for caching objects) forces the app tier to take on a lot of the traits of the database. You have to worry about object consistency and be fairly aware of transactions. When that's not fast enough what ends up happening is that more fine grained caching is added at the presentation tier to reduce round trips with the app tier.

Now you have three places that are maintaining pretty much the same state and that means you have three managability problems. But this is, you know, cool because it's really complex and sophisticated and the whiteboard looks interesting and lots of arm waving now.

Screw Fine Grained Caching

Shared Nothing says, screw that - the database is the only thing managing fine grained state because that's it's job, and then throws up caching HTTP proxy server(s) in a seperate (and optional) top physical tier. Cached state is maintained on a much simpler, coarse grained level with relatively simple rules for invalidation and other issues.

When the Shared Nothing cache hits, it provides unmatched performance because the response is ready to go immediately without having to consult the lower tiers at all. When it misses, it misses worse than the fine grained approach because chances are good you'll be going all the way to the database and back. But it turns out that it usually doesn't matter. My experience says that you get as good or better performance with the coarse grained approach as you do with the fine grained approach for much less cost, although it's hard to measure because the savings are distributed in very different ways.

The Shared Nothing + Caching Proxy setup scales like mad and I don't just mean that it scales to really massive user populations. It scales low too. It's easy to work with when you're developing and testing on a single machine. It's easy to have a simple prototype app done in a day. It's easy to teach someone enough that they can go play and figure stuff out as they go. It's easy to write tests because the entire system is bounded by the request and there's no weird magic going on in the background.

The big vendor / big tool architectures sacrificed simplicity and the ability to scale low because they decided that every application was going to have one million users and require five 9's from the first day of development.

As I write this, Bill de hÓra postulates: All successful large systems were successful small systems. I believe him and what that means to us right now in this article is that it is exceedingly hard to build large systems with the big vendor / big tool approach because it is exceedingly hard to build small systems with the same.

Let's get back to the woodshed

While Sabbah was critical of LAMP's capabilities, he said IBM is going to ensure companies which started with that model will be able to "grow and change with the rest of the world".

He believes most businesses want technology that is stable, evolutionary, historical and had support.

L A M P = (S)table (E)volutionary (H)istorical (S)upport

"What we are trying to do is make sure businesses who start there [with LAMP] have a model, to not only start there but evolve into more complex situations in order to actually be able to grow," he said.

This is where I really wanted to jump in because I think this mentality is holding back adoption of very simple yet exremely capable technology based purely on poor reasoning. This view of systems design says that complexity is required if growth is desirable and that complex situations can only be the result of complex systems.

There's a guy who just spent 50 years or something locked in a room writing a 1200 page book proving that this is just wrong. It would appears that there is very little relationship between the complexity of a program and the complexity of the situation it produces.

The complexity for complexity mindset is the bane of a few potentially great technologies right now:

• Static vs. Dynamic Languages
• J2EE vs. LAMP
• WS-* vs HTTP

I like to complain when someone calls Python a scripting language because the connotation is that it is simple. But it is simple, right? So there shouldn't be any complaining. I'm not objecting to someone calling Python simple, I'm objecting to then saying that because it is simple, it must only be capable of simple things.

Ahhh and we've finally reached the end of this one

"You've seen us do a lot with PHP and Zend and you'll see us do more. I can't say more. It [PHP] needs to integrate with enterprise assets but it needs to remember where it came from and where its value was. It can't start getting too complex as it will lose its audience," Sabbah said.

The need for complex systems in the enterprise was and still is greatly overestimated. The trick isn't to make PHP more complex, it's to make the enterprise less complex. You need to equate complex requirements with complex systems less and start asking "do we really need this?" more.

The funny thing about all this is that my opinion on this matter has formed largely based on concepts that you guys told me, so I'm sure you'll pull through on this one.

The site is back up after two days of downtime but if my theory is correct then no one noticed. Our records indicate that 95% of the (20) people reading this site do it via a proper RSS reader, so no one should have noticed a difference between the site being down and me just going on a quiet spell for a few days. The other traffic comes from search, and seeing as how my pages look almost flawless coming from google's cache, I don't see why I have to care about downtime all that much. Which is good because this site runs on and old P300 GNU/Linux box off my cable connection and has been since 2003 (talk about the 'ilities!).

Anyway, cha-changes... I usually don't get very personal in posts -- tending more toward commentary on larger issues -- but there's so much going on right now that I have to write some of this down if just for the archives.

One of the things I never talk about here is my day job. This isn't because it isn't related (most of my posts on technology are at least relevant to my day job) but because my employer had no formal policy on public discourse. That combined with the fact that I really enjoy being able to say whatever the fuck I want here led me to keeping work completely out of the picture.

But today, while waiting on The Exit Interview With The Human Resources Department, I was reading over my non-disclosure agreement when I noticed that it was fairly sane, forbidding only the disclosure of confidential company information. Me being employed there isn't confidential so I guess it's okay to say that I was employed by Sterling Commerce from March 2000 until about five hours ago.

Sterling has been around a loooong time for a tech company. This PR blurb does a surprisingly good job of providing some actual information on the company's background:

A pioneer of electronic data interchange (EDI) and secure file transfer technology, it has provided business process automation solutions to Fortune 500 companies and the world's largest banks for 30 years. Formerly a division of Sterling Software, Sterling Commerce became an independent corporation in March 1996, through an initial public offering. The company quickly grew to become one of the world's largest independent providers of multi-enterprise collaboration solutions. SBC Communications acquired Sterling Commerce in March 2000.

They're one of the few NASDAQ tech companies that didn't pop with the bubble. I attribute this mainly to the fact that they actually did something: boring old valuable-as-hell EDI and other tough business integration tasks. Most of their current initiatives involve doing the same boring old valuable-as-hell EDI stuff but this time over internet pipes instead of bisync modems.

I won't say much more about Sterling because talking about it is still gray to me and I don't want to piss anyone off (at Sterling). I really don't even know if I can talk about talking about it.

I would like to mention one product I worked on there because it was a kind of epiphany to me. The product was Webforms (re-branded Web Forms it seems <sigh>). It's just a web app that let smaller Mom-and-Pop shops talk EDI to all the big boys running them thar' expensive mainframes and bisync modems.

Now this was an important situation for a couple of reasons. First, Webforms was an extremely functional web app for the time (~2000). The web was all about animated GIFs, Java applets, <blink> and <marquee> tags, and was just very oh-we-can-make-money-by-just-looking-really-cool-ish in general. And these guys show me this plain old ugly white pages with big forms all over the place web-app. Maybe you'd see a small company logo here and there, maybe not. Fuck it, your company logo isn't helping anyone with their orders.

It was the plainest, most vanilla piece of pure gold I'd ever seen. Clay Shirky figured out the internet in 1996, I figured it out in 2000 while at Sterling Commerce. The internet is here to do the exact same shit we did before the internet was here but cheaper, broader, and with less fuss. ROCK ON! Let's do it. I'm pumped.

Unfortunately that leads me to the second reason Webforms was an enlightening experience. It was developed in a company that really hated the web. We owned a massive EDI VAN and we had desktop Windows and UNIX products... that let you talk to the VAN. This is what 2000 people at Sterling knew and understood. Combine that with the following snippet from whatis.com's definition of VAN:

... Before the arrival of the World Wide Web, some companies hired value-added networks to move data from their company to other companies. With the arrival of the World Wide Web, many companies found it more cost-efficient to move their data over the Internet instead of paying the minimum monthly fees and per-character charges found in typical VAN contracts.

And then these Webforms guys go and plop a web app down in the middle of all this - it wasn't pretty. Needless to say, excitement about the product never really made it out of development for the years I worked on it but seems to finally be picking up a little now.

So the other big thing "I got" about the internet was that providing real value would usually require being extremely disruptive, which would require pissing a lot of people off, which I will now dub Tomayko's Law of The Internet:

To provide value on the internet, you must piss someone off.

Cue:

• Napster / RIAA
• Wikipedia / Britannica
• RSS / Journalism
• Skype / The Entire Phone Industry (holy shit!)
• Free and Open Source Software / Microsoft (the whole industry is being flipped on its head in case no one noticed)
• BitTorrent / Television (yes, bittorrent will replace your television stations whether you like it or not).
• Google / Yahoo <snark>
• Corey Doctorow / Disney
• And the list goes on...

All provide value by pissing someone off.

Notable exceptions from this list are AOL, WS-*, Semantic Web, Java Applets, RealAudio streaming, Microsoft Passport, top posting, and DRM. Each of these technologies piss everyone off; this somehow lessens the value dramatically. So the trick is to find a group that's roughly the size of, let's say, an industry and piss them off as much as possible.

That's an insanely brief description of what I've learned at Sterling and this one is already running long so more on the ch-changes later.

Dave Winer invites us to get out the secret decoder ring:

Thankfully there hasn't been much format-bashing going on in the last year or so, but it looks like it might be starting again, so let's do something before it becomes a problem.

...

I've been reading a certain mail list, and observing that a certain person is now posting again, after a long hiatus that allowed us to find this relatively peaceful status.

Let's make it clear that we like the way things are now, and won't stand by and say nothing if the level of discourse starts slipping. Thanks for listening.

Eek! That last paragraph reads like some kind of threat, Dave, wtf?

My guess is that the mystery antagonist is Mark Pilgrim and the mailing list is that of the IETF Atom Syntax WG and maybe one of these is what has Dave giving out threats or mandates or whatever?

Well, I for one would love to see Mark back and stirring up the pot in full force. Somewhere deep down inside Dave must feel the same way because it's painfully obvious that he's provoking the exact behavior he is claiming to oppose. Mark is sometimes.. errmm.. harsh/pointed/blunt but he's also right most of the time.

Dave: you don't have a monopoly on being an asshole. Mark does a great job too. We like both of you for the assholes you are.

Mark: piss on Dave, man... Start posting again.

P.S. How's that for Canons of Conduct?