Entries classified under weblog/python

Jon was a Computer Science major at Ohio State University taking a course in artificial intelligence. The professor had set up an interesting group project where each student was responsible for writing an insect program that would be matched against all the other student's insect programs in a really cool network based insect war simulation environment thing that rocked.

The insect programs had certain constraints set by the professor. Size, shape, speed, and other traits were selected by each student but there were rules such that you couldn't just turn all the dials to full.

Once the basic properties of the insect were fleshed out, code was written to specify how the insect should act. There was an API for determining where your insect was located on the grid, approximating positions of other insects, moving your insect, attacking, rotating, etc. Pretty standard stuff.

The professor decided to pair students up: one smart kid with one dumb kid; the students who were having a hard time in the class would be able to work closely with a student that was excelling. Each student was responsible for their own insect but they were to debate their designs with each other.

Jon was one of the smart kids and was paired with a kid that wanted to change majors. Jon's dumb kid rarely attended class and seemed to dislike CS in general. He wasn't even in class the day the assignment was handed out and so Jon set out on his own to build the coolest and most advanced insect program ever created.

Over the course of a few weeks he burned through code until his insect was capable of responding intelligently to a myriad of changes in environment. It knew to run when outmatched by judging the relative strengths and weaknesses of an opposing insect. It would attempt to strafe and stay behind other insects. It would stay close to corners to reduce the potential attack positions of other insects. It was The Coolest Insect Ever.

The day before the competition, Jon's dumb kid decided to come to class. Jon asked him if he had finished his insect, to which the dumb kid replied he hadn't even started but would finish it that day, in class. Jon grinned smugly and tried to explain to the poor fool that he himself had spent all week working on his insect and that it still was not yet complete. The dumb kid shrugged and started in coding something that would get him the damn credit for the project.

Right before the class ended the dumb kid asked Jon to take a look at his insect. Jon had to fight the urge to laugh out loud when he saw that the entire insect was a mere 25 lines of code that barely made it through the compiler and with some lines having no chance of even being executed. The dumb kid had not even configured his insect's basic set of traits but had left them at the professor provided defaults.

Looking more closely, Jon found that the insect was programmed to do the same thing every time it had a turn to move:

1. Rotate 90 degrees.
2. Attack.

Turn and then attack. That's it? Jon asked, to which the dumb kid replied, Do you think I'll pass?

Jon tried to give the dumb kid some ideas on making his insect more advanced but the dumb kid wasn't interested. Jon decided that the dumb kid would most assuredly not pass.

The next day the competition was on. The professor loaded up the simulation program and everyone hooked their insects into the system. The dumb kid was late and then couldn't figure out how to get his insect loaded up. Jon helped him out while mumbling something about futility...

Finally the simulation began and Jon was excited to see his insect perform well through the first full round. In the second round, Jon's insect would get stuck in one of the corners, enter an infinite loop, and be forcefully removed by the professor. One by one all other insects would be killed by other insects or removed by the professor due to logic problems - that is, all but the dumb kid's insect.

As he sat watching the dumb kid's lonely insect turn-and-attack, turn-and-attack, turn-and-attack, as if to mock the whole class, Jon was forced to re-evaluate his definition of cool in relation to computer programs.

This story was told to me by Jon Miller (UNIX sysadmin) in the first person. It has stuck with me as an excellent illustration of the power of simplicity and the devil that is the human tendency toward complexity.

I'm catching up on the happenings of PyCon and thought this IronPython keynote summary was interesting:

The news from this morning's keynote is: IronPython released (at last). The running joke in Jim Hugunin's talk was pretending that it had only been two months since he joined Microsoft. In fact, it took about eight months to work out how to do an open source release once he got to Microsoft.

That's not funny - it's a miracle.

The new plan is to release every two weeks until there is a 1.0 release. There are one-and-a-half engineers working on IronPython. Jim is spending half his time evangelizing dynamic languages and Python within Microsoft. The hope is that the next version of CLR will have better support for dynamic languages.

I started ranting here about how quality dynamic language support on the VM is coming down to a battle of who will get lucky and be the less clueless between Microsoft and Sun. The more I think about it though, the less I care. Here's why:

• Small teams are good teams. Nine women can't make a baby in a month and all that... I'd rather have Jim-and-a-half take two years writing a quality Python implementation on the CLR than to have an, uhh, more traditional Microsoft product in six months.

• Maybe Patrick is right. Actually, I'm sure he's right; I just haven't decided if that means there's no value running dynamic languages on the VM. I think we need something to break through on one of the VMs if we're ever going to move this into the enterprise. There's no way I could bring Python into my place of business on any serious level. This really comes down to it not being blessed as Enterprise Class (blech!) by Sun. It seems we need the VM to get in the door and then maybe we can just move quietly toward CPython with a Java/CLR bridge?

Kid 0.6 is available.

This is a feature release with some pretty important enhancements including Template Inheritence, Match Templates (kind of like XSLT's), cElementTree support, and a refined Python API. Quite a bit of time was spent on the documentation as well.

The Release Notes have more information on everything that has changed.

Kid is an XML based template language for Python that merges the best of Zope's TAL, PHP, and XSLT into a single coherent package. It was created to provide a simple method of generating dynamic, well-formed XML documents using familiar concepts from popular text templating languages.

Download

http://lesscode.org/dist/kid/

Release Notes

http://lesscode.org/doc/kid/0.6/notes.html

Project Information

http://lesscode.org/projects/kid/

These numbers look pretty bad... or good, depending on where you stand, I guess. I find it hard to believe that Rails is that much slower than Quixote/Durus for a truly similar task but hey I'm on the right side of this one so I won't complain too loudly.

Mike, you've given some info about the tests but it would be really helpful if you provided everything needed to recreate them. Specifically, the versions of Rails/Instiki/Quixote/MoinMoin and the specific URI stems you were pointing to with ab. Otherwise stuff like this can be taken the wrong way and turn into a big Python-vs-Ruby flame-fest. If your numbers are correct, I can't see the test setup making a whole lot of difference but still...

There are lies, damn lies, and statistics, unless you provide absolutely everything needed to recreate the statistics, in which case there are only lies and damn lies. We can reduce this a little further by removing the speculative concept of damnation, leaving us with just lies. This is probably the best we can do without killing everyone, which I'm not sure we can accomplish in any predictable time-frame.

LATER: Mike takes a closer look at his tests, revealing some issues with the comparison method.

Anyway...

I also wanted to comment on some of the RDBMS vs. Durus/ZODB stuff as I've been down this road myself and have been meaning to write about it a bit.

Mike compares relational persistence and object persistence:

Rails has a SQL object relational mapper; Quixote imposes no such thing but doesn’t get in your way of using SQLObject, ORM, Cucumber, any other o-r mapper, or no o-r mapper at all. I’ve done all of the above, but am slowly retiring all my SQL based code (where sensible) in favor of Durus, a ZODB inspired object database. Quixote sister project “Durus” provides an easy to use object database which is far more productive than force fitting objects to SQL. Durus certainly not limited to writing web apps.

Object databases like Durus/ZODB and relational databases are apples and oranges (or possibly oranges and mangos). I've recently been down the let's put everything in ZODB path and it comes with some problems:

• ZODB/Durus using FileStorage can only be accessed by a single-process.

• ZODB/ZEO or Durus/ClientStorage can be accessed with multiple processes over a network much like most relational databases but there is no concept of "multiple databases". If you want multiple databases, you need multiple ZEO servers running.

  LATER: Richard Jones corrects me on this and also the packing thing below.

• Durus has issues with multi-threaded access.

• There's not a lot of cheap hosting options that have ZODB/ZEO running out of the box or that will let you run any number of long-life processes.

• ZODB maintenance (packing) is a pain in the ass, IMO.

• Indexing is hard. The BTree classes get the job done but handling multiple indexes on objects can be tedious as you have to keep multiple BTrees in sync manually. (No I'm not talking about the full text index catalog crap. I need fast lookups on specific attribute values).

• Only Python code can access the database. I shrugged this off at first but found that it's actually really² nice to have data in some kind of language-independent storage.

There are a whole range of situations for which these issues aren't really issues at all and relational persistence has its own set of issues. But in my experience, making the decision between ZODB and relational has been less about what's better for the code and more about what's better for the data. That being said, I use ZODB quite a bit where I used to throw up hacks around pickles and shelves and also when I know I will have complete control over the application environment and don't have to play nice with others.

I've been enjoying David Heinemeier's (Ruby on Rails / 37signals) blog lately. The kid has excellent chin, which seems to be a requirement for dynamic language advocates mingling with the static language community. In his latest, Serving the koolaid with the facts, he takes on some of the criticism coming out of the Java weblogs that targets various aspects of Rails that go against conventional static language thinking.

It's really easy to take a concept that fails in a given environment and put it into the that's always bad part of your brain. Later, you see the same concept in a different environment and the that's always bad fiber starts blipping like crazy and your mouth starts saying things like, That's always bad.

There seems to be a trend emerging in conversation between the dynamic and static language camps that I believe is based in this flaw in the human mind. There are so many aspects of dynamic language programming that really just don't work in a static language environment and the opposite is also true; one is an ocean and the other dry land. Unfortunately, this isn't immediately apparent and those with strong backgrounds in one environment have a tendency to misjudge the viability of approaches in another.

Fish? That will never work.

I think the key to overcoming this problem is just to keep on talking. Over time it will become apparent that you can't measure dynamic language approaches under the static language microscope and then maybe we can just get on with things and start figuring out where the two environments can work together. It is perhaps not an accident that some of the most beautiful places in nature are the coasts where two strange environments meet.

Dan Gillmor points to an excellent example of distributed journalism in action over at Daily Kos. I was completely blown away by what I saw when I got there. I'm still trying to soak in all the background around the Plame Leak / Jeff Gannon thing but, to be honest, the specifics of this event are not as important to me as the general phenomenon occurring there. This seems an obvious glimpse into a future where involvement by the general population in issues of import to the general population is increased substantially. Herewith some rant and analysis of our present toolset and humble suggestion for improvement...

Kid 0.5 is available and I think I finally like it enough to start dumping out some example templates and API usage...

This is the second article following up Phillip J. Eby's Python Is Not Java. In the first article, The Static Method Thing, we took a look at how Java static methods differ from Python class/static methods. This time we're going to dive deep into the evils of getters and setters.

I had a very small number of complaints related to basing Kid on ElementTree. This came in two forms:

1. SAX and DOM are “standard” and while ElementTree is a drastically improved system for processing XML in Python, it doesn't matter because everyone already knows SAX/DOM.

2. “libxml2 is teh rawk!”

First, if Python's W3C DOM standard based xml.dom package were a movie, it would be called Elf, staring xml.dom. It's the episode of Little House on the Prairie where Alien asks Michael Landon for permission to marry his daughter. It does not belong here!

Next, in terms of pythonicness, libxml2 is almost worse than xml.dom but you at least get something for it: they don't even have a word to describe this level of “fast” and it comes along with XPath, RelaxNG, XSD, XML-Base, XInclude, and XSLT. My issue with libxml2 is just that it's a bad dependency for a project like Kid that wants to be able to run on cheap web space with bare-bones Python support. There are a lot of hosting providers that aren't going to have libxml2 or the option of compiling from source.

I went with ElementTree because it's simple, pythonic, and fast enough. I also had a feeling we'd be seeing more development around ElementTree, which brings us nicely to why I'm posting.

I completely forgot to mention what is quite possibly the most important event to date in Kid history: the first real application to incorporate Kid templating is Ross Burton's sexy Tate (I said, “sexy Tate,” not “sexy Taint!”). It's an elegant, RDF / XHTML photo gallery that I'm dying to get my hands on. If you're into nice semantic usage of (X)HTML, make sure to peep the page source and check out the correct usage of <ul>’s and whatnot.

Ross has reported a few issues and even mentioned plans for packaging Kid for Debian. He's the proprietor of the excellent Sound Juicer CD ripper for GNOME 2, Debian contributor, and takes some kick-ass photos to boot.

Phillip J. Eby has a recent weblog entry, Python is not Java, wherein he points out a few aspects of Python that are notably different from Java but that are similar enough that Java coders working in Python mistake them for their Java counterparts.

He touches on a few of the characteristics that led me to make Python my general purpose language of choice, a title that had belonged to Java as recently as a year ago. I thought it might be useful to explain these characteristics without assuming the reader has a whole lot of Python experience but that they do have significant Java experience. This would have helped me immensely in determining whether Python was right for me much sooner and I hope it can be of use to someone else.

I'm planning on covering each of the items Phillip points out in his article over the next month and if that works out, I may extend the series to a few other things that may be useful.

This is not a Python is better than Java thing. Instead of bashing aspects of Java's language and library that are obviously valued by the Java community, or insulting the intelligence of Java coders, or using any number of other techniques you might find in Every Language War Ever, Phillip is simply asking Java programmers who may be dabbling in Python to take a closer look at some of the features that make Python unique and attractive instead of attempting to force-fit concepts from Java. If you base your expectations of Python on Java concepts, you are likely going to have a bad experience with Python.

Continue on for the first part of the series, which talks about some of the differences between Python class methods / class variables and Java static methods / static variables.

I was searching for a specific piece of Dive into Python when I ran into this classic from diveintomark.org:

Tom: "This is really good. You could probably make some money off this someday."
Mark: "Maybe, but I'm not going to. I'm giving it away for free."
Tom: "Why would you do that?"
Mark: "Because this is the way I want the world to work."
Tom: "But the world doesn't work that way."
Mark: "Mine does."

I had to pour out a little liquor for my homies.

This is the second post in response to cross-breading ZPT and XSLT. I'd like to dig into how I'd like templating to work in Kid and Leslie opens the door for me:

... maybe this is the sort of thing Ryan's thinking about-- I wonder how hard it would be to hack this into Kid? It would give only a subset of XSLT's capabilities in trade for simplicity, and would only offer the pull approach, but it would give XML-pipelining to a ZPT-ish technology.

Leslie de 0xDECAFBAD talks a bit about cross-breading ZPT and XSLT and mentions Kid along the way. This is the first in a series of response posts.

In Weapons and Coding, I made a prediction:

The first environment [Java / .NET] to successful mesh static and dynamic languages into a coherent platform will win the interpreted byte-code market.

Tim Bray must have come to a similar conclusion because he recently organized a meeting of the minds at Sun to talk about Dynamic Java. Here's a great pic of the BDFL and Larry Wall Tim shot right before they pulled out their katana's to settle the Python vs. Perl debate like gentlemen.

In my mind, Tim is moving into this small classification of people labeled, Hero. His past work on XML (as in, his name is on the Recommendation), recent work on Atom, declaration of The Loyal WS-Opposition, contributions through W3C TAG on the excellent Architecture of the World Wide Web, and now this seemingly unrelated initiative to get Sun to wake up about dynamic languages puts Tim on the right side of almost every major area of innovation I'm interested in. Go Tim, go!

I really hope this leads to some serious discussion on how we can bring static and dynamic languages together into a single cohesive platform. Drop the MFL is better than YFL talk found in Every Language War Ever. We need to start having these types conversations: MFL is a compliment to YFL. This is happening only in very small pockets right now and until today, they were pockets without a lot of potential for real impact.

So this is pretty crazy. I'm messing around with ElementTree (which has been nothing less than perfect) and trying to get it to act like a xml.dom.pulldom/XmlTextReader style pull-parser. But I'd like to be able to assemble a chain of generator producing/consuming functions (or other callable) so that the file can be read, parsed, filtered/mutated, encoded, and written all incrementally.

Check it out:

import sys
import pulltree    # that's what I'm working on :)

def upper_filter(source):
    for (ev, item) in source:
        if ev == pulltree.CHARACTERS:
            item = item.upper()
        yield (ev, item)

reader = pulltree.reader(sys.stdin)
filter = upper_filter(reader)
writer = pulltree.writer(filter, sys.stdout)

for (ev, item) in writer:
    pass

C-z

$ echo "<hello>world</hello>" | python test_filter.py 
<hello>WORLD</hello>

That felt good. More functional than a chain of SAX XMLFilters, almost as efficient, and muuuuch perdier.

Something like this might work someday soon:

import urllib2
from pulltree

XINCLUDE = '{http://www.w3.org/2001/XInclude}include'

def xinclude_filter(source):
    events = iter(source)
    for (event, item) in events:
        if event == pulltree.START_ELEMENT and elm.tag == XINCLUDE:
           href = item.attrib['href']
           for woot in pulltree.reader(urllib2.urlopen(href))
               yield woot
           pulltree.eat(elm, events) # eat events to the end of the element
        yield (ev, elm)

Granted, that's as basic an XInclude processor could be and still be useful but you get the point.

I've been searching for the perfect Python based XML template language. I was happy to find TAL (and specifically, SimpleTAL) a while back but, although neither of us wants to admit it, we've been growing apart for some time now. I spent last week looking for options and, after careful consideration and planning (read: beer and a nap), decided to just build the XML template language I really wanted.

There's at least four billion and nine text based template languages for Python but there aren't a lot of options that fit nicely into the XML tool-chain. Or, if they do fit nicely into the XML tool-chain, they don't fit nicely with Python.

My dreamboat XML template language would combine the pythonicness and simplicity of PTL, the templating features and pipeline-ability of XSLT, and the terseness of Zope's TAL. I'm building it, it's called Kid, and I'm making good progress to be honest.

But I have this overwhelming NIH feeling so I've decided the best thing to do is to run through the current set of tools and take a professional, objective look at why each isn't getting it done for me (i.e. make fun of minor flaws and limitations until I feel better about myself). Herewith, a look at the good and the bad in the Python XML templating space...

I've been working on a weblog/micro-content management system with what I believe are some unique qualities. I've wanted to write about some of the approaches I've taken and how they are (and are not) working out but feel I should provide some kind of context for my ramblings. So I'm going to try to summarize the main aspects of the system real quick so I can start digging in to the more specific stuff.

I've settled on the name "Splice" because I think it has a nice ring to it and is a pretty good one-word description of a major goal of the platform. Nothing has been officially released yet but I have allocated a project on sourceforge.net and plan on bombing the existing code up there within the next week. An initial 0.1 release should follow after about a month or so, assuming I can keep my current pace.

So without further ado, here's a quick breakdown of the planned features of Splice...

What you're getting into:

Mucking with builtins is fun the way huffing dry erase markers is fun. Things are very pretty at first, but eventually the brain cell lossage will more than outweigh that cheap thrill.

Barry Warsaw, 23 Mar 2000

Now, let's muck with some builtins.