Wednesday, October 31, 2007

Impending QCon

I'm speaking at QCon next week in San Francisco. Martin Fowler and I are pairing on an all day tutorial on Building Domain Specific Languages, a topic that he and I have much much interest. The tutorial takes an interesting format. The first part of the day will be a normal lecture/Q&A kind of affair. However, at some point in the afternoon, we're going to switch into workshop mode and actually build some DSLs based on suggestions from the audience. I think this is a great way to get people to not only understand what DSL's look like but get a bit dirty in designing some from scratch, working through the inevitable headaches that come up.

The other talk I'm giving is a "regular" presentation on Thursday about (you guessed it) Building DSLs. Looking forward to it.

Sunday, October 28, 2007

What High Coupling hath Wrought

A long time ago, I wrote a blog entry about the high price of coupling as it pertains to Internet Explorer and Windows. In it, I lambast Microsoft for the business decision to tie Internet Explorer so tightly into the underlying operating system. Developers all know that highly coupled systems are bad. And here's a stellar example.

Microsoft is finally going to release a "headless" version (called "server core") of their premiere server operating system (currently called Longhorn Server). Coincidental to this, they have built a new batch language for Windows called Windows Power Shell (nee Monad), which is simply brilliant. They have raised the bar on what a shell language can be. I can't heap enough praise on the coolness of WPS. What a great thing to have just as they are going to release a headless version of the OS.

Only one small problem. They want to make the footprint of the headless version of Windows Server smaller than the footprint of Vista (currently about 8 Gb, and that's just for the OS). The problem is that the headless version won't include .NET 2, because the .NET framework and libraries are coupled to just about every nook and cranny of Windows. If you include .NET 2, you pretty much include all of Windows. And there's the rub: Windows Power Shell is written on top of .NET 2. Which means that this brilliant tool for managing tasks, services, and general scripting of the operating system won't be available for the one system that could benefit from it the most.

I leave it to the reader to form their own conclusions.

Thursday, October 25, 2007

Developer Productivity Mean vs. Median


10 .
9 . .
8 . . . .
7 . . . . . .
6 . . . . . . . .
5 . . . . . . . . . .
4 . . . . . . . . . . . .
3 . . . . . . . . . . . . . .
2 . . . . . . . . . . . . . . . .
1 . . . . . . . . . . . . . . . . . .
0 . . . . . . . . . . . . . . . . . . . .
___________________+_+___________________
1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1 1 1 1 2
0 1 2 3 4 5 6 7 8 9 0



OK, first a little remedial math: mean is defined as the mathematical average of a series of numbers. For the set of numbers above, the mean = 4.55. The median is defined as is the number defined as lying at the midpoint of a sorted number series, which is 5.5 in this example.

Why in the world am I talking about this? Well, it pertains to developer skills. You see, really good developers are much more productive than average ones (this has been documented in several places, including the Mythical Man Month and Joel). In fact, some statistics say that really good developers are orders of magnitude better than poor ones.

Consider this version of the graph:

10 .
9 . .
8 . .
7 . .
6 . . .
5 . . .
4 . . .
3 . . . .
2 . . . . . .
1 . . . . . . . . . . . . .
0 . . . . . . . . . . . . . . . . .
___________________+_+___________________
5 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 1
0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0
0


For this one, the mean = 2.7 and the median = 2. If the height of the graph line represents programmer productivity, the second one is closer to reality.

So what does this mean to development? Managers used to think that this is a nice straight 45 degree angle, and that even if you can't get the best and brightest, you still get usefulness out of mediocre developers. But building software isn't like digging a ditch. Even poor ditch diggers can make a hole. In software, what you (and others) write today becomes the foundation for tomorrow. If you have bad developers building your foundation, the really good developers have to go back and fix it before they can build more useful stuff. Hiring mediocre developers (or just average ones) slows project velocity. Couple this with the fact that adding people to a late project makes it later, and you can understand why most enterprise development moves at a glacial pace. Building software in the large with lots of average developers emphasizes two negative project metrics: trying to scale by adding people and that average developers produce code on the average scale. But the truth is more like the lower graph: overall productivity is dragged down to the median level, not the mean level.

An entire industry has been built to solve this problem. It was thought that you can fix this problem in two ways. First, build restrictive languages that keep bad developers out of trouble. But, Glenn Vanderburg has a dead-on quote that reflects the reality that he (and I) have seen on projects: Bad developers will move heaven and earth to do the wrong thing. Command and control computer languages aren't like the governor on a car, forcing you to drive more slowly and thus more safely. At the end of the day, weak languages slow down your best developers and don't prevent the bad and mediocre ones from writing horrific code.

The other attempt at fixing this was to take restrictive languages and build really smart tools to help developers generate code faster. This is the classic implicit sales pitch for Visual Basic: no one can hurt themselves, and you can hire cheaper developers to write your code, without paying a good salary to those annoying software craftsmen. It is widely known that Microsoft internally calls mediocre developers "Morts" and targets them with some of their tools. Of course, this doesn't apply to all Visual Basic developers and not to good developers in the .NET space. But it is the reality that this dual strategy is the way vendors have sold tools for the last couple of decades.

And now we see that it doesn't work. Part of the ascension of dynamic languages is the realization that the experiment of command and control languages don't make average developer's code any better, and it hampers your really good developers. Why not figure out a way to unleash the good developers and let them go as fast as they can? Ruby on Rails is a good example of using domain specific languages to create a simpler language on top of Ruby for things like ActiveRecord relationships. For everyday coding, you can stay at the high, simple abstraction layer. If you really need it, you can dip below that abstraction level and get real work done in Ruby. For many of the Ruby projects on which I've been involved, using open classes is done strategically, accounting for between 1.5% and 4% of the total lines of code. Yet that surgical use of one of the kinds of meta-programming that Ruby allows eliminates hundreds of lines of code.

So what does all this mean? First, give good developers powerful tools and you'll get higher quality software faster. Second, as an industry we would become more productive if 30% of the existing developers were laid off tomorrow. Having warm bodies doesn't help projects, and having to baby-sit poor developers cuts the productivity of your good developers. Software is too complex to turn into an assembly line manufacturing process, at least for now. Good software developers are still craftsmen, no matter how much that annoys companies that wish that wasn't true.

Sunday, October 21, 2007

Ruby Matters: Unleash the Brainiacs!

ThoughtWorks was early in the game in the Java world. Lots of the standard parts of the Java stack (like CruiseControl, Selenium, etc.) were built by clever ThoughtWorkers from project work (there is a long list here). Working with smart, passionate people who have damn good ideas, coupled with the corporate culture that nurtures open source, is one of the advantages of working at ThoughtWorks.

Fast forward a decade, and the same thing is playing out again. Lots of ThoughtWorkers really like Ruby. And one of the things that I suspected early on was that the combination of ThoughtWorkers and a super powerful language would be a good mix. And I was right.

As we've been doing Ruby projects, ThoughtWorkers have been generating great ideas like mad. Because Ruby is such a rich language, really smart people build really cool things. And, of course, ThoughtWorks likes to open source stuff, which is a good combination. The Ruby part of this equation means that you can build stuff that, while sometimes possible in other languages, is too much trouble, so you just don't do it. Imagine that you were traveling to Chicago a decade ago, and you wondered about the weather forecast. How would you find it? You could call the hotel and trust that someone there knows it. Or you could watch the weather channel and wait for the Chicago forecast to come on. Or you could go to the library and find a Chicago newspaper and look there. All of these were so much trouble that you were likely just to skip it and guess. Now, you can find out in about 10 seconds by using the Internet. Convenience opens possibilities.

Here is a perfect example. Jay Fields, Dan Manges, and David Vollbracht has released a Ruby gem called dust, which makes it easy to do write better test names (and other stuff). Or what about deeptest, which allows you to run unit tests in parallel, written by Manges, Vollbracht, and anonymous z. Or what about Mixology. This is a gem created by Pat Farley, anonymous z, Dan Manges, and Clint Bishop in the Something Nimble crowd that allows you to dynamically change mixins. If you aren't sure what that means, think about the ability to change what interfaces a class implements at runtime, based on some criteria. Not just add new ones, but remove them as well. While not something you do everyday, it does come up and there are good uses for this capability.

As ThoughtWorkers are building Ruby projects, we're helping spawn lots of cool infrastructure (like CruiseControl.rb). Of course, ThoughtWorkers aren't the only ones. But you get a bunch of smart people together combined with a powerful language and good stuff happens.

Check out Something Nimble, a blog maintained by ThoughtWorkers working on Ruby projects, spawning cool stuff left and right.

Wednesday, October 17, 2007

Mocking JRuby

Testing is one of the easy ways to sneak JRuby into your organization because it is easier to write tests in dynamic languages (especially mock object tests) and it isn't code that is deployed, so it eases the minds of the furniture police somewhat. Here is an example, adapted from Martin Fowler's essay on mock object testing.

In Martin's example, he creates Order and Warehouse classes (and a Warehouse interface), then writes tests with them in jMock. Here is one of those tests, rewritten in JRuby, using the Mocha mocking library:

require 'test/unit'
require 'rubygems'
require 'mocha'

require "java"
require "Warehouse.jar"
['OrderImpl', 'Order', 'Warehouse', 'WarehouseImpl'].each { |f|
eval "#{f} = com.nealford.conf.jmock.warehouse.#{f}"
}


class OrderInteractionTest < Test::Unit::TestCase
TALISKER = "Talisker"

def test_filling_removes_inventory_if_in_stock
order = OrderImpl.new(TALISKER, 50)
warehouse = Warehouse.new
warehouse.stubs(:hasInventory).with(TALISKER, 50).returns(true)
warehouse.stubs(:remove).with(TALISKER, 50)

order.fill(warehouse)
assert order.is_filled
end

def test_filling_does_not_remove_if_not_enough_in_stock
order = OrderImpl.new(TALISKER, 51)
warehouse = Warehouse.new
warehouse.stubs(:hasInventory).returns(false)
order.fill(warehouse)

assert !order.is_filled
end

end

The order object is a pure Java object, tested with a mocked out version of the Warehouse interface. Now, this isn't exactly what Martin wrote in in excellent essay about mocks vs. stubs, but the intent is preserved in this example.

Most of the really crufty stuff in mock objects in Java is getting the compiler to agree with you that the mock object type is what you want it to be, and of course that problem goes away in JRuby. Also notice that I can create the mock object directly off the interface. Because JRuby creates proxy objects for the Java classes, you can create a mock from the interface directly by calling the new method on it. The other cool thing about this example is the require "Warehouse.jar" line at the top. JRuby allows you to require a JAR file, which gives you access to the Java classes within it.

Saturday, October 13, 2007

Ruby Matters: Contracts vs. Promises

In teaching Ruby to Java developers, two things seem to shock and annoy developers: open classes and the lack of interfaces. Because Java developers are inundated with interfaces, it's hard to imagine a language whose libraries are strictly defined in this manner.

Interfaces in Java and .NET are exactly like legal contracts (where the compiler is the all-powerful judge). If you implement an interface yet don't meet the terms of the contract, the judge refuses to allow you to proceed.

Mixins in Ruby are more like promises than contracts. When you agree to pick up a friend at the airport, you don't sign a contract, you promise you'll be there. The punishment is purely social: if you break a promise, your friends won't do stuff for you. Here's an example.

In Ruby, a common Mixin is Comparable, which gives you the same behaviors as the Comparator interface in Java. The promise in Comparable is that you will implement the "spaceship" operator (<=>), which returns negative if the left-hand side is less than the right, 0 if they are equal, and positive if the LHS is greater than the RHS (which is exactly like the similar compare() method in Java).

class Employee
include Comparable

attr_accessor :name, :salary

def initialize(name, salary)
@name, @salary = name, salary
end

def <=>(other)
name <=> other.name
end
end

list = Array.new
list << Employee.new("Monty", 10000)
list << Employee.new("Homer", 50000)
list << Employee.new("Bart", 5000)

list.sort!
# Monty vs. Homer
list[0] < list[1] # => true

# Homer vs. Monty
list[0] > list[1] # => false

# Homer is between Bart and Monty?
list[1].between?(list[0], list[2]) # => true

If you violate the promise by mixing in Comparable and yet don't implement the spaceship, nothing bad happens...until you try to ask your friend Comparable to perform a service for you (like compare two things with between? or call the sort! method). Then, you get an error message complaining "undefined method `<=>' for Employee". And this happens at runtime, not at compile time.

This is a fundamentally different mindset than the legalistic strongly contracted languages. It demonstrates one of the reasons that you can't slack off on writing your unit tests in Ruby: you don't find out important things until runtime. It is certainly a different way of thinking about implementing APIs. While lots of Java developers think this leads to chaos in large development teams, I have not found that to be true (and I think that most Ruby developers would agree with my assessment). The trade-off is strict control vs. flexibility. If there is one thing that my experience has taught, it is that flexibility is the most important thing for strong programmers, echoed by many of Paul Graham's essays.

If you must have stronger contracts in Ruby, there is a framework called Handshake that allows you to create pre- and post-conditional invariants ala Eiffel. While I understand the appeal of Handshake, I would be loathe to use it on a project because I prefer the power of flexibility rather than the strictures of constraints.

Monday, October 08, 2007

Ruby Matters: Language Beauty Part 2


A dead chrysanthemum
and yet - isn't there still something
remaining in it?
Takahama, Kyoshi

Let me not to the marriage of true minds
Admit impediments. Love is not love
Which alters when it alteration finds,
Or bends with the remover to remove:
O no! it is an ever-fixed mark
That looks on tempests and is never shaken;
It is the star to every wandering bark,
Whose worth's unknown, although his height be taken.
Love's not Time's fool, though rosy lips and cheeks
Within his bending sickle's compass come:
Love alters not with his brief hours and weeks,
But bears it out even to the edge of doom.
If this be error and upon me proved,
I never writ, nor no man ever loved.
William Shakespeare

Can computer languages be beautiful? Is it possible to create a beautiful COBOL application? I think it is possible, but you can't compare different languages as if they all had the same constraints. For me, these comparisons are like comparing poetry styles. Within a particular poetry style, you can clearly have beauty. But some poetry styles have serious constraints. For example, a Haiku has exactly seventeen syllables, in three lines of five, seven, and five. A sonnet is a poem with fourteen lines, using any formal rhyming scheme, having 10 syllables per line. Shakespeare wrote some stunning poetry within these pretty serious restrictions!

Computer languages are closer to poetry than prose. The syntax of the language imposes order, just as the conventions of different poetry styles constrain what you can create. Within those constrains, though, you can create wonderful things. But the strictures of the poetry style necessarily prevent you from creating more ambitious works. William Faulkner famously said that he wanted to be a poet and failed, then he wanted to be a short story writer and failed, and eventually became a novelist because he couldn't say what he wanted to say within the boundaries of the other styles.

A lot of developers who find themselves "at home" in a particular language can relate to this. For many developers, Java (the sonnet) feels just right. Yes, there are restrictions, but they are well known and seem right. Other developers chafe at the imposed limitations and seek more expressive media, like Perl or Ruby. And if it was just about aesthetics, we could all live happily, each developer writing code in whatever medium he or she liked.

But with computer languages, expressiveness == power. Being able to say what you mean succinctly and effectively matters. The only reader who you must satisfy completely is the execution environment. Thus, when making a language choice, it makes sense to pick the most powerful language you can stand. Of course, different developers rate power differently. For me, meta-programming is extremely valuable. It's one of those features that, once you've used a language that supports it extensively, it's hard to go back to a weaker one.

Which brings me around to Paul Graham again. When I first read Hackers and Painters, I remember thinking that about 80% of what he said was dead on, and the other 20% was either just crazy or intentionally inflammatory. Over the last few years, though, as I've learned more about languages myself, I go back to his writings (most of which are online at paulgraham.com) and realize that he's exactly right about something I used to disagree with him about. Now, I'm down to thinking only 5% is crazy or inflammatory, and that number is shrinking all the time. Anyway, one of those things was his language power scale, which I discounted at first but now believe to be an outstanding yardstick by which to measure language power.

For me, Ruby has a great combination of succinct expressiveness and super powerful meta-programming support. I think that it is currently the most powerful language you can get paid to write. Until Smalltalk makes a come back or Lisp eventually takes over the world, it's the best game in town. But that doesn't mean that I can't still write sonnets in Java and C# from time to time, when the circumstances warrant.

Thursday, October 04, 2007

Ruby Matters: Language Beauty Part 1

Beauty without expression is boring.

Ralph Waldo Emerson


I was talking to a colleague (named Vivek) recently who is originally from India. He was telling me about "Hin-glish", the merging of English an Hindi. I am already familiar with "Singlish", the flavor of English tinged with Malay, Hindi, Chinese, and several other languages spoken in Singapore. Vivek was telling me that it is quite common when talking to someone else who knows both English and Hindi to freely flow back and forth between the languages. That got me to thinking about expressiveness in language. We do this all the time in English without specific foreign influences: we use phrases like "Joie de Vivre" and "Cul de sac" all the time. Expressions in other languages just make more sense; some languages have found a useful, concise label for common human experiences and other languages don't bother creating their own term for it. Phrases like this encompass much more than the literal translation. They manage to encapsulate tons of connotation, not just the denotation of the meaning of the words. (For a supreme distinction between connotation and denotation (and none too shabby language either), check out this Robert Graves poem The Naked and the Nude).

Expressiveness of language makes a big difference. Having concise, nuanced representations of common concepts means less context establishment. This is one of the compelling arguments for domain specific languages: if you encapsulate high-level meaning in code, you spend less time talking about it. This is the vocabulary that describes your business, and everyone talks the same language so that you don't have to start from scratch every time.

Talking about beauty in programming languages sounds silly, because they are after all just ways to drive Turing machines. But people still talk about it. O'Reilly has a popular recent book named "Beautiful Code", which has essays by famous programmers. Reading this book clearly indicates that this opinion is purely subjective. My friend Brian made a great observation about this book: the essays written about other people's code really do illustrate beautiful code, while the ones written by developers about their own code are mostly about cool hacks. Most of those examples I would not consider beautiful by any measure, even if clearly highly functional.

Glenn Vanderburg has a great talk entitled "The Beauty of Ruby", discussing the elegant language design entailed in Ruby. For example, it is necessary in object-oriented languages to distinguish member variables from local variables. Java does this either via scope (which can be ambiguous) or the this keyword. Python does this with self (which is required in some places). Ruby uses the @ symbol. It is unobtrusive, unambiguous, and small. And it conveys exactly the meaning you need it to convey without being too chatty. Similarly, class-level artifacts in Ruby are identified with two @ symbols.

For me, Ruby does in fact have beautiful syntax. It is easy to write, easy to read, just succinct enough to keep you from typing too much but not so succinct to become cryptic. Many Perl developers think that Perl is beautiful (and, Perl was after all developed by a linguist). But I think it goes too far, bordering on cryptic (and I'm not the only one: many people refer to Perl as a "write only" language because it is so hard to read later).

Interestingly enough, even when I was primarily using Java, I never considered it beautiful. Utilitarian, but lacking beauty. But most of the post-Java developers I know still talk about the beauty and elegance of Ruby. I've never heard those phrases so much about other languages.

Language beauty will always be subjective. I explore this idea further in part two of this topic, coming up soon.