Pivotal Blabs

This is suddenly more relevant after Adam's excellent post on Aristotle and software (and while you're at it don't miss There Is No Agile). The following is just an application of virtues to writing...and in my opinion, very relevant to the practice of making software.

Following from the assertion that you can apply writing advice to code...

Zen and the Art of Motorcycle Maintenance (book, wikipedia) is an exploration of Quality (wikipedia) (not in the sense of QA or building cars). This is a long passage from Chapter 17 - Phaedrus is teaching a writing course at a local college - one of my favorite parts of the book, it's a model for examination that I keep coming back to. My comments follow at the bottom.

"I think there is such a thing as Quality, but that as soon as you try to define it, something goes haywire. You can't do it."

(continued after the jump)

On occasion, someone will ask me what I do or, more commonly, ask me what Pivotal does. The title on my business card says "Agile Developer," which nearly inevitably leads to one of a few reactions, almost all of which boil down to this:

"What exactly does Agile mean?"

You know what? It's a good question. And, after thinking about Agile for several years now, here's my answer: Agile originally was, and still is, a marketing term.

Allow me to explain.

With the recent holidays, along with a hint of flu season, things have slowed down a bit, and I've finally had some free time to go back and watch a few of the talks I missed at RubyConf this year (recordings are available here). Notably, I also took the opportunity to re-watch Jonathan Dahl's talk on Aristotle and the Art of Software Development. For anyone who missed it, I highly recommend you check it out. It was, in my opinion, one of the best talks of the conference.

You may be wondering what philosophy has to do with writing software. Watch the video. Listen to what Jonathan has to say. I believe fairly strongly that questions of how we think, how we communicate, how we interact with one another, etc. will become increasingly, and overshadowingly, important as the software industry continues to grope its way toward some sense of what it is we're all doing, and what it means to do it well.

We released the first version of the Pivotal Tracker API a couple of months ago, and already there is a growing list of useful 3rd party tools out there:

Pivotal Tracker for Ruby

This is a ruby wrapper for the API, written by our friends at Mobile Commons.

http://mcommons.com/developers/pivotal-tracker-for-ruby

Email Integration

Also from Mobile Commons, this tool allows your to automatically create Tracker stories from inbound emails.

http://mcommons.com/developers/pivotal-tracker-email-integration

GitHub Post Receive Hook

This is web service that you run on your own server, written by Chris Bailey. It automatically updates stories in your project(s) based on GitHub commit comments:

http://codeintensity.blogspot.com/2008/12/github-post-receive-hook-for-pivotal.html

Pickler

Synchronize stories in Pivotal Tracker with Cucumber features. If you haven't heard of it yet, Cucumber is a really cool app that can execute plain-text documents as automated functional tests.

http://www.github.com/tpope/pickler/tree/master

More Tools

There are more, most of them live in GitHub:

http://www.github.com/search?q=pivotal+tracker

If you have written something you'd like to share, or know of other useful tools, let us know! Also, we're planning on making the API better, and would love your feedback.

Interesting Things

• Ruby Hash is really, really, really fast

If you're building a data structure and you need it to be perfomant, Ruby Hash comes highly recommended from Steve Conover. If you're doing a dance and you need it to be awesome, I highly recommend the Robot. Or maybe the Cabbage Patch.

• Counter cache, fixtures, and invalid data

Invalid counter cache data can cause unexpected behavior. For example: size() returning a bad count or associations asserting they're empty when they aren't. In this case, the invalid counter cache data was caused by bad or missing fixture values, a situation that was not caught out by the debugger. With this in mind, it may be useful to resort to puts/p statements if you suspect the counter cache is the source of the problem.

• Count or Size methods may return incorrect values from associations or named scopes using GROUP BYs

When calling 'count', or 'size' on an association, Rails replaces the select of the actual query with a COUNT(*), and strips GROUP BY statements. This can cause the returned count to differ from the actual number of records. A simple (and expensive) workaround is to use .length, which will force the association to be loaded and then return its count. A better method is to pass a :select value to count which selects a COUNT(DISTINCT(foo)) where foo is the column you are grouping by. It is worth nothing that COUNTing DISTINCT records is much less of a performance hit then actually returning their values, so the resulting query is faster than you might expect.

• first and last on has_many associations

This has been previously mentioned in this space, but as we're on the topic of unexpected ActiveRecord behaviors, it's worth reiterating. If you have model Foo, which has many Bars, calling foo.bar.first will always go to the database. This means, for example, that the following statements will not have the expected result:

foo.bar.first.some_value = 'baz' foo.bar.first.save

You would normally expect this to set some_value on foo.bar.first to 'baz' and then save it, but the foo.bar.first object that has some_value is blown away by the foo.bar.first.save statement, which again retrieves the first object from the database (and then saves it). last behaves in a similar manner. A workaround is to always load the results of first or last into an variable and then work with it. In other words:

my_foo = foo.bar.first my_foo.some_value = 'baz' my_foo.save

For a much more thorough treatment of this subject, please see Frederick Cheung's post First, foremost, and [0].

• ActiveRecord::BaseWithoutTable

ActiveRecord::BaseWithoutTable is very handy for when you want ActiveRecord validatioons on a model that does not have a corresponding table (for example, a feedback form).

Edward pointed out the great article by Kevin Matheny, featured in BusinessWeek, on Agile, and our experience on BestBuy Remix.

I'd like to highlight this passage:

Trust is tied closely to how you deal with change. Often, extending trust is hard for businesspeople working on technology projects, because we don’t know how to do the work. We often look to the documentation — requirements, design specifications, and the like — to give us the feeling of control over the outcome. Don’t bother. If you can’t trust your team to deliver, you have the wrong team. Find people you can trust, and then let them do the work. Talk every day, and make sure that the development team has direct access to someone who will be using the product every day after release. For Remix, we’ve never had a formal project plan, never had a requirements-gathering session, never created a requirements document. We chose the right partners, told them what we needed, and got to work. We have control over the outcomes, but we’re not worried about trying to control the details of how we get there.

Of course without trust, any project - "agile" or not - is at risk. But practices typically associated with agile let you go further with trust: everyone is in close communication*, and all levels of the project - from test-driven code written by developers, to regular demos to the client of the latest features - are oriented around fast feedback.

In other words, you the customer trust us in part because what we're doing is visible and tangible to you. If we're going in a direction you didn't intend, or what the team planned a few weeks ago just doesn't seem relevant anymore, we all talk and we correct course.

* for close communication, see Pivotal Tracker

Kevin Matheny, Senior E-Biz Architect at Best Buy, has an excellent article today on BusinessWeek.com about Best Buy's take on Agile software development and Best Buy's experiences as a client of Pivotal Labs. As he mentions in the article, Pivotal Labs has been helping Best Buy build "Remix", an API for the BestBuy.com product catalog. Kevin describes the agile methods that Pivotal Labs uses and how they've helped with what he calls "Corporate Agility", which he describes as "working components instead of complete solutions, expecting and responding to change instead of trying to eliminate it, and trust rather than control." He also describes how Pivotal Tracker fits into Pivotal's agile process:

For example, I recently added a story to the tracker for Remix that read simply "flag products as new if their start display date is less than 30 days in the past." That's all the up-front documentation needed for Pivotal Labs, a development company that specializes in agile software development, to code that function into Remix. Any additional information can be gathered in the daily 15-minute team meetings or in a longer follow-up if more time is required.

Thanks for the mention, Kevin, and we're very glad that the project is proving to be successful. Pivot Steve Conover is at the helm.

net/http is slow. (and so are libraries that depend on it, like open-uri)

Performance Disclaimer: this ought to matter in your app, measurably, before you do anything about it. If you profile and ruby-prof is showing a bunch of classes like BufferedRead and Timeout at the top of the list, your app qualifies. And in addition if you know that your app is dependent on data transfer over http (let's say you're interacting with a Solr server, and you're storing sizable documents in Solr), you should be aware of the problem.

Otherwise net/http or open-uri might be just fine for you.

The problems with net/http, and benchmarks of ruby http client lbraries are nicely written about in An analysis of Ruby 1.8.x HTTP client performance.

Some good alternatives:

• curb (libcurl wrapper)
• rfuzz
• httpclient / http-access2

Our findings matched the article referenced above - the alternatives have pros and cons but each was at least 10x faster than net/http for transfers of 50-300k response bodies.

The fastest solution we found was curb, reusing the Curl::Easy object:

require "curb"

curl = Curl::Easy.new

2.times do
    curl.url = "http://www.pivotaltracker.com"
    curl.perform
    puts curl.body_str
end

Interesting Things

• Net::HTTP is slow

Recommended alternatives: Curb, httpclient, rfuzz.

Please see Steve Conover's writeup for more details.

• Marshall Dump/Load encoding issues foiled by Base 64.

When storing to a string via marshal_dump, it can be handy to encode to Base 64 first. The same string -- once marshal-loaded and Base 64 decoded -- should be free of any encoding and/or escaping errors. Is there anything Base 64 can't do?*

* the answer I am looking for here is "win a knife fight against tigers."

How many times has this happened to you? You get a cool design for your website, and you spend a bunch of time lining up all of your images and roundy-corner widgets and input boxes just... so..., and everything looks great. But, then you submit a form without typing in your favorite ice cream (a required field, of course), and suddenly your layout is splattered about like an extra large scoop of rocky road in the hands of a two year old. It's enough to make you want to stab your eyes out with a hedge trimmer.