Feedback Junkies

You're Already Using Continuous Flow (poorly)

Many teams out there are doing Scrum with a 30 day sprint. If you ask them, they'll tell you one of the reasons they adopted this process is to be able to quickly respond to change. If you look at the way they work, it's very much in line with that stated goal. The problem, however, it that what they actually do is totally different than what Scrum prescribes.

Lets examine the process of a typical Scrum team doing 30 days sprints:

• On day one of sprint #1, someone introduces a bug. They don't realize it, and it makes it way through the end of the sprint.
• On day one of sprint #2, a stakeholder happens to be playing around with the version of the product that was demoed for sprint #1. She notices the bug, and it's critical, so she adds the issue to the very top of the backlog. However, planning for sprint #2 has already come and gone, so it will have to wait for sprint #3.
• On day one of sprint #3, a developer fixes this high priority bug. The fix is demoed at the end of sprint #3, and the fix is now (at least, theoretically) ready to ship as we begin sprint #4.
  

This is what your certified scrum master will probably tell you is the "correct" approach. Do we actually address critical bugs this way? Of course not, because waiting 90 days from the point where a critical bug is introduced to the point where it's fixed is absolutely insane! No rational team would do that. Here's what really happens on these teams:

• On day one of sprint #1, someone introduces a bug. They don't realize it, and it makes it way through the end of the sprint.
• On day one of sprint #2, a stakeholder happens to be playing around with the version of the product that was demoed for sprint #1. She notices the bug, and it's critical, so she goes to the team lead and asks to have it fixed right away for the PR demo she's doing on Thursday.
• The team lead and the stakeholder argue about process and deadlines for about an hour.
• The team lead then relents, branches the sprint #1 demo code. He works into the night to fix the bug, test it out, deploy the new version, test it out again, and merge the change into the trunk.
  

At Improving Works, we use a continuous flow process for handling changes. As new bugs and features are pulled from the backlog, they are completely defined, implemented (using TDD) and deployed one at a time. That is, each completed feature or major bug fix results in a new release of the product. As a result, we release often. Very often. Infinitest, for example, is released an average of every 2.1 calendar days. These releases go directly to customers, because until we put those changes in the hands of our customers, they might as well not exist.

What is happening in the second Scrum scenario we outlined above is not all that different, from a process perspective, from what we do at Improving Works. We're both fixing critical bugs as soon as we possibly can. The difference is that at IW, the technical practices we employ give us confidence that our frequent releases will be stable, valuable, and work as expected. Ask the technical lead that hacked out a fix at 2 in the morning whether or not he feels the same confidence, and I suspect you'll get a very different answer.

Pretending that you don't have to respond to a crisis is irresponsible and unrealistic. The Agile value of "Responding to Change" demands that we rigorously employ the technical practices that allow us to react calmly and effectively when things go wrong.

TDD Survey Results

The results of our first annual TDD and Automated Testing Survey are in, and the results are surprising.

First, some interesting data:

• 49% said they ran a local build 10 times a day or more. 
• 50% said their local build failed 2 times a day or more.
• 23% said their build takes 5 minutes or more to run.
• 54% said they break the build on their CI server at least once per week
• 41% said they work on projects with test coverage greater than 75%

One of the things I was looking for in this survey is some data that reflects the cost of a failing build. We don't want our builds to always pass. You don't get any feedback from an indicator that always reports the same thing! But we also don't want them to fail too often because each failure has a cost. Whether it's having to fix the problem and re-run the test, or blocking the team while you fix the build on the CI server, there's a balance between the feedback we get and the cost we pay for failure.

One way to think about this is the cost you pay for getting the feedback is a hedge against finding out something is broken later. For example, I'll gladly take a minute to be 99% sure that my 30 minute deployment process is going to work. I had suspected that the typical failure rate for a local build is around 10%. However, the data seems to point to a failure rate that is closer to 20% (maybe more!).

If this is true, then it's very likely that people are paying way too much when running local builds. That is, they need to adopt practices like Continuous Testing, that help them fail faster. I'd be willing to be that there's also an inverse relationship between build length and failure rate. I think faster builds would lead to smaller steps, which naturally decrease failure rates. I'm looking forward to following up with more surveys to prove this theory.

You can get the raw data from the survey here.