Reblog

Lean startup thinking and acceptance tests

Listening to this great podcast where Eric Ries speaks about product market fit and validated learning I thought that lean startup thinking can drastically reduce the importance of automated acceptance tests to the bare minimum. Nothing that I had the pleasure to test in person unfortunately, but something to think about for the next project with the right business context.

What’s the deal with acceptances? At the last Agile 2009 I participated at a panel where the audience was required to brainstorm on the need of acceptance testing and the options to replace them some way or the other. The outcome of the session was a general consensus that acceptances are hard to maintain and they should only cover the most important paths of the application and accepting the fact that exploratory or manual testing is still needed to cover the rest of the application. How true. I was burned several times by the huge maintenance cost, cryptic deployment issues and general outcome disappointment even for the best crafted acceptance suite. After that experience I started reducing the acceptance suite size accepting a small manual and not automated testing phase as an unavoidable fact.

Lean startup thinking is a set of principles and values that can be applied to an existing process such as Extreme Programming. There are a few main goals (this is just my modest understanding of the whole thing which is way bigger to summarize here):

• The process should focus on the steps to achieve the product market fit which is the point where the core fundamental business value of the product is appreciated by customers and put to good use. You can measure product market fit with customer retention rate for example.
• The development process should be able to quickly create and execute feature changes based on the customer feedback as part of the iteration loop. The product market fit won’t be achieved If the project is too slow transforming the customer feedback into value. This is especially true for startups where there is always a good amount of uncertainty about the business value of the final product until real customers determine its success or not. Validated learning takes care of carefully collect the customer feedback and put in place the necessary metrics (such as A/B testing percentages) to validate that what we think the customer wants is really what the customer want.
• Since we want customer feedback so badly we have to deploy as soon as we add new value to the product. The immediate feedback cycle loop helps the team deciding what has to be changed to achieve market fit as soon as possible. The team embraces the inevitable presence of defects in the final product but learn how to keep that number low and how to revert and fix the problem keeping the damage as low as possible. Continuous deployment on a cluster connected to business metrics such as the revenue/minute and an automatic halt and revert system is an optimal context to create such a tight loop while minimizing the revenue loss

The last practice is specifically related to acceptance testing. We strategically accept a compromise. The compromise is especially useful in a startup context where the business value of the final product is a moving target. By delegating to a small subset of our customer base the manual testing of the application we can keep the acceptance test suite size at the minimum. Sounds scary uh? But just think for a moment: the project infrastructure is able to revert the system in minutes as soon as some sensible metrics drop (remember also that the latest build went only to a portion of the cluster exposing the new deploy only to a subset of the customer base), stop new commits until the team fixes the issue (in case of real disasters) or just alert the team that the results of the A/B metric from the last deploy was just not as appreciated as expected. Sure you risk to have customers pissed about your product, unless…, unless you demonstrate that you recover in minutes and respond to customer feedback on the next iteration. Lean thinking.