I’ve been reading the Lean Startup by Eric Ries and he has introduced me to the concept of Validated Learning. Validated Learning is the practise of effectively measuring the accuracy of assumptions and using the results of the Validation to understand whether the assumption was correct and if so, continue onto the next test. If not, decide whether your strategy, assumption or feature needs to be improved or to change direction.
The main thing that I found interesting was an example of how this could be integrated with Kanban, a practise that I am an avid user of since launching Mousebreaker.com in 2008.
The example spoke of the founder of Grockit, an online teaching site that enables students to learn either socially with other students online or individually. They assumed that a user story was not actually delivered until it had been confirmed as a success through measurement of Validated Learning.
In addition, the feature is aligned to the product’s strategy and therefore had a set of assumptions associated with it. The process of Validated Learning either proves or denies the assumptions.
If it is found that a feature has not been a success and actually improved the product, then that product is removed. In addition, the feature is aligned to the product’s strategy and therefore had a set of assumptions associated with it. The process of Validated Learning either proves or denies the assumptions.
In the example, Grockit uses A/B Testing and cohort analysis to validate the success of the feature being delivered by the development team.
A/B testing allows you to test different versions of pages or parts of pages to different proportions of your users to test a metric such as registrations or visits to sections of your site – although you can measure much more than this. You can then determine which version works best and then make the winner available for all users.
Cohort analysis is very similar but takes place a step before the A/B Testing. You may use a number of different marketing channels using a number of different messages, for example. Each user that comes from one type of channel/message would be associated with that channel/message and then tracked throughout their lifetime on your product. This would allow you to understand the cohort group’s lifetime value for example, so that you can understand the value of those messages or that channel.
You can use each of these methods exclusively or together and measure against the metrics that matter to your business. What you are looking for is actionable data, something that up you can make a decision on or understand the value of a change.
I really like this concept; however it does seem to go against the basic principles of Kanban. Kanban is an agile methodology that is used, partially, to eliminate waste during the build process of a product through practises such as limiting work in progress and kaizen – the constant improvement of the process.
My question is whether removing a feature that is not deemed a success as wastage; and if it is, could that have been avoided?
Whenever I have built a web site and we have launched a new feature, we have measured and iterated based on the data collected until it has worked – a form of validated learning. Only, of a feature has not worked, I have seldom, if ever, removed this feature. My main thought on this is that it would have been a waste of the time to develop the user story.
Perhaps I would have removed the feature if it had a detrimental effect on the metrics used to validate the functionality.
I would be happy to do this as long as it informed decision making in the future. This is what Eric Ries is arguing. A user story or feature that is being developed is actually a test of a theory or an assumption. You are assuming that this new feature will improve the product and will improve a core measurement to your business.
If it does not do this, why keep this feature live? If it does not deliver enough improvement, then why keep it live? As long as these lessons learned helps to change the strategy or assumptions around your business, you should be able to reduce wastage as your strategy should be directly influenced by validated learning through the use of your product.
I really like this concept and will keep this in mind when I next get the opportunity. I will definitely be posting about it once I have some personal evidence of this in practice.
Please comment if you have used Validated Learning in practise and how you have found it in the comments below.