Many product managers depend on evidence or analytic data to make winning product decisions, how many times does the improvement go live and fail to have the desired impact? Why is it that great product managers consistently deliver improvements that move the needle?

I recently was reminded of a anecdote where a maths teacher plays a simple game with their class. The teacher sets out a challenge, he has a rule which results in a sequence of numbers "2, 4, 6". The pupils can make as many guesses about the next numbers and the teacher will answer with "follows the rule" or "breaks the rule". The pupils can only have a guess the rule once.

This feels very similar to product managers using analytic evidence to validate their assumptions about a new enhancement.

The first pupil guesses 8 and the teacher replies "follows the rule", the pupil then guesses at 10, the teacher again replies "follows the rule". Feeling confident, the pupil guesses the rules as being "add 2 to the previous number". The teacher says "No that is not the rule."

The pupil obtained evidence that showed his assumption as being correct. Pleased with positive results the pupil made the common mistake, they failed to validate their assumption by attempting to break it. We typically look to prove we are right, instead of proving we are wrong.

A different pupil, unaware of the incorrect answer, also had a hunch the rule was add 2 to the previous number, they also guessed 8 and then 10. However, instead of guessing the rule, this pupil then guessed 13, the teacher responded "follows the rule". By attempting to break his rule the pupil discovered his assumption was wrong - 10 add 2 is not 13.

He had another idea, so guessed 10 (so the sequence is 2,4,6,8,10,13,10), the teacher replied "breaks the rule". The pupil then guessed various numbers lower than 13 and then numbers bigger than 100 getting larger and larger. The pupil then offered a solution - "The next number has to be bigger than the previous number", the teacher congratulated the pupil on getting the correct answer.

When product management validates assumptions there is often an eagerness to find evidence that the assumption is correct. This results in the common mistake of product enhancements being invested in which will deliver no additional value or return. 

Great product managers validate their assumptions by looking for evidence to show the assumption is wrong. This approach results in more enhancements being developed that increase value. It also encourages a culture surrounding product that failing quickly is good and improves our understanding. 

AuthorDave Martin