Mistakes to avoid when product testing

Testing your product is the most important activity your company will undertake. It validates that there is a market for your product and helps you iterate the product for that market. Here is a breakdown of the most common mistakes when product testing and how best to avoid them.

The mistake of testing without a hypothesis

One of the fundamental mistakes is testing your product with consumers as quickly as possible without a hypothesis – i.e. a grounded assumption of the product’s purpose/goal for a defined customer. Product testing is simply an experiment to test your hypothesis or assumptions. Based on the results of the experiment, the hypothesis is either proved or disproved; without a hypothesis, testing your product is utterly meaningless.

If you do not have a grounded assumption of the product’s purpose, you cannot validate that there is a market for it. If you do not have a defined customer, you cannot qualify or disqualify customers’ feedback. For example, my company is developing a computer game, Cerebros, which is a fast-paced First-Person Shooter, a genre which has a complicated control scheme. If we tested the game on 10 70-year-old customers, they would complain that the game moves too fast and that the control scheme is too complicated. If we did not have clear grounded assumption of Cerebros’ purpose (we cannot validate there is a market for a product that has no clear purpose), and if we did not have a defined customer (we cannot disqualify a 70-year-old customer’s feedback), the conclusion we would have to accept is: we have slow the game down and simplify the control scheme.

If we slowed the game down and simplified the control scheme, the 10 70-year-old customers would be satisfied. However, if we tested this new version of the game on 10 20-year-old customers, they would complain that the game moves too slowly and that the control scheme is too simple. If we had the hypothesis that Cerebros is a fast-paced First-Person Shooter for a young, experienced gamer, we could have immediately disqualified the 10 70-year-old customers’ feedback and not waste development time changing the game for customers fit for the product’s purpose.

The mistake of ignoring bias

The most obvious bias is your own bias for your own product. You’re never going to be perfect and you shouldn’t expect your product to be perfect. You have to accept criticism of your product, and ultimately criticism of you, and understand other people’s viewpoints of your product. It is always hard to separate your identity from your product’s and it does not get easier; in fact, it gets harder as the more time and money you invest in your product, the more your identity is tied to your product and it becomes something like your child.

On the same subject, have you ever criticised a child in front of their parent? If yes, what did you use to treat the black eye? If no, you probably already understand why you don’t criticise a child in front of their parent, because that child is that parent’s pride and joy. Similarly, people think the same of your product, as precious to you as your child is. Your customers will be biased towards not criticising your product when talking to, or in front of, you.

You need to understand this bias and enable the customer to voice their criticism of your product. Essentially, you have to give them permission to criticise the product. This could be as simple as asking them bluntly questions such as:

• “What didn’t you like about the product?”
• “What would you change?”
• “What would make it better?”

Even if people are sincere and they are giving you honest feedback – whether positive or negative – people generally do not know why they act the way they act. There is a bias the customers have of themselves. So, attempting to extract introspection on their processing of their experience with your product should be treated with caution. What people repeatedly DO is a far better indicator of their behaviour and preferences than what people repeatedly SAY. So, if people ask to test your product or try it at an expo, keep coming back to test your product, or ask for more information, that’s a good indicator your product has potential.

Finally, there’s the bias you yourself will have if you are a likeable, charismatic person. I know a game developer who is quite possibly the best showman I’ve seen who wasn’t in show business. At game expos, he set up his game and created a great atmosphere around his exhibition stand that would always draw crowds and excitement. However, he always remarks to me that this enthusiasm he generates at expos (he’s received prestigious awards, etc.) has never transferred to sales and he cannot understand it.

To circle back, as he has never granted me permission to criticise his product, I am always prevented from giving him honest feedback. The problem is the customers are buying him, a very charismatic person, and the event he creates around his exhibition stand, not the product – his game – he’s ultimately selling. So, be careful when product testing: customers should be ultimately making a decision on whether or not to buy your product, not whether or not to buy you.

Key product testing takeaways

For the mistake of testing without a hypothesis, I used an exaggerated example demonstrating the disparity between two groups of customers; it illustrated that having no hypothesis will lead you to change the product, sometimes to its detriment, and waste precious development cycles. It won’t always be that obvious whom you should qualify or disqualify as your target customer, but it significantly helps if you have a grounded assumption of your product’s purpose for a defined customer.

Ignoring bias in tests or opinions is an easy mistake to make, especially if the tests confirm your hypothesis or the opinions coincide with yours. It’s always best to look at the behaviour of people interacting with your product and look at the amount or sample size of people who exhibit this behaviour and their profile.