The Most Common Experiment Pitfalls and How to Avoid Them

The best plans for experimentation don’t always come through. We’ve learned this in working with teams to design, run, and analyze experiments over the years. Part of learning this process is becoming more proficient at quickly running experiments while making progress through data-driven insights. Below is a list of the most common experiment pitfalls we’ve seen in the field. Hopefully, by identifying these pitfalls early on in your testing process you will be able to avoid these mistakes. 

Time Trap 

Not dedicating enough time. 

This is often the strongest pain point we hear from innovation teams. Teams that don’t put in enough time to test business ideas won’t get great results. Too often, teams underestimate what it takes to conduct multiple experiments and test ideas well. 

Things you can do: 

• Set a regular cadence every week to test, learn, and adapt. Use Team Ceremonies (in Testing Business Ideas) to help you structure pace of your meetings and define the objective. 
• Set weekly goals in regard to what you’d like to learn about your hypotheses
• Visualize your work, using the Team Alignment Map (High Impact Tools for Teams) so that it becomes clear when tasks are stalled or blocked. 

Outsource Testing 

When you outsource what you should be doing and learning yourself. 

Outsourcing testing is rarely a wise idea. Testing is about rapid iterations between testing, capturing insights, and adapting your business idea accordingly. An agency can’t make those rapid decisions for you and you risk wasting time and energy by outsourcing. Insight by definition is “the capacity to gain an accurate and deep understanding of someone or something”. Without this deep understanding, how can you have the confidence to make rapid decisions on what to do next. You will only risk wasting time and energy by outsourcing. 

Things you can do: 

• Shift resources you reserved for an agency to internal team members.
• You should always design the experiments but should outsource the design components of your experiments. Eg, landing pages, explainer videos, digital brochures. This allows you to maintain control over the experiment while freeing up your time. 

Analysis Paralysis

Overthinking things that you should just test and adapt. 

Having ideas and concepts are good, but too many teams overthink and waste time, rather than getting out of the building to test and adapt their ideas. Keep your eye on the prize. Ideas are not the most important thing. What’s more important is to run experiments so you can gather enough evidence to inform your next decision. 

Things you can do:

• Timebox your analysis work.

• Differentiate between reversible and irreversible decisions. Act fast on the former. Take more time for the latter.

• Avoid debates of opinion. Conduct evidence-driven debates followed by decisions. 

Running Too Few Experiments

Conduct only one experiment for your most important hypothesis. 

 Few teams realize how many experiments they should conduct to validate a hypothesis. They make decisions on important hypotheses based on one experiment with weak evidence. 

Things you can do:

• Conduct multiple experiments for important hypotheses.

• Differentiate between weak and strong evidence

• Increase the strength of evidence with decreasing uncertainty.

Incomparable Data/Evidence 

Messy data that are not comparable. 

Too many teams are sloppy in defining their exact hypothesis, experiment, and metrics. That leads to data that are not comparable (e.g., not testing with the exact same customer segment or in wildly different contexts). 

What you can do: 

• Use the Test Card.

• Make test subject, experiment context, and precise metrics explicit.

• Make sure everybody involved in running the experiment is part of the design.

Weak Data/Evidence

Only measure what people say, not what they do. 

Often teams are happy with running surveys and interviews and they fail to go deeper into how people act in real-life situations. 

What you can do

• Don’t just believe what people say.

• Run call-to-action experiments.

• Generate evidence that gets as close as possible to the real-world situation you are trying to test. 

Confirmation Bias

Only believing evidence that agrees with your hypothesis. 

Confirmation bias is the tendency to search for, interpret, favor, and recall information in a way that confirms or supports one's prior beliefs or values. Sometimes teams discard or underplay evidence that conflicts with their hypothesis. They prefer the illusion of being correct in their prediction. 

Things you can do

• Involve others in the data synthesis process to bring in different perspectives.
• Create competing hypotheses to challenge your beliefs.
• Conduct multiple experiments for each hypothesis. 

Failure to Learn and Adapt

When you don’t take time to analyze the evidence to generate insights and action. 

Some teams get so deep into testing that they forget to keep their eyes on the prize. The goal is not to test and learn. The goal is to decide, based on evidence and insights, to progress from idea to business. 

Things you can do:

• Set aside time to synthesize your results, generate insights and adapt your idea.
• Always navigate between detailed testing process and big picture idea: which patterns that matter are you observing?
• Create rituals to keep your eyes on the prize: ask if you’re making progress from idea to business.