Design, Conduct, and Validate Your Experiment

Learning Objectives

After completing this unit, you’ll be able to:

  • Start designing your own experiments.
  • Identify things that can cause an experiment to fail.

Designing Your Experiments

The design of an experiment happens after you’ve formed your hypothesis. You’ve identified the potential value of the experiment, now it’s time to create your steps to test it. The key is to keep this as simple as possible—you are looking for an indicative outcome, not an absolute solution. 

Typically, an experiment design encompasses the following principles.

  • Short time frame: Keep the whole plan to executable tasks that take the shortest time possible (less than 10 days is ideal).
  • Focus: Determine your high-level activity for the experiment (for example, build mock-ups, execute with a test group).
  • Key resources: Include only the key resources (people, equipment, technology, and so on) that help you execute the experiment.
  • Exit criteria: Decide what factors determine when your experiment has come to an end.

In building the design for the experiment, ask yourself these questions.

  • What blockers can I avoid through experiment design (availability of data, infrastructure constraints, and so on)?
  • How can I get my idea, method, or activity tested in the most basic way?
  • What resources do I need to help?
  • Am I testing to find correlations or causal outcomes?

In the previous unit, we outlined a few examples of hypotheses in experiments. Let’s build on those examples and demonstrate a design for an example experiment.

Customer Support IVR

Hypothesis: By introducing an IVR to the Customer Support Line, we can reduce Agent-Facing Account Queries by 30%.

Design: The experience designer produces a series of mock-ups with a storyboard to replicate the flow of the IVR in a number of varied scenarios. These scenarios are scripted, and rather than develop a chatbot, a human plays the role of an IVR and follows these scripts for calls until 100 calls are processed. The calls that are managed in this way without requiring further agent engagement are recorded.

Premium Subscription

Hypothesis: By offering a Premium Subscription Service in addition to the Basic Plan, we can increase our revenues by 20%.
Design: Draft a feature set that outlines what the Premium Service offers over and above the Basic Service. Sales reps approach customers and ask if they are interested in upgrading. A control group is offered similar feature sets as a part of their Basic Plan to determine if the pricing is an issue.

Executing Your Experiments

Now that experiment design is covered, it is time to consider how to execute it. When experimenting, it is crucial that you maximize the opportunities while minimizing the time, effort, and cost that’s required. 

Just like the hypothesis, you don't want to overcomplicate the execution phase of your experiment. Here are some steps to follow that can help keep things simple.

  • Retain a clear view of the action you plan to take. Do not deviate.
  • Follow the design while always questioning if the process can be simplified.
  • Continually measure against the intended outcome.
  • Keep your stakeholders in check.

Sometimes, it's tempting to introduce new elements into an experiment (that is, additional features, technology, people, or resources), particularly when it appears to be going well. Expanding the scope too early can quickly turn a simple experiment into a resource-hungry beast.

Experiments Fail

Experiments fail. By fail we don’t mean the hypothesis was not proven. A failed experiment happens when it’s not completed. 

Typical reasons for failed experiments include:

  • Sponsorship either was not in place or the experiment was not approved.
  • It was lacking resources.
  • There was an unclear hypothesis.

These are just reasons an experiment can fail. Essentially, you can look at an experiment as a short-scale, low-impact project. Treat it as that from a planning perspective.

However simple an experiment intends to be, it is critical that you follow the appropriate channels for approval and that you do not get started before you’re ready. Keep experiments on-point and your experiment shouldn’t fall into the pitfalls noted previously. 

Validating Your Experiment

Awesome—you have moved from idea to action. We can assume that you now have a hypothesis, an experiment design, and you have executed against the design. Now is the time to validate your experiment against the hypothesis.

It is important to remember that when validating results, an experiment which did not meet your hypothesis is not a failed experiment. It’s a chance to learn something new, even if it doesn’t prove your hypothesis to be true.

When validating your experiment, the hypothesis becomes critical. This is why we stressed its importance at the start of this trail. You essentially have your measure to determine the degree of success. Let us have a look at a few of those same scenarios we discussed previously.

Customer Support IVR Can Reduce Agent Calls by 30%

The experiment found that introducing an IVR reduced calls that required agent engagement by 43%. The recommendation became to develop the IVR solution, retiring the human intervention, and testing across a larger number of calls in a follow-up experiment. By identifying demographic engagement with an IVR, we can tailor IVR engagement on calls to reduce the number of calls going to agents by 50%.

Premium Subscription Increases Revenue by 20%

The customers interviewed overwhelmingly agreed with the additional feature-sets. However only 10% signed up for the additional service. When assessing the control group feedback, an overwhelming majority (70%) said they would utilize the additional features. It was recommended to rerun the experiment with a focus on price variation.

Resources

Keep learning for
free!
Sign up for an account to continue.
What’s in it for you?
  • Get personalized recommendations for your career goals
  • Practice your skills with hands-on challenges and quizzes
  • Track and share your progress with employers
  • Connect to mentorship and career opportunities