Craft Your Questions

Learning Objectives

After completing this module, you’ll be able to:

• Use aim and scope to help craft your questions.
• Structure questions that flow effectively within your sessions.
• Craft questions that are open, non-leading, and unbiased.

The Art and Science of Crafting Great Questions

You’ve learned how to set up and define your approach for the user research. Now, let’s get creative and learn how to build the questions you’ll ask your users. Building questions is an art and a science, so no surprise, there are some guidelines around crafting them. When crafting questions it is important to consider the:

• Aim and scope of the study
• Structure and flow of the questions
• Way the questions are worded

As you develop questions it’s super important to work closely with the folks who are most interested and affected by the results of this study—your audience. Get your audience engaged up front by having them provide feedback on your questions. Use their feedback to improve the questions until you’re ready to interview. Use these iterations to build stakeholder engagement in your project.

Use Aim and Scope to Shape Your Questions

First some definitions:

•  Aim is the intent of your work—what you’re trying to find out.
•  Scope defines who you’re researching and how you’re doing your analysis.

Defining the aim and scope of your study helps focus the direction of your questions so your questions can be analyzed with ease.

Let’s take a look at how each of these factors influence the structure—and eventual design—of your questions.

What Are You Trying to Explore?

As a researcher you’re trying to understand your users’ current experiences and their initial thoughts about new products and services. This means that your questions will need to ask about your users’ aims, motivations, their work, and how they use your product or service to get that work done.

Once you’ve asked these background questions, you can begin to explore what works and doesn’t work in their current workflows. This exploration helps discover unmet needs and opportunities to make the overall experience better. You’re looking to discover what works well, what doesn’t, and what could work better. Building questions that bring to light challenges can help you gain insights into your users’ way of thinking.

Help jump-start the conversation by having visuals for your users to give feedback on. This could be a sketch of your feature or maybe even the product itself. When you do this, focus your questions on:

• Whether users can easily identify and tell you what steps they're supposed to take with the product—namely the ones you want them to take.
• What their initial impressions are of the various parts of your concept.
• Whether they understand the changes that you're proposing.
• What their impressions are on how the screens flow together.

In her 2013 book, The User Experience Team of One, Leah Buley describes the “Quick and Dirty Usability Test,” where you ask users what they see being presented to them and what they would expect to see as they progress through the designs. This approach enables you to see how the concepts you propose align with what your users expect.

Using Personas in Research

When you’re developing your questions, along with considering the aim and focus of your study, it is important to think about who you want to interview.

In unit 1 we covered some aspects of using segments—also known as personas—to choose participants. Personas can be used to discuss who your key users are. A persona is a realistic character description that represents a particular set of your users. Personas can help identify, through conversations with your team, the users you decide to recruit for your study. Personas help with team dialogue and in making design decisions. In the UX Personas for Salesforce module you will get to know the Salesforce personas and how to use them.

How Are You Going to Analyze Your Data?

 Quantitative data: When you’re crafting your questions, it’s always a good idea to think about what you want to get out at the end. For example, if you need to collect quantitative data, you might consider having your users perform a task using your concept. You can capture quantitative data such as:

• Time taken to complete a task
• Number of errors made in completing a task

 Qualitative data: It may be fairly straightforward to think about how you will analyze quantitative data. For example, you might use formulas like averages or differentials. But how will you analyze the qualitative data you collect, such as the notes on what your participants have said? Qualitative data is trickier to analyze, because you can’t simply calculate a result. Think ahead to how you’ll collate your notes and then categorize and analyze all your data. We’ll cover how to analyze your data in unit 4.

Define a Structure and Flow for Your Questions

Now that you’ve had the chance to think about the aim of your study and the type of users you’re investigating, we’ll tackle how to structure your session.

Think of your interview like an hourglass. Start broad and general, then narrow your focus to specifics, then go broad and general again to wrap up.

 The opening: You can use the beginning part of your session to ask questions about your users’ work, how they currently use the product, and general thoughts on how things are currently working for them or not. Ask questions like:

• Can you describe for me what a typical workday looks like?
• What are the biggest challenges you face with respect to your daily work?

 The middle: Next narrow your focus with questions that help drill down on detailed feedback on the feature that you’re trying to improve. For example, ask about specific features of your app, elements of a concept, or even fields on a page. Dig into answers provided by asking “Why?” which gets at the reasonings behind your user’s thoughts. Ask questions like:

• What do you think about the new/revised [topic of study] features in the app? Why do you feel this way?
• Before we go to the next screen, what do you expect to see? Why?

 The wrap up: This gives you an opportunity to ask big questions about users’ overall impressions and to capture last-minute and often super-interesting thoughts. Ask questions like:

• If you could change one thing, what would you change?
• How would you rate your experience (from 1 = very poor to 7 = excellent) with this design/prototype?
• How would you rate the usefulness (from 1 = not useful to 7 = very useful) of this new features/design?
• Do you have any further feedback for us?

Checking back in with Carla, she’s started to think about how she might analyze the answers she gets. She wants to uncover overall impressions of the new sales application by counting the number of times people have positive or negative reactions. She can also collect patterns in the feedback on her concept and use those to identify possible solutions. Take a look at her draft questions:

 Opening 

1. How long have you been in this role?
2. What does a typical day look like?
3. What’s the most challenging aspect of your role?

 Middle 

1. What do you think about these first two screens?
2. What do you expect to see next?
3. What do you like about this screen?

 Wrap up 

1. Having now seen the entire design, how much do you like it?
2. Do you have any further feedback for us?

This is a great start, but it could also use a second or third pair of eyes. Once the rest of her team has reviewed it, she’s ready to screen for bias.

Remove Bias from Your Questions

When thinking about the way your questions are worded, it’s important that they’re open and not leading. As you might guess, leading questions lead the person you’re interviewing into giving you a specific answer.

Leading questions create bias. And the scary thing is, it’s really easy to fall into the trap of asking biased questions. A tip that helps avoid bias is to remember that you’re trying to get at what your users are thinking about.

That seems obvious, but let’s check out an example. See if you can recognize the differences between these two questions:

• What do you like about this feature in the app?
• What do you think of this feature in the app?

The first question might lead your user to provide only positive impressions that they have of the feature, while the second allows for both positive and negative impressions. The second is an example of an open question. You might also notice that the first question asks how a person feels instead of how they think. Emotion is really hard to analyze because it’s very personal.

As you can now see, if a question is asked in a biased way, it can skew the responses you get, which leads to findings that are off the mark. And we definitely don’t want that!

 Bias-busting tips: 

• Stay away from leading questions
• Ask open questions
• Ask what your users are thinking rather than what they’re feeling

Looking back at her questions, Carla realizes that a couple of them might be biased (after all she is pretty excited about the design her team has developed!). She changes a couple of her questions.

Old Questions	Bias-Busting Questions
What do you like about this screen?	What do you think about this screen?
Having now seen the entire design, how much do you like it?	Having now seen the entire design, what are your thoughts about it?

Check out the resources we’ve added that show great examples of which questions to use and which to avoid.

Practice Makes (More) Perfect

It’s a basic truth in life that the more you practice the more perfect your delivery—this also applies to conducting interviews. This step helps you:

• Plan how long your interview will be
• Trim down or bulk up the questions you ask
• Improve the flow of your questions
• Refine and clean the bias out of your questions

The best way to practice is to find a volunteer within your organization to act as your user. Make sure you’re comfortable enough with this person to ask for constructive feedback on the experience, and give them information about the persona they’re embodying. Practice going through each question, let them answer, and time the dry run from start to finish.

Expert tips!

• During the actual interview, some folks will be chatty. Ask your volunteer to answer concisely, and then pad your interview duration with 5–10 mins.
• Short precise questions tend to lead to short precise answers—use this strategically.
• Ask your important questions early in case you get a verbose user. Highlight the “must have” questions to ensure you ask them.
• Do your practice run early enough to adjust your questions before your first interview.

We’ve covered how to create your questions and how to structure your session. Next, we’ll look at how to conduct user research in a way that avoids bias and collects great insights. You’ll also learn how to interview your users about what they do and what they need. Exciting stuff!

Resources

• Open versus Closed Questions: Reference table describing differences between them
• Portigal, S. (2013). Interviewing Users. Rosenfeld Media LLC. Brooklyn, NY
• Buley, L. (2013). The User Experience Team of One. Rosenfeld Media LLC. Brooklyn, NY.
• Pruitt, J. and Grudin, J. (2003). Personas: Practice and Theory. ACM
• Understanding Your Users, Second Edition: A Practical Guide to User Research Methods (Interactive Technologies)