Usability testing, like any component of development, involves a
certain amount of planning, thought, process development and execution.
Encouraging and delivering a well-delivered process and execution
for usability testing can help bring usable products to intended
consumers.
Whether your organization conducts its own usability research
or an outside agency handles it, each key person should be concerned
with the process of conducting usability research.
Although we cannot exhaust the subject of usability research
or testing here, this article offers up some topics of consideration
and steps to follow when conducting usability research. You can
make the following assumptions when reading this article:
- The particular focus of usability is on that of the user interface
or design of a web site or software application.
- Our style of usability testing is qualitative in nature, gathering
data from actual users though successive, in-person, one-on-one
interviews.
Part I: Planning your usability test
Step 1. Start thinking about usability testing.
Usability testing should be a part of product development. Adding
usability testing to the development process at the beginning
of development will aid in the execution of usability testing.
Devote ample time to determining a product’s usability before
you release it to the world. The expectations of usability testing
should be determined in the process plan. Set the expectations
of the product’s usability ahead of time, involve key players
and bring in the decision makers. Finally, set a schedule for
each step in each iteration of user testing (see a sample
schedule of one iteration).
Step 2. Define the audience and the goals of usability testing.
Once the usability testing phase of product development begins,
you'll need to define or redefine the audience demographics. Determine
who will use your product, what their specific demographic characteristics
are and how best to approach the testable individuals. Ideally,
every product has a viable and definable target audience (why
wouldn't it?), so that product should be made usable for the intended
audience. The goals of usability testing should therefore be focused
on determining whether this interface is usable and whether the
intended audience, and anyone else who might come in contact with
it, can use it. The needs and desires of the intended audience
often drive the goals of usability research.
Defining the goals of usability testing isn’t always as
simple as addressing the most obvious things, such as “can
they use it?” or “do they like the colors?” Other
goals often address perceived usability problems for existing
products, such as installation instructions, site drop-offs or
customer complaints. For online stores, most existing site stakeholders
are concerned with users dropping off at certain points in the
shopping or checkout process.
Develop your goals for the usability testing project by asking
these questions:
- What should the intended audience be able to get out of the
product?
- What do you want to know about your product?
- What do you already know about the usability of your product?
- Where are the known problems, issues and bugs? In particular,
are there any issues that won't be resolved before you launch?
(A more extensive list of questions appears
here.)
Perhaps this is the first question you should ask:
- Are the problems actually usability problems or are they product
problems that go beyond usability?
Knowing the answer to that question is key. If the product crashes
your user’s machine, then it's probably safe to say that
the issue goes beyond usability. For this reason, you should only
test what is testable. Don’t take the time to develop a usability
testing plan that tests something that lacks stable functionality,
isn’t ready to be tested or is already planned to be significantly
changed in development before the final release.
Be sure to set reasonable goals. Most of the time, you will not
be able to achieve all your usability goals with one test. Conduct
iterative testing projects and keep your goals focused and reasonable.
Step 3. Create the test script.
After you determine the goals of your usability test, turn them
into a test script—the guide to be used by the moderator
when interviewing potential or existing users. The test script
is a critical step for getting good data. It should be written
as though it will be read aloud to the testing participants. When
writing the test script, you should include not only all the questions
about the product’s usability, but also any statements, phrases
or verbiage that will actually help the person being interviewed.
The participant needs to be the focus of the test script. The
questions asked, tasks presented and opinions sought should be
written and spoken in a way that allows the participant to feel
as though they are contributing to the project, rather than being
tested by the moderator. When asking a participant how they would
add something to their online shopping cart, check out and complete
the order process, the task should not be a test of the participant's
ability to use the computer to check out, it should be a test
of the checkout process itself.
Before asking questions regarding the product, your test script
should start with information to help participants understand
what it's all about:
- Explain why the participant is there.
- Recap what the non-disclosure agreement means.
The script should ask participants questions about themselves
to make them feel comfortable and help set the tone for the interview.
Using questions where the answers are already known—such
as "How long have you been using the Internet?" or "What
kind of computer do you have?"—will help users through
the remainder of the session. Questions about the product’s
usability should not be a series of yes/no questions, nor should
they be interrogative in nature. If they are, you won't get any
good data because users will feel uncomfortable.
The format of the test script should include qualitative questions
and directed tasks regarding the product. Give users tasks to
complete and ask them questions about what they’re seeing.
Here are some examples:
- Do you like the colors?
- Can you read the text?
- Where would you click to get information on the company?
- You’re being shown a list of steps to complete the installation
of the product. Could you read that list and tell me if the
instructions are clear? Do they make sense? Do you know what
you’re supposed to do next?
- Why are you being asked to register?
- Could you please show me how you would buy [name of product]?
- How did you feel about that order process? Was it easy? Hard?
Did it take too long?
The final step in the test script development should include
a test drive that's conducted with the most key player in the
organization concerned with the product's usability. This could
be the information architect, designer, developer or marketing
manager. The test drive helps determine the organization of the
test script, length of the test, flow of the questions and scope
of the usability research.
In addition to writing a test script that works well for the
participant, please keep in mind that participating in a usability
test can be a nerve-wracking experience. Participants arrive at
a place they’ve probably never been to, they’re asked
to sign non-disclosure agreements, they’re brought into a
somewhat sterile environment and then they're sat down with a
camera pointed at them. They usually figure out they’re being
watched, too.
Step 4. Recruit your users.
This phase of usability testing can be conducted in conjunction
with your script development. Look at your intended audience and
fine-tune it based on the logistics and practicality of the usability
testing. For example, if user testing takes place in San Francisco,
you’ll probably only want to recruit participants in the
San Francisco Bay Area.
Figure out also how many participants to interview. Although
this topic is often debated in usability circles, we recommend
no fewer than eight for any usability testing project. Five users
help develop trends in a product’s general usability as reported
by the participants, and three more help qualify those findings.
So eight is the bare minimum.
Develop a screener that helps you find the participants you want.
If you make it broad or too vague, you'll end up spending more
time weeding out inappropriate respondents. Use the screener as
an e-mail message or phone script. Examples of screener information/questions
include the following:
- Name
- Phone number
- E-mail address
- Age (or age range)
- Income level (or range)
- Do you use a PC or a Macintosh?
- What is your connection speed?
- Job title or occupation
- Do you use [name of product] for your job?
- How long have you been using [name of product]?
- Have you used other similar products? If so, which ones?
Recruit users far enough ahead of time to allow for cancellations,
schedule reorganizations and for any focus changes. Allow enough
time to get at least three times the number required for your
usability testing project. By doing so, you’ll weed out many
respondents. Receiving a good number of responses gives you the
luxury of picking and choosing appropriate participants for your
study.
Part II: Conducting your interviews
Now that you’ve finished your goals, test script, test drive
and recruiting, it's time to test. The moderator should be comfortable,
healthy, focused and mentally ready for a day or two of usability
testing sessions. Things to keep in mind when moderating and gathering
data include the following:
- Set up and conduct interviews in a comfortable space.
Make sure the room is quiet and private. Provide snacks and
beverages. You may want to use contextual spaces such as an
office environment.
- Set up the video equipment correctly. The user should
not be staring down the camera’s lens. The video camera
should not be pointed at the screen, either. Use an angle that
incorporates the screen and some part of the user. You may wish
to consider picture-in-picture videotaping.
- Be aware of the time. Interviews should last their
intended time and not much longer. If the session lasts too
long, the user will be become agitated or bored.
- Stay flexible. Don’t go into “clinician”
mode. Users are unique and the moderator should do his or her
best to be comfortable so that the comfort level and ease are
conveyed to the participant.
- Don't place any importance on one user. One participant
may present the most compelling data about your product. However,
one user’s comments will not solve a product’s usability.
As exciting as the user’s comments may be, stay focused
on each user’s comments.
- Be opportunistic. If a participant says something
interesting or compelling, ask them to clarify it. For example,
if a participant says “I didn’t see that. Why is it
over there?” when probing the interface, ask where they
think it should be. Seek help from their expectations of the
interface. Similarly, if a user hesitates it often means that
something is confusing or unclear (unless they're reading!).
Follow up on these hesitations by asking if the information
makes sense or if they know what they’re supposed to do
next.
- Take notes. For many user testing projects, employ
a separate note taker to observe the user testing sessions.
We recommend that the moderator be the primary note taker. The
moderator is the person who interacts directly with users. The
data that the moderator gathers is possibly more valuable than
other data (such as a note taker one step removed from the interview
or the videotape itself).
- Be attuned to observation bias. Users are being tested
and want to give the "right answer." Let users structure
tasks whenever possible. Encourage participants to talk freely.
- Respect participants. Be patient and considerate. Keep
in mind their confidentiality as well as the product's. Makes
sure users know that the information they provide is confidential.
Make sure they know that their information will not be provided
for any further purpose beyond the session. Remember to thank
them for their participation.
Between sessions, take time to think about the data that the
user provided. Organize your observations toward an analysis of
the results. Remember to get all the data from the participants
before making any final determinations.
Part III: Organizing data, analyzing data and presenting your
findings
After completing the sessions, review the data and make a list
of the top issues you discovered. From there, start flushing out
the usability concerns and the recommendations for improvement.
When analyzing your results, keep the users’ comments at
the front of the analysis. It's too easy to incorporate your own
biases and expectations here. Try not to do so. Marry the users’
comments with the analysis. To illustrate:
Observation: Five users tell you that the search box
is in the wrong place on the page because they don't see it.
Two of them say the colors are too similar to the rest of the
page; two of them don't see it because it's in the lower-right
corner of the page.
Correct analysis: Because the users report that they
cannot see the search box, it should be moved to a place on
the page where they can see it. The designer may want to rethink
the colors used in that area as well.
Incorrect analysis: The users are simply not accustomed
to the placement of the search box, but after using the site
for a certain amount of time they should become comfortable
with it.
Pay attention to patterns. After a few interviews, you may receive
very similar information regarding the product. It's important
to notice these patterns, along with any deviations from them.
If four of eight participants believe the text in the content
area is easy to read, but the other four complain that they cannot
read it, you should investigate further and analyze what specifically
they could not read and why. Was the font size too small? Did
those four complain about other text on the page? Were those four
trying to get through the task (too) quickly?
Keep a perspective. Remember that even if the data gathered consists
mostly of complaints about the interface or design, those complaints
can help. Don’t take it personally and don’t negate
the comments. The users, no matter how cynical or whiny, provide
the most useful information for the product’s usability.
Present the results to the key members of the product’s
development and ownership teams. Keep in mind the audience for
this presentation. Remember their goals as well as the product’s,
along with the original goals of the usability project. Make recommendations
based on the findings and help provide positive information wherever
possible.
As you've probably figured out by now, conducting usability tests
takes time. Executing a well conceived and well though-out project
will help you get great data and hopefully improve the product’s
usability. Careful considerations towards a product's goals and
the intended audience(s) need to be made before designing and
planning usability testing. Comments and feedback provided by
user testing participants can really make the difference in a
product’s usability. There really is no substitute for direct
user feedback.
Resources
Association
for Computing Machinery Special Interest Group on Computer-Human
Interaction
Usability Professionals'
Association
Usable Web
UseIt: Jakob Nielsen's
Website
User Interface Engineering
Usability
First
WebWord.com:
Usability and Human Factors
|
