Metriclab | Relentless research and innovation

With more companies today putting a stronger emphasis on gaining a deeper understanding of their customer, it’s not unusual for us to be called in for a project to find that our clients don’t have a lot of experience with research and don’t know what to expect.

This article is for every designer, architect, manager, engineer, and stakeholder who wants to know more about research and is intended to provide you with the most critical tools for interacting with researchers and understanding how the work that we do can make your job easier.

This article will also outline what to expect from researchers and some ways to recognize when you’re working with a good one. These are indicators, not standards, based on what we’ve found to be effective. There are many ways to do research and every research study is different so it doesn’t mean that a researcher is incompetent if he or she doesn’t conform to these indicators. One sign of a strong researcher is that he or she will educate you throughout the process so that you know what to expect. With that in mind this article is ultimately intended to provide a useful starting point.

Recruiting

One of the most critical and time-consuming elements of test preparation is defining the right target audience and recruiting participants. Participant recruiting is usually conducted by professional recruiters who typically consult databases of potential participants. Sometimes researchers will do the recruiting themselves, but it’s usually more cost effective to use a specialist.

Recruiting will almost always take two weeks or more depending on the number of participants and the type of research, so make sure that you get started early enough for the recruiter to have enough time to find the appropriate participants for the study. Recruiting for phone interviews may take slightly less time and any kind of home visit will likely take longer (ethnography or contextual interview). Your researcher should be able to provide you with an estimate at the time of initial engagement.

A week for recruiting tends to be difficult and any less than that is pretty much unthinkable. Short-changing the recruiting could result in participants that don’t properly fit the target market segment, don’t provide quality feedback, or just don’t show up at all. All of these can have a negative impact on the data. Even if it is possible to get participants faster, it’s usually better to take the time to ensure that you are getting the right people. Your researcher should know all of this and recruiting participants is where he or she will start after getting a basic understanding of your product and schedule.

A recruiter will need a screener to get started. A screener is a description of the target user with open and close-ended questions about the participant that will help the recruiter to select the right people. What you can do to smooth the process along is to have a prepared concept of your target user. This does not need to be a full market research report—just an outline of the types of users that will use your product.

Your researcher should dig deep with questions that include more than demographic information by asking behavioral questions. Behavioral questions can include such topics as TV watching behavior, purchasing behavior, internet use, etc. Typically behavioral questions will give you a stronger understanding of those who are being recruited than demographics alone. These are important elements of market segmentation that are sometimes organized into profiles called personas.

Personas are useful because they create a consistent concept of the intended market segment that can guide the design process through multiple iterations. Personas can also be adjusted following deeper discovery research, such as in-depth interviews, as more information about the intended user comes to light. Within a few days, the researcher should present a screener that includes behavioral questions as well as demographics.

Scheduling

When creating a schedule for data collection, the researcher should know that you cannot run participants back to back. It’s generally not feasible to squeeze in 8 one-hour sessions in a single day, because of all of the activity that must occur between sessions. In an eight hour day, a researcher can perform four (maybe five) one-hour sessions but any more than that will take more time. Here are the reasons why:

One-hour sessions rarely go exactly one hour, some are shorter and quite a few will run longer. This can be due to a variety of reasons such as the product malfunctioning, the participant arriving late, or the participant providing lots of feedback. My rule of thumb is to allocate 50% of the session length as a buffer between sessions to allow for overrun, not including time needed to set up for the next session.

For sessions at an office or lab, some participants will arrive 10-20 minutes early, at which time they will need to use the restroom, sign NDAs and consent forms, and generally get comfortable. Comfortable participants give useful feedback, while uncomfortable participants tend to clam up and provide short, unemotional responses.

The researcher needs to set up and get ready. For usability or experience testing, the test will need to be reset, notes and documents need to be filed and new ones prepared. For any kind of home or location visit, the researcher will need to pack up all equipment and travel to the new location and set up equipment again.

Thus for every one-hour usability or experience testing session, there’s forty-five minutes to an hour of buffer and setup time. Home visits can take much longer.

Test Plan

A test plan should take no more than a week to develop and the researcher should give it to you for review and approval before being finalized. The test plan should specify the research and business goals associated with the project. During this period the researcher will need a significant amount of time with the product, either with a prototype or available concepts, while writing and checking the test plan. The better the researcher understands the intended final product, the more valuable the information he or she can get from the participant.

For usability or experience testing, the researcher will test the tasks with the product prior to a pilot test. He or she will need to make sure that there are no glitches, no unexpected areas under construction, and nothing giving away future tasks when performing each of the tasks with the product. With that in mind, it’s important to give the researcher a stable product or prototype and avoid drastic changes to the product prior to the test.

You should receive a well-written and organized test plan that details each research question and how it will be addressed. For usability testing this will include a list of tasks, what each task is intended to examine, approximate wording for the task (avoiding leading language), and detail on how each task will be scored or evaluated. For discovery research, it will include a list of topics to be addressed such as processes, environment and context, and expected pain points and needs.

Data Collection

When the data collection starts, it’s important to let the moderator work. During this time, the participant should feel comfortable enough to open up and provide honest feedback. In order to do this, it’s important to try to minimize observer impact during the testing session.

If you don’t have a separate place to watch the session (e.g. behind a two-way mirror or through a video feed), don’t make it obvious that you are paying close attention. Think about bringing in a laptop during the session to make it look like you’re doing other work. One way of doing this is telling the participant that you are also a researcher but you’re just going to be taking notes.

When you’re observing, remain objective and don’t make judgments based on one or two participants. It’s not uncommon to see a couple participants have a completely opposite reaction to a product compared to ten other participants. The researcher’s job is to sort through all the noise and report the real trends in the research. Take what you see with a grain of salt and listen to your researcher.

At the same time, it’s important to try to observe as many sessions as possible and give your researcher feedback between sessions if there are certain aspects of the user experience you want to know more about. The researcher should put the participant at ease and extract a great deal of information, including details that might have been overlooked or emotions that the person experiences. Different researchers will tend to achieve this in different ways as everyone has their own style, but you’ll notice by paying attention to the participant and seeing if they feel relaxed or nervous throughout testing.

Findings

Frequently, stakeholders will want to make immediate changes to a design, product, or prototype and won’t have the time to wait for the researcher’s final report. People have schedules that need to be met so it’s understandable that a project can’t always wait for the final report but the researcher should be able to provide you with quick findings within 24 hours of the last session.

For usability research, these quick findings should consist of a couple of short paragraphs including problems in the interface, possible solutions to these problems, and participants’ general reactions to the product, its look and feel, and expected usage. For ethnography or other forms of discovery research quick findings will tend to consist of expected usage of the product, expected value, high and low value features, and general trends about the intended user. Quick findings aren’t comprehensive and come before the researcher can get a complete look at the data, but it will provide you with the overall themes from the study.

When you do get the final report, make sure you take a look at it. It will tell you two things:

Detailed findings regarding the interface, product, features, and intended user
The quality and clarity of the report will tell you quite a bit about the quality of your researcher.

There’s one other thing to keep in mind when you are processing the findings from a usability test. The participants will tend to focus on the more obvious problems with a product or interface. There could be other, smaller or more abstract problems that are not identified in the first pass of usability testing. It’s usually a good idea to perform another test on the product after making changes to ensure that the changes you made were effective and identify any additional issues.

Summary

In summary, here are the most important points for non-researchers to know about the research process: