Qualitative Quants for Usability Researches

Any UXer would have been asked at least once before; Qualitative vs. Quantitative? Why not both? Which one first?

Some problems need creative solutions. We think we have a good one here. But first, Let’s explain what is the meaning of Qualitative and Quantitative for the visitors who are not so familiar with the terms.

 

Quantitative research

Quantitative research is expressed in numbers and graphs. It is used to test or confirm theories and assumptions. This type of research can be used to establish generalizable facts about a topic.

Common quantitative methods include experiments, observations recorded as numbers, and surveys with closed-ended questions.

 

Qualitative research

Qualitative research is expressed in words. It is used to understand concepts, thoughts or experiences. This type of research enables you to gather in-depth insights on topics that are not well understood.

Common qualitative methods include interviews with open-ended questions, observations described in words, and literature reviews that explore concepts and theories.

 

Qualitative

Analyzed through math and statistical analysis

Mostly expressed in numbers and graphs

Requires many respondents

Close-ended questions

Quantitative

Analyzed by summarizing and interpreting

Mostly expressed in words and videos

Requires few respondents

Open-ended questions

Can we bring these two closer?

Quantitative studies are structured, controlled, and unmoderated by nature but Qualitative studies are generally unstructured and moderated.

Can we use technology to get qualitative results from quantitative studies?

Well, we think we can!

This is how we do it

Step 1 - Prepare an advanced prototype

Get ready, we are going to have to develop the s**t out of this!

We will create an advanced prototype, but this time it will not only for showcasing the product but also for collecting the data from the users. For this job, we are using a mixture of a few different tools and technologies. Framer X, Real-time data capture API, Analytics, JavaScript, PHP and more.

Now We will need to add short surveys with open-ended and close-ended questions to show whenever you need to get input from the participants. These surveys should be triggered by an event, i.e. completion of a task, time-out or a fail.

We need to capture two types of data; automatically and manually entered. Automatic data must include: Unique ID, IP, device information, and as many events (Page load, Scroll, Click/Tap, and etc.) as possible.

Useful Events to Capture

- Device related (i.e. Model, Screen size)

- Software related (i.e. OS, Browser)

- Network related (i.e. IP, Location)

- Events related (i.e. Click/Tap, Scroll)

It is common to get responses like these:

Which Industry does your company belong to?

sdfsfdsfdsfsd

Top 3 reasons for selecting your current bank

useful

Just filter out these participants, report them to your panel provider. If you have the panel yourself, blacklist them, and ban them from future studies.

Step 2 - Trim the fat

Some people will be here to just make noise, ignore them.

If you have done a quant survey before, you will know that there will be a portion of the respondents who are just here for some quick money. They just select the answers without reading the question and type random keystrokes to any open-ended questions. We need to filter these participants out for more accurate results.

There are some methods to identify these participants, hence the best method will still be a mix of computer algorithms and human. A basic grammar check will filter a lot of random keystrokes answers. Unfortunately not all grammatically correct answers are legit, and vice versa is true too. It is very common that participants enter "beautiful" or "good" to any question, and you can't kill all the participants with typos. So prepare yourself to read a lot of entries.

Step 3 - Be agile and run the study in waves

Smaller waves will help you to dive deeper into findings

The best part of a moderated study is, we can deep dive into any unusual/interesting findings during the interview. Dividing the study to smaller waves will enable us to achieve that.

We will use the opportunity to change the prototype and survey a bit in between to experiment and understand the users' behaviors and preferences better. See below for an example schedule from one of our studies (n=200)

We found 2 weeks study sprint in 4 waves is the optimal approach that fits for many UX studies. Keeping Mondays and Thursdays for studies, remaining days for analyze and tweak the prototype and surveys enables us to deep dive and experiment more with the product.