Rumors hinted at it. A whistle blower tipped off Der Spiegel Online, one of the most widely read German-language news websites, about data collectors’ cheating. In one case, of 10,000 completed records in a data set, over 8,000 were falsified by the field service. Imagine the impact on companies that use research results to make decisions.
The intent of this article is to share some of the ways the qualitative experts associated with Think Global Qualitative work to omit cheaters and repeaters and to ultimately recruit true-blue participants.
Among the companies that use research results to make decisions are those that manufacture medical devices and over-the-counter medicines, as well as prescription pharmaceuticals. Not surprisingly, this raised concern among both clients and researchers.
Corporate decision makers are rightly questioning whether or not they are receiving accurate information from their field services. Researchers with experience and integrity know how to avoid this pitfall.
Interestingly, one of the articles written after the Spiegel exposé, called out open-ended queries as a way to help determine whether or not a survey respondent was real. Reviewing what people wrote in their text-box responses gave researchers a window into the authenticity of individual respondents.
For example, did the response make sense in terms of the question? Was the text response consistent with the other check-box responses?
As someone who has read quant responses line-by-line and routinely includes open-ended queries to ensure date integrity, this is old news. Importantly, this kind of attention to respondents’ words stems from years in qualitative research.
Experts in open-ended queries, qualitative researchers have long had specific processes in place designed to exclude inappropriate participants and to prevent cheating.
Database Diligence
In several countries, like Brazil, Canada, Mexico and the Czech Republic, national participant databases help confirm a person’s demographics as well as whether or not an individual’s past participation included any incidence of cheating.
In America, independent recruiting partners keep track of participants and omit reported cheaters from their database. Additionally, they track the frequency of participation and use this information to curb repeat participation.
Of course, these national associations, as well as the independent recruiters’ efforts, are not fool proof because the process depends on self-reporting.
Think Global Associates take additional action to ensure they include truly qualified individuals. The following examples illustrate this.
Using Pre-Work to Confirm Appropriateness for a Study
In countries where smartphone technology is widely used and relatively affordable, we routinely ask prospective participants to send in a photo during the recruiting process. Photo evidence literally becomes part of the screening process.
This request might be for a picture of a specific product in the place where it is kept in the home. Posed pictures of new product displayed on a table raise questions about whether or not the person really uses the product and alerts moderators to instruct recruiters to ask additional follow-up questions or to simply omit this prospective participant. Pictures of a half used tube of toothpaste tucked away in a bathroom vanity drawer more likely indicate actual use.
Not only are we able to confirm category and/or brand use, we get wonderful insights into product usage from these pictures.
For example, who knew that floss pick users stored these products in the console or door pocket of their cars? This single example of incorporating photo evidence of use opened our eyes to an aspect of the product that we later confirmed during subsequent research as a strong love of how convenient the product was to use.
Pictures also can show whether or not someone really holds a membership or subscription.
Say you need Netflix subscribers or those served by a certain utility company. We ask reported Netflix users to show their subscription information or request individuals that say they use a certain utility company to send a copy of their bill that includes their name.
This even works with confirming ownership of high-end products. For example, we might confirm that someone really drives a Lexus or wears a particular type of jewelry by requesting they send photos of themselves with the product.
And, we’ll be very specific, instructing potential participants to show themselves inside the car or wearing the jewelry so that we can see their face.
Absolutely, foolproof? No. Effective? Yes.
Rescreening
It’s common practice to ask to rescreen participants at the research facility before they come into the focus group room. This is a simple pen and paper screener.
We then review these screeners before taking qualified participants into the room, and we leave unqualified participants in the waiting area.
Even with rescreening in the holding room, it’s possible for an inappropriate participant to get into the focus group room. So those of us with a depth of experience routinely include questions at the very beginning of focus groups that help us determine the suitability of each participant.
For example, we might say: “Tell us your name and one thing that you really count on Product X providing.”
By listening carefully to these early responses, we can determine if someone is inappropriate for the research or not. Some of us refer to this as our sixth sense. We have a feeling, and we know how to quickly ferret out the truth using seemingly innocuous questions.
We also routinely have our recruiters’ supervisors rescreen participants. And in several instances, we have an internal person who rescreens, or as it’s referred to locally in India, “back-checks” every participant. Back-checking might even involve going to participants’ homes to ensure their appropriateness.
Verifying Identity
In face-to-face research, we ask to see participants’ identification as they sign in. This might be a driver’s license or some other form of photo identification. This keeps the neighbor down the street from sitting in for a properly recruited participant. While it might be hard to believe, this has happened.
For both face-to-face and online work, we use various methods to ensure participants are who they say they are. We might double check a recruit via Facetime or SKYPE so that we can see participants in-person and to verify their surroundings. Is their office really in their home? Do they really have a dog?
Also, we might recruit through social media, targeting groups that we know have a high likelihood of fitting the participant profile our clients require.
Let’s say we need to find men who have prostate cancer. There are a number of sites on Facebook where we are likely to find these gentlemen.
Similarly, we might go to groups on Facebook to find those who care for a parent with Alzheimer’s.
We’re transparent in our offer to give voice to these individuals, and know from experience it helps us find the right participants.
And, we might physically go to where the appropriate participant is. For example, we might go to the beach to find people who are using a specific kind of sunscreen or wearing a certain kind of flip flop. Then, there is no doubt. We see them using the product.
Employing the Sixth Sense
Experienced qualitative researchers, like TGQ associates, have developed a strong ability to read people. We are trained observers who listen intently to what participants say and how they say it. We stay tuned to participants’ body language and their tone of voice.
It is this intense attention to the participants that helps us spot red flags. For example: A participant says she’s really interested in trying a certain product or service, but she sits way back in her chair with her arms crossed.
This participant is no more interested in the product than day is night. Possibly, she’s trying to give the “right answers.” Who knows?
What we do know is that we need to respectfully explore this participant’s point of view in a way that spotlights her truths, whether she likes the product or not. And, it may mean that we need to omit her from the research.
There is nothing wrong with replacing a participants or cutting an interview short when it’s clear an individual is not qualified to be in the research. And, there is everything right about digging for truth on behalf of our clients.
Virgin Participants
Individuals in a consumer culture are aware of marketing. It would be hard not to be aware because marketing is so pervasively integrated into most societies.
Nonetheless, we honor requests for what many term “virgin participants,” those who have not participated in research in the past. We believe in the value of including fresh faces in ways that further our clients’ objectives.
To get these individuals, we start by using a past-participation screener. Sometimes these individuals are easy to find. Other times it takes hours to find just one. However, it can be important.
For example, sometimes we want that new Mom who has never before been asked about a new baby product. Other times, we know someone who was in research about shampoo a short while ago, isn’t necessarily inappropriate for research on nutrition products.
Combating Lowest-Price Pressure
Particularly in Europe, companies have started asking those who submitted the best proposals to come back with “their best price.” Companies are effectively asking top-quality researchers to submit a lower bid on their work so that the corporate entity can choose among the most qualified companies and get a really low price.
There is an old adage from India that goes “if you pay peanuts, you get monkeys.” It is this lowest-price pressure that some in the industry say is the reason field services in Germany cheated.
Industry spokespeople ask: How can researchers make a profit if they agree to work at a deficit? This pressure leads some to cheat.
From our point of view, sometimes it’s important for us to say, “No, thank you.” Doing work well costs a certain amount of money. In competitive bid situations, we put forth our best ideas and our best prices. After all, we want the work or we would not invest time and resources into the competitive bid process. So, there are times when we have to respectfully decline to compete in a way that jeopardizes the quality of our work.
The Value of Experience
TGQ associates bring the value of experience. Our intent in posting this information is multi-fold:
- To offer assurance to clients that there are best practices and researchers who employ these
- To demonstrate some of the ways we work to ensure true-blue participants
- To share with others so that they can employ these approaches in their work (After all, a rising tide lifts all boats, no?)
- To encourage others to share their best practices
It is our sincere interest to promote high industry standards by holding to best practices.