Symmetric Sampling

It’s no secret that the entire marketing research industry is concerned about data quality.

Since the introduction of online survey methodologies in the early 2000s, sample and other research providers have been struggling to protect clients from bad respondents, who can ruin the quality of the data clients receive from them. As the saying goes, “Bad people give you bad data.” And if that is not a saying, it should be!

Marketing researchers should place questions in the screener to help identify (and remove, post hoc) low-quality respondents. Red herring questions, choosing too many items in a multiple-choice question and eliminating respondents who complete the survey too quickly are designed to 'trap' users who are either not paying attention, speeding through it or do not fit the respondent profile. These techniques are common attempts to protect data quality on the survey side.

There are also precautions that are routinely instituted by most sample providers:

• Checking the respondent’s digital fingerprint can eliminate some of the offensive respondents. According to Wikipedia, a digital fingerprint is information collected about a remote computing device to fully or partially identify individual users or devices. Tracking and monitoring respondents’ digital fingerprint allow sample companies to ensure that each respondent can only submit one survey, preventing multiple submissions from the same machine.
• Bot prevention devices are becoming increasingly used to identify automated respondents. One of the most popular, CAPTCHA (which is actually an acronym that stands for Completely Automated Public Turing Test To Tell Computers and Humans Apart) can be used in the screener to eliminate machines programmed to respond to surveys.

  All of these methods are effective and can result in removing between 2% and 5% of completed surveys.

  A Good Start, But Not Sufficient

  The problem becomes that many respondents are onto our methods – and know how to beat them. To truly improve data quality, we need to take it a step further. We need to add humans to the data cleaning process.

  For example, Symmetric Sampling adds a single open-ended question to each survey that asks: “For quality-control purposes, please tell us the main topic of this survey. Be specific and include details.” We then have our data quality team review every response and make the judgment to keep or eliminate that respondent. If the respondent has been taking their time and reading and answering the questions thoughtfully, they will be able to provide an acceptable open-ended response. However, if they have not been reading carefully, they cannot give an adequately detailed and specific response. We remove those respondents, adjust the quotas, and collect better data from engaged respondents.

  We have found that this simple approach identifies another 5% - 7% of bad respondents. That’s right – an additional 5 – 7%, bringing the number of respondents removed for low quality to between 7% and 12% of the total sample.

  This approach adds very minimally to the cost and delivery time of the study. We have found that most researchers find this approach extremely appealing, even when using samples drawn from their customer base. After all, if more than 7% of your responses are low quality, that could potentially be a much higher cost for your company in misinformation and bad decisions!

  If you are concerned about sample quality impacting your research results - and who isn’t? - consider using this method. You owe it to yourself, and you might be surprised by the result!

  Concerned about data quality? Add the Human Touch! Contact Symmetric today:
  Phone: (817) 649-5243
  Email: info@symmetricsampling.com
  Jason Thomas
  President of Symmetric, a Decision Analyst company