# How to improve survey data quality at scale - URL: https://www.askattest.com/blog/guides/survey-data-quality - Published: 2020-11-24 - Updated: 2026-08-11 - Description: Survey data can look credible and still be wrong. This guide explains how to prevent poor-quality responses, remove suspicious data during fieldwork and maintain consistent data quality as your research program scales up. --- Problems with survey data aren’t always obvious. A study can appear credible while poor targeting, fraudulent responses or weak survey design skew the findings. And as organizations run more research across teams and markets, they may also rely on several sample providers with different quality standards. This makes issues harder to catch and easier to mistake for genuine insight. By the time problems become visible, teams may already be acting on the wrong conclusions. That’s why data quality needs to be protected at every stage of the research process. This guide explains what causes poor-quality survey data, how to improve it at scale and how Attest helps teams collect insights they can use with confidence. ## TL;DR In this guide, you’ll learn: - Survey data can be compromised by poor audience targeting, fraudulent responses, weak survey design and inconsistent research methods. - Data quality starts before a survey launches. Teams need to define the right audience, write clear questions and provide answer options that reflect respondents’ experiences. - Once surveys are live, automated checks can identify suspicious behavior, contradictions and low-quality open-text responses. Human review is still important when the evidence isn’t clear. - Invalid responses should be removed and replaced before the survey closes to reduce cleanup and help teams reach their planned number of valid responses. - As research scales, shared templates and consistent quality standards help teams produce results that can be compared across studies. - Attest supports this process through survey design guidance, live quality checks, human oversight and integrated respondent sourcing. ## **What causes bad quality survey data?** There are a range of factors that cause bad survey data, and the issues below show up most frequently. ### Unrepresentative or poorly targeted samples Survey results can become skewed when the people who respond don’t represent the wider audience you’re trying to understand. Certain subgroups may be more likely to take part, while others are underrepresented or missing altogether. The findings then reflect the people who answered rather than the target audience as a whole. Collecting more responses doesn’t necessarily solve this problem. Here, sample source matters more than sample size because it impacts which groups participate in your survey. Even a large study can produce misleading results if important subgroups aren’t adequately represented. ### Fraudulent respondents, bots and duplicate responses Survey fraud is another major threat to data quality. It can come from people who deliberately misrepresent themselves or from bots that submit automated responses at scale. For human participants, incentives can sometimes encourage dishonest behavior. Respondents may misstate their demographics or experience to qualify or complete the same survey more than once. Some people even use generative AI to produce plausible open-text answers that aren’t based on their real opinions. Bots create a different problem. They can submit large volumes of fake responses in a short period, and more advanced bots can mimic realistic behavior or generate convincing written answers. This means fraudulent responses do not always look obviously fake. Both forms of fraud can distort findings and make it harder to trust that the results reflect genuine consumer views. ### Poor survey design and respondent experience Poor survey design can undermine data quality before fieldwork even begins. When a survey is difficult to follow or doesn’t feel relevant, respondents are more likely to rush through it or provide inaccurate answers. Common problems include: - **Unclear wording:** Leading questions can steer respondents toward a preferred answer, while ambiguous wording leaves too much room for interpretation. - **Poorly structured answer options:** Long lists can overwhelm respondents. The order in which options appear may also influence what they select. - **Questions that do not fit the respondent:** Irrelevant questions ask people to comment on experiences they haven’t had. For example, someone who never drives cannot say how much they enjoy driving. - **Options that do not reflect reality:** Yes-or-no questions can oversimplify more nuanced views. When alternative options are missing, respondents may choose the closest answer simply to continue. ### Inconsistent quality across panel and sample sources Response quality can vary from one [online panel provider](https://www.askattest.com/blog/articles/online-panel-survey) to another. These differences often become more noticeable when organizations combine several sources or [run research across multiple markets](https://www.askattest.com/blog/articles/how-to-run-multi-market-research-effectively). Providers may recruit respondents in different ways or apply different verification standards. Their processes for removing and replacing invalid responses may also vary. This means one part of the sample may have been checked more rigorously than another, which makes the final dataset less consistent. That inconsistency can make survey data harder to trust. What appears to be a meaningful difference between markets or audience groups may in part reflect how respondents were sourced and verified. ### Inconsistent methodology across teams and markets As [organizations scale research workflows](https://www.askattest.com/blog/articles/how-to-scale-research-workflows) across teams and markets, small differences in methodology can make results unreliable to compare. One market may use its own screening criteria, while another changes the wording of a core brand tracking question or uses a different answer scale. Each survey may be well designed on its own, but the results are no longer measuring exactly the same thing. This makes it difficult to tell whether differences reflect the market or the way the research was run. ### Manual QA that happens too late Data quality becomes harder to protect when responses are only reviewed after fieldwork has closed. By then, fraudulent, careless or inconsistent answers may already be mixed into the dataset and affect the results. Researchers then have to identify which responses should be removed, determine whether replacements are needed and repeat parts of the analysis. With fieldwork already complete, there may also be less time to investigate questionable responses or apply exclusion rules consistently. Issues that could have been addressed while the survey was live create more manual cleanup, delay reporting and reduce confidence in the final findings. ## **How to improve survey data quality at scale** If you want reliable survey data, quality needs to be built into the full research workflow, from survey design through data collection. There are three layers to improving survey data quality: - Design surveys that make accurate answers easier - Validate responses while the survey is live - Standardize quality controls across your organization ### Build quality into the survey design #### 1. Start with the right sample Reliable survey data starts with defining who needs to answer the research question before even writing the questionnaire. Identify the demographic, behavioral or category-specific criteria that make someone relevant to the study, then decide how that audience will be reached. Don’t treat [sample size](https://www.askattest.com/blog/articles/how-to-determine-sample-size) as the only measure of quality. Two thousand responses from the wrong audience are less useful than a smaller sample of people who genuinely match the research criteria. Before launch, work backward from the analysis you plan to run. Identify the audience groups you will need to compare and make sure each one will have enough respondents to support meaningful conclusions. Use [quota sampling](https://www.askattest.com/blog/articles/quota-sampling) where particular subgroups need to be represented. Finally, check that the audience is still feasible once targeting criteria, quotas and qualifying questions have been applied. Combining too many narrow requirements can make some groups difficult to reach or leave you with too few responses for reliable analysis. #### 2. Keep answer lists focused Long lists of answer options in “select all that apply” questions can encourage respondents to choose more than genuinely apply to them. Where appropriate, ask them to select their top three answers instead. This encourages prioritization and gives you a clearer view of their strongest preferences. The answer list itself should also stay focused. For example, a brand awareness question should include the brands you need to evaluate rather than every name in the category. #### 3. Randomize answer options One of the best ways to avoid question bias is to randomize the order of your answer options. That way, the answer that was first on the list for one respondent won’t be first on the list for another. This cancels out a respondent’s propensity for picking the answer that appears first. #### 4. Make sure your questions are detailed Questions should give respondents enough detail to understand exactly what you mean. Vague wording leaves room for interpretation and makes answers harder to compare. For example, “Have you driven very far recently?” could mean different things to different people. “In the past three months, have you driven more than 100 miles in a single journey?” gives respondents a clear distance and time frame to work from. #### 5. Include ‘none’ and ‘other’ as answer options Answer options should reflect the range of experiences respondents may have. When no option fits, people may choose the closest answer simply to move on, which impacts data quality. Include “none” or “other” in answer lists where they represent a valid response rather than forcing everyone to select from the options provided #### 6. Avoid yes or no questions Yes-or-no questions can be too simplistic when you’re asking about opinions or experiences. They may also introduce agreement bias, where respondents are more likely to answer “yes” because they assume that is the expected response. Where the topic allows for more nuance, use a [rating scale](https://www.askattest.com/blog/articles/survey-rating-scales) or a [multiple choice question](https://www.askattest.com/blog/articles/multiple-choice-survey-questions) instead. #### 7. Use qualifying questions Demographic targeting can help you reach a broadly relevant audience, but it may not confirm that respondents have the experience your study requires. Add a qualifying question at the start of the survey to check for the behavior, knowledge or category experience that makes someone eligible. For example, a pet food study might target dog owners, then ask whether they have bought dog food in the past three months. Choose which answers qualify respondents to continue and screen out anyone who doesn’t meet the criteria. #### 8. Avoid leading questions Leading questions suggest the answer you expect or make one response seem more desirable than another. This can push respondents toward your preferred outcome rather than capturing what they genuinely think. Review each question for assumptions or wording that points toward a particular answer. For example, replace “How much did you enjoy our new campaign?” with “What did you think of our new campaign?” It’s also a good idea to have someone who is less invested in the outcome review the survey before launch. ### Build a better survey from the start Use Compass to turn your research goal into a survey draft, improve question wording and catch potential issues before launch. [Learn more about Compass](https://www.askattest.com/compass) ![](https://emx2zzfzxax.exactdn.com/wp-content/uploads/2026/06/Group-48095923-1-1024x1011.png?strip=all) ### Validate responses during fieldwork Strong survey design gives respondents the best chance of answering accurately, but it cannot prevent every careless or fraudulent response. The second layer of quality control is to review incoming data while the survey is still live, so you can identify suspicious responses before they reach the final dataset. #### 1. Use automated checks to identify suspicious behavior Manually reviewing every response is difficult once a survey reaches hundreds or thousands of people. Automated and AI-enabled checks can flag responses that may need closer review as they come in. For example, at Attest, we look at several types of activity: - **Technical patterns:** Anti-bot measures, duplicate response detection, digital fingerprinting, geolocation checks and device analysis can identify repeat participation or responses that do not appear to come from the expected audience. - **Response behavior:** Unusually fast completion times, straight-lining, excessive non-answers and overclaiming may suggest that someone is rushing or not engaging properly. - **Answer consistency:** Impossible demographic combinations or contradictions between different answers can show that a response does not add up. - **Response quality:** Failed attention checks and suspicious open-text answers can reveal careless, copied or potentially AI-generated responses (more on this below). No single warning sign proves that a respondent is fraudulent. Someone may complete a familiar survey quickly or fail an attention check by mistake. Quality decisions should be based on the overall pattern across the response, not one unusual data point. Running these checks during fieldwork means poor-quality responses can be reviewed, removed and replaced before the survey closes. This gives teams a cleaner dataset to analyze and reduces the amount of manual QA required after data collection. #### 2. Assess the quality of open-text responses [Open-ended questions](https://www.askattest.com/blog/guides/open-ended-survey-questions) are vulnerable to low-effort, copied or AI-generated responses. Some answers may be obviously irrelevant or nonsensical. Others may look polished while saying very little about the respondent’s actual experience. When reviewing open-text responses, check whether each answer: - Addresses the question directly - Provides specific and relevant detail - Is consistent with the respondent’s other answers - Repeats language or sentence patterns found across several responses - Sounds polished but remains overly generic AI-enabled checks can help flag these patterns across large volumes of responses. Human review is still important for borderline cases, especially now that synthetic answers can appear plausible at first glance. Together, automated review and human oversight make it easier to separate genuine feedback from answers that should not enter the final dataset. #### 3.Include attention checks Attention checks help identify respondents who are clicking through the survey without reading each question carefully. They usually ask people to follow a simple instruction, such as selecting a specific answer from a list. Attest uses this type of check to assess engagement, particularly in longer surveys where attention may decline. Use attention checks sparingly and make the correct response unambiguous. They should be easy for an attentive respondent to answer, rather than designed to trick people or make the survey feel adversarial. A failed check should also be considered alongside the rest of the response rather than treated as automatic proof of poor-quality participation. #### 4. Remove and replace poor-quality responses before fieldwork ends It’s important to check responses while the survey is still live. That way, poor-quality responses can be removed and replaced. This helps ensure the final dataset contains the number of valid responses originally planned, rather than leaving researchers to collect replacements after fieldwork has closed. It also reduces the amount of manual cleaning needed before analysis can begin. Suspicious patterns can be identified earlier, before they affect more of the dataset, which makes delays to analysis and reporting less likely. ### **Standardize quality controls across teams and markets** The third layer of improving survey data quality is standardizing how surveys are designed, reviewed and monitored as more people run research across different teams and markets. One well-designed survey isn’t enough if other teams follow different processes or change the methodology over time. Build the following controls into the way every survey is run: - **Use approved templates for common research workflows.** This gives teams a reliable starting point and means they aren’t rebuilding surveys from scratch. - **Standardize the core methodology**, including question wording, answer scales, screening criteria and routing logic. This makes it less likely that changes in survey design will affect the results. - **Document survey requirements**, like target audience, sample sources, quotas, validation checks and exclusion rules. Everyone involved can then follow the same criteria. - **Assign clear ownership** for reviewing surveys before launch and monitoring response quality during fieldwork. This reduces the risk that important checks will be missed. - **Track recurring quality issues** across markets and panel sources. This helps teams identify where problems are happening repeatedly and take action. - **Preserve the methodology in recurring studies** so results remain comparable across markets and survey waves. Standardization doesn’t have to slow research down. Shared templates and clear decision rules reduce rework and manual cleanup. Teams can move faster without losing confidence in the data. ## **How Attest helps teams collect reliable survey data at scale** Improving survey data quality requires clear survey design, ongoing response validation and consistent controls as research scales. Attest supports teams at each stage, from survey design through fieldwork, so they can prevent avoidable issues before launch and address poor-quality responses before data collection ends. Before launch, [Compass](https://www.askattest.com/compass) helps turn a research goal into a survey draft. It can refine questions, recommend suitable answer options and flag issues such as biased wording or typos. This helps teams prevent avoidable design problems. Once the survey is live, [Attest protects data quality](https://www.askattest.com/blog/attest-news/raising-the-bar-on-data-quality) through three connected layers: - **Automated and AI-enabled checks** identify signs that a response may not be genuine or reliable. These include attempts by bots to complete the survey, impossible demographic combinations, geolocation mismatches, repeat participation and unusual device patterns. Attention checks, AI-powered open-text review and behavioral analysis also flag respondents who may be rushing or answering dishonestly. - **Human review and oversight** add researcher judgment where automated checks aren’t enough. Attest’s customer operations team manually reviews open-ended responses and spot-checks quantitative data while a survey is running. - **Deep integration with panel providers** gives Attest access to hundreds of respondent sources. Underperforming panels can be removed without limiting the team’s ability to reach the audience it needs. Attest also combines the fraud detection and respondent validation used by panel providers with our own independent checks. Together, these three layers allow Attest to protect data quality throughout fieldwork, rather than leaving QA until the survey has closed. Responses that don’t meet Attest’s standards can be removed and replaced before data collection ends, so teams reach their planned number of valid responses with less manual cleanup before analysis. With quality controls built into survey design, respondent sourcing and fieldwork, Attest helps teams scale research without shifting the QA burden onto researchers once data collection is complete. The result is reliable survey data that teams can use with greater confidence. ### Own every insight with data you can trust See how Attest combines smarter survey design, live quality checks and human oversight to help you collect reliable consumer data at scale. [Explore Attest’s data quality approach](https://www.askattest.com/data-quality) ![](https://emx2zzfzxax.exactdn.com/wp-content/uploads/2026/06/Group-48095923-1-1024x1011.png?strip=all)