A Quest for the Perfect Questionnaire 

A clear questionnaire is the backbone of a good survey. Without solid, thoughtful, and specific questions, your survey’s results will be loose and amorphous, lacking the context and solidity you need to convert those results into manageable data. This is why every good researcher knows that it’s the questions in their surveys that are the gateway to success.  

However, pursuing the perfect questionnaire is an ongoing quest that can sometimes seem to lack an end. Yet, despite the frequent ups and downs, it is still a quest worth partaking in. Industry trends are always in flux, and writing the right questionnaires for the right audiences is an often iterative process. But the more committed you are to the journey, the better equipped you’ll be to overcome obstacles and seize your own success.  

With this eBook, we’ll be walking you through some of the most common survey design mistakes, how they can affect your journey, and ultimately, how you can prevent them from delaying your quest for success. 

Common Survey Design Mistakes 

Surveys are one of the foundational cornerstones of the market research industry. Without them, gathering data on a specific population would be far more cumbersome than it already is, and the overall quality of work researchers produce would inevitably decline.  

If surveys are the cornerstones of research, then questionnaires are the cornerstones of surveys. Survey data is gathered based on how respondents interact with the questions on the survey. If your questions are flawed, however, then the answers your respondents provide you with will also be flawed. And as you can guess, flawed data leads to flawed research.

To help you produce the best questionnaires you can, here are some of the most common survey design and questionnaire mistakes you can make. By avoiding these kinds of unclear questions, you can better equip yourself with the knowledge and tools you’ll need to avoid them in your own research. 

UNCLEAR WORDS AND JARGON 

While a questionnaire can be written in a formalized and professional manner, you’ll still want to avoid using complex words or phrases that would needlessly complicate the information you’re trying to convey to a respondent.  

Additionally, unless you know for sure that your audience is educated in a specific field, try and avoid industry jargon, abbreviations, and acronyms. Your goal should be to keep your words and questions as short and as simple as possible in order to avoid any confusion or misinterpretations. 

UNCLEAR CONCEPTS AND UNITS OF MEASUREMENT

Be sure to clearly specify concepts and units of measurement. Some concepts are straightforward—such as several days or height and weight—while others are vaguer and can lead to different interpretations. For example, words such as “usually” or “often” do not have a specific measurement, and as a result, can be misconstrued. 

For example, if a respondent were to answer the question below, they might consider 1-2 days a week “Very often,” while another might consider 1-2 days a week “Not very often.”  

UNCLEAR OR INCONSISTENT SCALES 

Always use the same scale throughout your survey. If you start with a 5-point scale, for example, make sure any other questions with a scale are also using a 5-point scale. You also want to make sure similar questions use the same scale in the same order. If possible, verbally label all points on your scales and make sure the responses are balanced and match the question stem. 

ASKING BIASED QUESTIONS 

As surveys have grown in popularity and accessibility, the research industry has encountered a multitude of improvements that help researchers better generate and analyze their survey results. However, this growth has also come with an unfortunate side-effect, as surveys are now so accessible that coming across surveys with biased and misleading questions is more common than it should be. 

Biased questions aren’t only problematic in that they expose the personal preferences of the researcher, but they will also skew respondent data by persuading them—either implicitly or explicitly—to answer a certain question in a certain way. As you’re approaching your questionnaire, here are some of the types of biased and misleading questions you should try to avoid: 

LEADING QUESTIONS 

Surveys and questionnaires should never be worded in a way that may sway a respondent to one side of an argument or another. For example, asking the question “Is the school year too short” is a leading question, as you’re prompting the reader to interpret the school year as being “short.” 

A better way of wording this question would be: “How would you describe the length of the school year?” This way, you’re not pushing a specific interpretation onto the respondent, but rather allowing them to interpret the length of the school year themselves and then provide you with an honest answer. 

LOADED QUESTIONS 

When a question forces a respondent to answer—inadvertently or not—in a way that doesn’t fully or accurately reflect their opinion or situation, then they’re answering a loaded question. These questions are one of the leading causes of survey abandonment, as they can alienate respondents and make inhibit their ability to respond to your question. 

For example, asking someone “How much do you enjoy birthday parties?” implies that, whether it’s true or not, that the respondent enjoys birthday parties. Instead, try phrasing the question like this: “Describe your thoughts on birthday parties?” This approach allows the respondent to answer in a way that reflects their personal experience, which will provide you with an honest answer. 

DOUBLE-BARRELED QUESTIONS 

Another thing to pay attention to is the grouping of your questions. Not only will the order of your questions affect the context of the survey as a whole, but when done poorly, can inadvertently lead respondents to answer a certain way or scare them away entirely. 

“A double-barreled question asks about more than one construct in a single survey question.”1 For example, a double-barreled question can look like this: “How satisfied or dissatisfied are you with the comfort and size of your home?” 

When you ask a question that’s really two questions in one, then it becomes impossible to tell how the respondent is weighing the different elements in the question. If the respondent values comfort over the size of their house, for example, then their answer to the example above would be biased, and as such, misleading in your research.  

Instead, frame the question like this: How satisfied or dissatisfied are you with the size of your home?” This way, you’re asking a single question that can be easily answered and effectively included in your overall research. 

Pew Research says that2 “it is important to use the same question wording and to be sensitive to where the question is asked in the questionnaire to maintain a similar context as when the question was asked previously.” You want your questionnaire to ask questions that flow into one another in a logical and productive way.

Survey Design Do’s and Don’ts 

To help you prevent the common survey design pitfalls we’ve talked about, and design a questionnaire that’s effective, unbiased, simple, and easy-to-read, here are some core survey design do’s and don’ts to abide by. 

THINGS TO DO 

DO PROVIDE POLICIES 

Defining your survey’s policies is one of the first things you should communicate to a respondent before they begin answering any questions. Policies are meant to direct the actions and decisions of your research team, and more importantly, protect the respondents. When outlining policies for your questionnaire, you’ll want to consider things like: 

• Respondent anonymity or confidentiality 
• Who will have access to the collected survey data 
• How the respondent’s personal information, if collected, will be used 
• What is the intent of the survey 
• Who is conducting the research Survey policies are essential, as they hold the researcher accountable for their work and provide comfort to the respondents by informing them how their answers and information will be used. Failing to communicate your policies, however, can give the respondent the impression that you’re untrustworthy, which will keep them from interacting with you and your surveys. 

Communicating your policies can be explicit or implicit, but it must be done. Vague or confusing policies will only hurt you and your research, so always go the extra mile and make sure you’re being honest with your respondents. 

DO PROVIDE CONTEXT 

If you want to engage your respondents in the questions you’re asking them, you need to provide them with context. Without a contextual framework, respondents won’t be able to give honest and valid responses because they won’t be able to properly decipher your questions. 

 The way you introduce your survey, the way you collect feedback on the survey, and the way you communicate the intent of the survey are all important things that inform the context of a survey and provide the respondents with the scaffolding they need to provide valuable answers.  

However, establishing context can be difficult, since context itself can feel like a somewhat ambiguous term. Context, as defined by the Merriam-Webster dictionary,3 is “the parts of a discourse that surround a word or passage and can throw light on its meaning.” Another definition of context is “the interrelated conditions in which something exists or occurs.”  

You want your questionnaire to have a natural progression, and the context of your survey should inform the tone, wording, and style of questions you ask. The more clear and consistent your context is, the clearer and more consistent your respondent’s answers will be. 

DO TEST YOUR SURVEY 

When it comes to designing a clear survey or questionnaire, one of the most important things you can do is test it. Not just once either, but over and over again under as many situations as possible. You want to make sure that your questions are easy-to-answer, unbiased, context clear, intent honest, and the resulting data useful. 

This involves testing, and often lots of it. Let as many people as you can take your survey internally, gather all of their feedback, and use it to fine-tune your questions. This way, when you release it to your targeted audience you’ll know that it’s as complete and polished as possible.  

Releasing an untested survey, however, can be the setup for disaster. Not only do you run the risk of your respondents being confused by the survey itself, but if your questionnaire is unstructured or biased, your respondents will abandon it and be less likely to take a survey from you again in the future. 

THINGS NOT TO DO 

DON’T FORCE INCORRECT ANSWERS 

Make sure you do not force respondents to answer a question that either doesn’t apply to them or that they don’t (or can’t) know the answer to. Like we talked about earlier, leading and loaded questions can create a bias that will negatively affect the quality of your survey results. 

Even if you believe you provided all the options you need to, it’s still a good idea to include an “other” option, just in case you missed something. Additionally, including an “other” option is a good way to reduce a long list of answer options allowing you to exclude answers that will rarely be selected. 

There are also some questions a respondent might not know the answer to or might not be applicable to them. To avoid answer selection bias in those scenarios be sure to include a don’t know and/or none option(s). 

DON’T ASK TOO MANY OPEN-ENDED QUESTIONS 

Open-ended questions are a valuable way to capture a deeper understanding of the respondent. However, they’re also difficult to capture and analyze and can even lead to respondent fatigue and abandons. As a result, you should usually try and avoid open-ended questions whenever possible.  

While there are times when an open-ended question can be useful, like Vernon Research Group says,4 “most open-ended questions can be turned into quantitative questions with a few minutes of brainstorming.” The more clear and specific your questions will be, the happier your respondents will be, and the better the answers they provide you with will be. 

DON’T OVERSTAY YOUR WELCOME 

 Most people don’t mind taking a survey, but will quickly abandon it if they deem that it’s too long. While you don’t want to limit yourself so much that you inhibit the value of the answers you receive, in most situations a shorter survey will receive higher response rates.  

Your questionnaire should be long enough so the answers you receive are valuable, but not so long that anyone would be intimidated. For example, creating interactive surveys5 can be a great way to keep your respondent engaged, improve the quality of their answers, and ultimately, ensure that your surveys never overstay their welcome. 

Be Specific 

You can’t have a questionnaire without questions, and you can’t have a good questionnaire without good questions. When preparing to write the questions for your survey or questionnaire, B2B International6 recommends that you try thinking about the questions from the respondents’ point of view. 

Researchers usually know exactly what it is they want from a survey, but that objective can rarely be converted into a single, straightforward question. This is why it’s so valuable to consider the questions you write from the perspective of the respondent. If you were responding to your own survey, would you find it biased? Difficult? Overly long? Unclear? 

 If your answer is “yes” to any of those questions, then you need to fine-tune your questions further. This will help you make your questionnaire as specific as possible, something that your respondents—and the data they give you—will appreciate. 

SHORT AND SIMPLE IS THE KEY TO SUCCESS 

 In the same way that specificity will improve the quality of your respondent’s experience with your questionnaire, so too will brevity and simplicity. Most people don’t mind answering questions, but if your questionnaire is too long—or contains too many complicated, jargon-heavy questions— then your respondents may abandon it prematurely, costing you valuable data. 

It’s more important to fill a survey with simple, short, and easy-to-understand questions7 than it is to cram 50 questions worth of content into 10 questions. In the end, your respondents would rather answer a larger quantity of simple questions than a smaller amount of dense questions. 

Dillman and others⁸ believe that although a formalized and professional questionnaire is one way to establish credibility with respondents it may backfire if complex words or phrases and technical terminology are not understood by all respondents. The authors go on to explain that “Part of writing simple questions is keeping them short and to the point. The longer the question, the more information respondents must take in and process, and the higher the likelihood that they will not fully process or understand the question. Many complex words and phrases can be easily replaced by more generally understood terms.” (117-120) 

The longer the question, the more information respondents must take in and process; consequently, there is a higher chance they will not fully process or understand the question. 

CLEAR QUESTIONNAIRE CHECKLIST  

• Use simple words and avoid jargon  
• Provide context  
• Don’t ask leading questions  
• Ask one question at a time  
• State the unit of measurement  
• Use clear concepts  
• Make sure question and answer stems match  
• Verbally label answer/scale categories  
• Include all possible answer options 
• Make answer options mutually exclusive 
• Randomize responses if there is a concern of selection bias 
• Test your survey 

Completing the Quest 

Clarifying your questionnaire is a key and foundational step in making a strong survey. Questions are the backbone of a survey, after all, and as such deserve as much attention, care, and revision as they can get. 

If you want to get a high response rate, then you’re going to need to put in the effort to make sure that every question on your questionnaire is in clear alignment with your survey’s goals, is easy to read and answer, and ultimately, is able to provide you with the kind of answers and data you need.  

When you familiarize yourself with the common survey design mistakes we’ve outlined, and take to heart the solutions we’ve provided, then you can capably and confidently design a survey that won’t only be able to sidestep mistakes but will also be able to extract the best data from your target audience. 

WHAT IS JIBUNU? 

The best in… 

• Survey Programming
• Technology Consulting
• Custom Tool Creation
• Qualitative Tools
• Process Automation
• Collaboration 
• Much More 

WHAT WE DO 

QUANTITATIVE: Complex logic and branching, trackers, segmentation, patient charts, concept testing, discrete choice, shopping exercises, custom-built tools, and much more! 

QUALITATIVE: Interview flow programming, secure content hosting, interview scheduling, interactive exercises and process automation. 

SUPPORT SERVICES: Consultation, process analysis and enhancement through technology, broadcasting, product photography, incentive fulfillment, coding, data analysis, sample procurement, and translation. 

Sources 

Sage Research Methods – Double-Barreled Question 

Pew Research – Questionnaire Design 

Merriam-Webster Dictionary – Definition of Context 

Vernon Research Group – Survey Design Best Practices: Part 2 

Jibunu – The Ultimate Guide To Creating Interactive Surveys (Including 25 Examples) 

B2B International – Chapter 8: An Introduction to Questionnaire Design 

Don A. Dillman – Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method (4th Edition) 

Don A. Dillman – Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method (4th Edition) (pages 117-120)