Four Things We Learned From Talking to Panellists


Tags: Crowdologypanelquestionnaire design

This year (2015) we were once again given the opportunity by panel review website SurveyPolice to conduct a question and answer forum. The aim of this was primarily to allow panellists to ask questions about online panels, surveys and market research and to provide them with insight from the perspective of the panel owners (us!). However, we also recognised this as a fantastic opportunity to understand the needs and concerns of panellists, which we hoped would enable us to improve our surveys and service.

So, what did we learn?

  1. Panellists dread grid questions
  2. Screening questions are not always placed appropriately
  3. Panellists are genuinely interested in the companies behind the panels
  4. There is growing discontent with the mobile accessibility of surveys


1. Grid Questions

OK, so this wasn’t really new, we know panellists dislike grid questions, but I don’t think we fully understood the dread that these can foster in people. Common complaints were:

  • Difficulty in distinguishing between the lines
  • Not being able to see the option headers
  • Length of grids

Several mentioned that if they were faced with a long grid they would quit the survey, while others mentioned they are aware that they don’t always give the best quality answers due to their frustrations and the inaccessibility.

During the week this was probably the most mentioned topic/issue, and as such we turned the Q&A on its head and asked two questions ourselves:

  1. Which style of question is preferred?
  2. How might grid questions be improved?

We were incredibly appreciative of how forthcoming and detailed responses were and have picked out some of the suggestions here. Not only will Redshift Research be paying heed to these, but we will also be actively passing on this knowledge to those that we work with.

General improvements for all questions:

  • Ensure instructions are repeated if continued onto a new page
  • Character counters for open ended questions
  • Example responses for open ended questions
  • Alert respondents at the beginning if they will be required to view videos

Suggestions for grid improvement:

  • Alternating row colours
  • Column headings after every few rows
  • For very long grids – split across pages and indicate how many pages of grids there are
  • Use alternative question styles:
    • Flashcards
    • Sliders

Clearly grids are a big concern for panellists and therefore should be so for market researchers due to their potential of increasing drop out rates and affecting data quality.


2. Screening Question Placement

A common complaint was that panellists are often screened out of surveys at a point that they find unacceptable. Some quoted 40-95% of the way through, others mentioned after 20 minutes of question answering. The lesson here is to double check where the screeners are – if they aren’t at the front either move or remove.


3. Interest in Market Research Companies

Perhaps we are just naïve, but we were quite surprised by the genuine interest in the market research industry, such as trends, qualifications, and the companies within it.Slide1

There seemed to be particular interest around the relationships between panels and market research companies (which can be as complicated as an East End soap!).

Time for more transparency perhaps?

4. Mobile Surveys

We all know about (and most likely are experiencing) the rise of the smartphone and how it has taken on the challenge of providing everything we need in one small, pocket-sized (or not so much) device. This extends to survey taking as well, as you might expect. But it seems that survey software isn’t always keeping up. Nor, in fact, are survey lengths.

What is needed to facilitate survey taking on mobile devices?

  • Fewer (or more accessible) grid questions
  • Shorter surveys
  • Responsive design


These are perhaps topics for another time but with regards to survey length there seems to be small currents moving in the right direction but we are still waiting for the tidal wave.

Finally, we would like to thank Survey Police for facilitating, and all those who participated in the forum. Here’s to the next one!

Posted in blog Tagged with: , ,

I’d just like to take this opportunity to express my immense gratitude to the entire team at Redshift for all their outstanding work over the past ten series of the show. I know that some of our requests have been quite trying at times, but no matter where we set the bar, you guys were always able to clear it by miles. So please could you pass my thanks onto your fantastic team.


Redshift’s Red Hot Tip

Questionnaire Design:

Keep the number of questions to a minimum to ensure highest possible data quality