“Test to break” an online survey prior to launch
Purely technical walkthroughs prior to launch
As one of the staff who support Qualtrics at work, I am always suggesting that the researchers and staff put their survey through its paces. Many will, particularly if their surveys are straightforward. They will try out the survey on different web browsers and ensure that all the functionality that they built in works as advertised.
The surveys that need more attention are those that use branching logic, embedded data, display logic, and more complexity. In these, every “state” has to be tested. If authentication is used, then the system has to be tested in terms of how it behaves with other systems and databases. Likewise, if there are third-party tool integrations (beyond those built into the Qualtrics instance), those have to be tested.
It helps to test on mobile devices, given that many will be using mobile devices to respond.
The settings on a survey can be misunderstood. Some settings may cancel out other settings. Sometimes, people set up full anonymity on a survey and do not realize that they will also lose out on IP addresses and latitude-longitude data. To this end, it helps to use faux data and real human-input data to see what it looks like on the back end, to make sure that the research questions can actually be answered.
Others do not realize that a survey may not be partially recorded based on their settings. Some do not realize that the survey records partially at the end of each page. I have seen surveys where there were many questions…but only one page break at the very end. If people do not correctly complete, the researcher is left with nothing in terms of partial data.
Another misconception is that particular data is collected when it is not. Many assume that personal identifiers are available even if there are no conscious fields collecting such data.
Sometimes, users will assume that a tool is more powerful than actually is true, such as the accessibility tool. The built-in accessibility tool points to questions that require the use of a mouse to respond to. Input devices to computers such as puffers and eye-blink devices require access to the keyboard shortcuts in order to function. There is not an equivalent input to using a USB or wireless mouse. These (matrix questions, slider questions, drag and drops, side-by-side questions, “pick, group and rank” questions, and others) are inherently inaccessible. The tool does not identify inaccessibility in inline-framed (iframed) videos for the survey which do not have closed captioning; it does not identify images without alt text. Understanding the limitations of particular features is important to use a tool well.
Even after a survey has gone live, it helps to stay atop the survey because changes on the cloud-based online survey system (such as removal of various modules) can affect the availability of the survey. Not all functionality works as advertised either, such as dates in the “embedded data,” used in relation to > and < .
Testing to break
When I ask people to “test to break” a survey, I mean that I want them to do everything in their power to “break” the survey from the front end. I want them to find every weakness in it instead of coddling it and wanting to make sure it “works.” Few people are willing to do this because they just assume that the survey they made will work as they conceptualize.
At a university, there are mission-critical trainings that go out. There are surveys sent out to hundreds of thousands of people. There are longitudinal research works with years and years of data. It really makes a difference to “test to break” early on, before anything goes broadly live. Problems may not be seen in many cases until people try to use the survey...from various devices...with various web browsers. The Preview simulation also is not sufficiently revelatory about what is working and what is not because it is only a light simulation. After the breakages are found, the fixes should be emplaced...and another round of testing should follow...until it all functions as designed, developed, and expected.
Testing for legal, ethical, original, and relevant
Table 1: Testing Online Surveys for Quality in the Common Phases of Research (Stratified by Principals in the Research Space)
|
|
|
|
|
|
|
| ||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Conclusion
There is a number of excellent and eminently readable commercial books about how to build surveys that are efficacious. There are various methods to maximize the power of such research using computational analyses, including artificial intelligence (AI)-informed works, and the more traditional statistical analyses (even in the knowledge of the limits of self-reportage).
About the Author
| Previous page on path | Cover, page 14 of 23 | Next page on path |
Discussion of "“Test to break” an online survey prior to launch"
Add your voice to this discussion.
Checking your signed in status ...