Navigating the jungle of survey tools

Creating online surveys, to put it mildly, is a test of optimism, perseverance, and character!

Tools like SurveyMonkey are known for their ease of use. Anyone can create a simple survey within minutes. That is, if you have just a few questions to throw together.

When working with 100+ questions, it’s a completely different story. For the last few days I have been wrestling with the tool Qualtrics our school uses as part of an evaluation trial. Here’re some of the issues I struggle with – I’d like to hear what others use, whether you experience similar problems, whether you have figured out solutions around the problems, and what are the best tools you’ve used so far.

  • Server-side processing means minimal risk of data loss, real-time updates, and lots of other good things. However, loading time increases exponentially as more and more questions are added to the survey
    • I’m at question 64 now and it takes almost 3 seconds to reload after each action – 3 seconds to copy, 3 seconds to enter the edit mode, 3+ seconds to edit (with delays in loading multiple tabs of the edit page), 3 seconds to save and get out of edit mode, 3 seconds for the screen to refresh and reload questions 1-64 from the server, and another 3+3+3+=12 seconds down the drain if I made a silly mistake and had to go back in and correct it.
    • Want a page break? What about an introduction to a section? However moving questions around? Every nicety comes with a price tag (measure in reload time)
  • No batch processing capability
    • When I discovered font inconsistency in some of the questions, I have no choice other than correcting each of them individually – which means 3+3+3+3=12 seconds for each font correction IF I corrected all errors in one trip.
    • When I need to copy 30 out of 60 questions from one block to another, there’s no way to delete 30 questions in a batch. Of course each delete triggers the reloading of the rest of the questions from the server. I wish someone warned me about this ahead of time so that I could avoid the chore of having to delete a large number of questions!
  • No importing capability
    • Anyone who has naively budgeted 2 hours for creating an online survey (aka me) should seriously consider increasing their budget by 1000%. With Qualtrics, 20 questions is my maximum daily workload. Go over the daily limit and face the serious risk of making multiple errors (e.g., forgetting to type a question mark at the end of a question) and having to take even more time to correct them.

    OK. You’ve been warned now. If you still decide to do a Qualtrics survey, you know what you’re getting yourself into!


4 Responses

  1. Wow Rachel, it sounds like Qualtrics is a bear! Best of luck with it.

    I’ve used SurveyMonkey, and it’s worked reasonably well for me. SurveyMonkey recently revamped their interface, making it much easier to upload and edit questions and preferences. It’s been a few months since I dealt with SurveyMonkey (so I’ve forgotten the details), but I can tell you that it is much easier to handle than Qualtrics (based on Rachel’s comments). My survey had ~140 questions, yet even at that number SurveyMonkey didn’t really present a hassle in terms of editing and load times.

  2. Hi, Rachel,

    Thanks very much for warning us away from Qualtrics.

    I’ve used an account through my school on Washington State University’s CTLSilhouette (a.k.a. Flashlight Online) for survey creation. You can’t upload questions, but creating them online isn’t bad, and once you’ve done that, you can take the code and edit it to your heart’s content to control presentation. There’s a FAQ at the URL I’m listing. It talks about a maximum file size for the code. I’ve had surveys feed one to the next – instead of a thank you page, you get the first page of the next survey – so it’s an easy limitation to get around.
    In the past, our IRB wasn’t fond of having subjects’ confidential data on commercial sites, but they didn’t question WSU as survey host.

    I’ve heard of using Survey Monkey’s question skip logic for quasi-random assignment to treatment conditions. Pretty slick. Last time I used it, CTLSilhouette did not offer that.

    Happy surveying.

  3. Hi Rachel,

    I use Prezza Technology’s Ultimate Survey Tool… and I think it works reasonably well… If you would like I can send you soft copy of its user manual. You can export the survey questions in XML and also import in XML (and other) formats. There is possibility of branching and looping etc.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: