Creating online surveys, to put it mildly, is a test of optimism, perseverance, and character!
Tools like SurveyMonkey are known for their ease of use. Anyone can create a simple survey within minutes. That is, if you have just a few questions to throw together.
When working with 100+ questions, it’s a completely different story. For the last few days I have been wrestling with the tool Qualtrics our school uses as part of an evaluation trial. Here’re some of the issues I struggle with – I’d like to hear what others use, whether you experience similar problems, whether you have figured out solutions around the problems, and what are the best tools you’ve used so far.
- Server-side processing means minimal risk of data loss, real-time updates, and lots of other good things. However, loading time increases exponentially as more and more questions are added to the survey
- I’m at question 64 now and it takes almost 3 seconds to reload after each action – 3 seconds to copy, 3 seconds to enter the edit mode, 3+ seconds to edit (with delays in loading multiple tabs of the edit page), 3 seconds to save and get out of edit mode, 3 seconds for the screen to refresh and reload questions 1-64 from the server, and another 3+3+3+=12 seconds down the drain if I made a silly mistake and had to go back in and correct it.
- Want a page break? What about an introduction to a section? However moving questions around? Every nicety comes with a price tag (measure in reload time)
- No batch processing capability
- When I discovered font inconsistency in some of the questions, I have no choice other than correcting each of them individually – which means 3+3+3+3=12 seconds for each font correction IF I corrected all errors in one trip.
- When I need to copy 30 out of 60 questions from one block to another, there’s no way to delete 30 questions in a batch. Of course each delete triggers the reloading of the rest of the questions from the server. I wish someone warned me about this ahead of time so that I could avoid the chore of having to delete a large number of questions!
- No importing capability
- Anyone who has naively budgeted 2 hours for creating an online survey (aka me) should seriously consider increasing their budget by 1000%. With Qualtrics, 20 questions is my maximum daily workload. Go over the daily limit and face the serious risk of making multiple errors (e.g., forgetting to type a question mark at the end of a question) and having to take even more time to correct them.
OK. You’ve been warned now. If you still decide to do a Qualtrics survey, you know what you’re getting yourself into!
Filed under: Research
Wow Rachel, it sounds like Qualtrics is a bear! Best of luck with it.
I’ve used SurveyMonkey, and it’s worked reasonably well for me. SurveyMonkey recently revamped their interface, making it much easier to upload and edit questions and preferences. It’s been a few months since I dealt with SurveyMonkey (so I’ve forgotten the details), but I can tell you that it is much easier to handle than Qualtrics (based on Rachel’s comments). My survey had ~140 questions, yet even at that number SurveyMonkey didn’t really present a hassle in terms of editing and load times.
Hi, Rachel,
Thanks very much for warning us away from Qualtrics.
I’ve used an account through my school on Washington State University’s CTLSilhouette (a.k.a. Flashlight Online) for survey creation. You can’t upload questions, but creating them online isn’t bad, and once you’ve done that, you can take the code and edit it to your heart’s content to control presentation. There’s a FAQ at the URL I’m listing. It talks about a maximum file size for the code. I’ve had surveys feed one to the next – instead of a thank you page, you get the first page of the next survey – so it’s an easy limitation to get around.
In the past, our IRB wasn’t fond of having subjects’ confidential data on commercial sites, but they didn’t question WSU as survey host.
I’ve heard of using Survey Monkey’s question skip logic for quasi-random assignment to treatment conditions. Pretty slick. Last time I used it, CTLSilhouette did not offer that.
Happy surveying.
Sorry, that URL is http://ctlsilhouette.ctlt.wsu.edu/ctlsilhouette2_5/
Actually, I have used Qualtrics on my laptop and have never had that problem with load time. In fact, it is the best survey product that I have ever seen when it comes to functionality. Survey Monkey is easy to use, but you lose valuable functionality. Qualtrics is also very easy to use, but is very powerful and very flexible.
Bryce, I think you’re my Qualtrics sales rep….
Rachel,
Hey! I actually am really glad to get your feedback about what improvements can be made. I actually have made several requests for batch processing to be implemented, so I can imagine that in the near future we will have that capability. Also, I do have an account in Survey Monkey…and I just dont think that it has the flexibility to do any extremely advanced research. Yes, it is very easy to use which is nice, but it all depends what you are looking for I guess. If you have any other feedback for me, please let me know and I will see what I can do to help you out. Thanks!
Hi Rachel,
I use Prezza Technology’s Ultimate Survey Tool… and I think it works reasonably well… If you would like I can send you soft copy of its user manual. You can export the survey questions in XML and also import in XML (and other) formats. There is possibility of branching and looping etc.