Exploring ORMs | Technical guide | Choosing software for online questionnaires

Introduction

Online questionnaire software is designed to make the process of producing the HTML and necessary scripts possible for researchers with little or no technical knowledge. It generally provides a form-style interface to allow the questionnaire to be built up through inputting the questions and response types, and selecting pre-programmed features such as required responses, skip patterns, randomised ordering of choices, and resubmission checks.

There is a huge range of options available. At the time of writing, the WebSM searchable database of online questionnaire software and services has 408 entries categorised according the following:

Type (software or service);
Code availability of software (open or closed source);
Charges;
Language;
Country where offices are located.

The Association for Survey Computing searchable software register also lists 79 examples of software that can generate online questionnaires. A simple Google search reveals many more examples of online questionnaire software with a vast range of features and prices.

This page will offer examples of the different types of software available and consider the different features and levels of sophistication commonly offered. It will also explore the factors involved in making a choice of software, allowing you to develop a checklist for use when comparing different options.

NB. Any references to prices or features are correct at the time of writing, but should be seen only as a snapshot of what is on offer in April 2005. References to particular software providers are given only as examples of the options available and should not be seen as an endorsement of any particular products or services, nor as a guarantee of quality.

Software plus hosting

For researchers without suitable access to a server, or for those who prefer to avoid installing and maintaining software, a wide range of hosting services are available. In the majority of cases, the whole process of producing and implementing the questionnaire is carried out online using a form-based interface. Many companies that offer 'software only' options also tend to have hosting facilities which registered users can take advantage of.

There are a great many companies offering services targeted at all levels of users from those offering design, hosting and analysis facilities for individual researchers for free, to those offering solutions to multinational companies for thousands of pounds. Many offer a variety of products targeted at different users with special rates for small-scale individual use. Others offer educational rates or even free services for researchers who are prepared to fulfil certain conditions such as having a link to the company on the institution website, and being prepared to contribute their survey questions to libraries of question types for reuse.

If the researcher decides that this type of service is suitable for his or her study, a number of questions must be considered when choosing a service. The aim should be to find an appropriate balance between cost and features required.

Cost

The cost of the services is often dependent on the number of respondents, questions and or surveys. Most charge a fixed monthly or annual fee, often setting limits on the number of completed surveys and charging additional fees for each survey beyond this maximum.

Many offer different levels of service, at different prices, usually with names such as 'basic edition', 'standard edition' or 'professional edition'. This generally involves differences in the maximum number of surveys allowed and differences in the levels of technical support offered. It may also involve additional features such as more advanced analysis and reporting tools, and more sophisticated question types and questionnaire options.

Given the competitiveness of the market, the majority of the services offer clear pricing information which can be linked directly from the home page. This tends to allow relatively easy comparison of different providers.

Available features

Many of the software company websites offer product demonstrations, tables of features, example surveys, and/or the creation of free demonstration surveys to 'sell their wares'. This makes it relatively easy to check the features that are available, but can also quickly lead to information overload.

A useful example survey which includes comments on the features illustrated by particular questions can be found on the demo pages of the SelectSurveyASP service at http://www.classapps.com /SelectSurveyASPDemo.asp (Note that this software does not offer a hosting service).

Surveyz is one of many options that offer a fully functional trial of their software and hosting service which allows the user to quickly get a feel for the functionality and options available. Registration is required to activate this trial and the trial is restricted to a one-month period, one questionnaire active at any one time, and a total of 25 responses. http://www.surveyz.com/

It is a good idea to create a checklist of the features that you are likely to require. This can then be compared to the features offered by different providers. It can be particularly useful in enabling a minimum level to be established where a provider offers increasing levels of service at different prices.

The 'creating a checklist' activity below will guide you through the process of developing such a checklist according to the needs of your study.

Examples of services available

The following examples are chosen as being representative of some of the different types of services available. In each case, a range of comparable options may be available.

Service	Comment
Bristol Online Surveys	Targeted at institutions requiring the option to have a number of different surveys and survey administrators. Highly customisable to the style needs of institutions including an option to run surveys on their server with an address that appears to be that of the institution.
Globalpark	Provides'EFS Survey' software which offers relatively advanced features and a high degree of flexibility. Also makes the software available to academic researchers for a nominal fee through unipark providing excellent value for these users where advanced functionality is required.
SurveyConsole / QuestionPro	Both are divisions of the surveyanalytics company and they use the same software and interface, but with different pricing. May offer sponsored use for academic or not-for-profit projects if certain conditions are met. Also offer a range of free resources such as articles and question templates.
surveymonkey	Compares well with many of the other available services in terms of features, but is one of the cheapest commercial options.
surveywriter	Relatively expensive, but unusual in that charges are not made per period of use, but per completed survey and email invitation with a minimum of 200.
Surveyz!	A range of relatively sophisticated features. Targeted at individual researchers or at institutions. Offers academic pricing and free use for academic projects if certain conditions are met. Also has a range of resources such as articles on online questionnaires and copyable templates of questionnaires and questions.
websurveyor	Relatively expensive, but with a wide range of features. Offers both hosting and software only solutions.
zoomerang	Offers pricing for not-for-profit and educational institutions. Also offers a range of research services such as questionnaire administration, panel services and translation.

Software only

The use of software-only solutions depends on the researcher having access to a suitable server with adequate administration permission to, for example, create and access a database. It also requires the software to be installed and configured correctly and for any possible problems that might emerge to be dealt with. Where these requirements can be met, this type of software can have a number of advantages. These may potentially include:

Savings of cost, especially when used in the longer-term;
Increased control over data collection and storage;
More opportunities to work offline if required.

Some examples of this type are also primarily designed as survey analysis software which can offer generally more sophisticated analysis tools where required.

Open source options

Locating software

Two of the major sources of information on Open source software are http://sourceforge.net which is a repository of open source projects, and http://freshmeat.net/ which is a listing of new software releases.

A simple search for 'survey' reveals a range of projects providing software for online questionnaire creation. These range from simple survey software with limited question types and functionality, to options with features which compare very well with the majority of commercial options.

Choosing options

When choosing whether or not to use an Open source solution, it is important to consider the following potential problems:

They do not tend to offer hosting, and installation and configuration can be complex (though this may be less problematic than with commercial software-only alternatives);
The user-interface for the software can be more complex in some cases;
Documentation such as user-guides may not be as extensive;
Where problems are encountered, suitable technical support may not be guaranteed.

Making a successful choice depends on identifying whether or not the software provides the features required. The 'creating a checklist' activity below will guide you through the process of developing a checklist according to the needs of your study which can be used when comparing software. In many cases the information on the features available is less extensive than commercial options, though many of the more established options offer demonstrations, examples and feature lists.

It is also necessary to ensure that the software can be installed and configured on a suitable server, and that someone with the necessary skills is available to maintain it successfully and deal with problems. An indication of what is typically involved in this is given by the administrator's manual for Lime Surveys, which can be found by following the 'Documentation' linkat http://www.limesurvey.org/ . Many of the other open source options also have similar installation guides which can be accessed through the project website or downloaded with the software.

Finally, it is important to check that the software has an adequate level of reliability. The JISC open source software advisory service, OSS Watch, at http://www.oss-watch.ac.uk/ offers useful guidance on criteria for choosing reliable software. Some key factors identified are:

Reputation - Have you spoken to people with experience of a particular product?
Ongoing effort - Is there clear evidence of active development of the software?
Community support - Is there an active community of users on the project mailing list ready to answer questions from users experiencing problems?
Version - Has the latest stable version been available for some time and is there evidence that problems have been identified and fixed?
Documentation - Is this sufficient to allow you to decide whether or not the software is sufficiently well-developed for your purposes?

See the page of 'Top Tips For Selecting Open Source Software' at http://www.oss-watch.ac.uk/resources/tips.xml for further details.

Examples

The following are some of the main examples of open source software for online questionnaires (generally the more established and/or sophisticated options). A range of other options may be available.

Software	Comment
LE Survey	Designed as a tool for running questionnaires as part of longitudinal studies. Allows respondents' responses to be matched to responses to previous questionnaires while maintaining confidentiality. In early stages of development at the time of writing.
phpESP	Well-established software with a working demo available allowing the features and user-interface to be tested. Offers low cost hosted services.
LimeSurvey	A range of relatively sophisticated features. Well-established with useful documentation. Working demos are available allowing the features and user-interface to be tested.
Mod_Survey	Well-established with sophisticated features such as dynamic content generation depending on previous answers. Requires the user to learn to use XML syntax particular to the software.
VTSurvey	Easy to use and particularly useful for straightforward questionnaires as only the four main types of questions are supported (Multiple choice with radio buttons and Check boxes, and short and long text entry boxes).

Commercial options

Choosing options

Documentation and technical support for commercial options tends to be more extensive than that offered by open source software, but in many cases there are charges for a support package. They frequently offer powerful statistical analysis, and many can be used to develop and analyse questionnaires off-line. In the majority of cases, hosting services are offered for an extra cost.

A number of the commercial options were originally designed as software to aid in the production and analysis of questionnaires in paper-based and other modes, often with online questionnaire functions added later and offered at extra cost. Others offer online questionnaire functions as part of a set of 'add-ons' allowing, for example, computer-assisted telephone interviewing (CATI) or surveys for Personal Digital Assistants (PDAs) or other mobile computing technologies. This can make them a particularly effective choice when there is a need to integrate mixed-mode surveys.

It is useful to list the features required for your study which can be used when comparing software. The 'creating a checklist' activity below will guide you through the process of doing this, providing you with some of the main questions to consider.

Examples

The following are some examples of commercial software for online questionnaires. A range of other options are available.

Software	Comment
ProntoSurvey	Shareware tool from Bebosoft which can be downloaded for Mac as well as windows. Relatively inexpensive and straightforward to use with clear and extensive documentation and tutorials.
Questionmark Perception	An educational assessment tool which offers many of the key features needed to create basic online questionnaires and has some of the more advanced features such as randomisation of questions and conditional branching. Potentially useful option if the software is available through the researcher's institution.
SelectSurveyASP	Relatively inexpensive. Offers 'classic' and 'advanced' versions with different levels of features at different prices. Has a working online demo and a useful example survey which includes comments on the features illustrated by particular questions. Offers a free installation service and free technical support.
Snap Surveys	Extensive options for mixed-mode surveys, offering a 'core product', Snap Professional, with add-ons for questionnaires via internet and PDAs, and to allow scanning and multiple data entry. Expensive example of 'high-end' options.
StatPac	The online questionnaire software does not include analysis tools, but it can be purchased alongside the statistics tools offered. Has basic statistical tools or an advanced version allowing multivariate statistical techniques. Technical support and updates are available free for three months, but are chargeable via annual support/maintenance agreements thereafter. A fully-functional version of the software can be downloaded and used for free, limited to 35 respondents for each survey. Download includes tutorials and extensive user guide.

« BACK UP NEXT »

	Access control through passwords or by limiting access to those from particular IP addresses.
	The option to require a response to certain questions.
	Randomisation of the order of questions or responses within questions.
	Extra design features on individual pages, such as images, extra text or line breaks.
	Features such as images, headers or footers that can be applied to all pages.
	Skip patterns so that the participant is directed to alternative questions depending on responses to particular questions.
	The inclusion of multimedia stimuli such as video and animation.
	The inclusion and placement of a progress bar.
	The facility to enter a closing date for the questionnaire and add a customisable message to those who attempt to access it once closed.
	The option of receiving the results by email alongside saving them to the database.
	The option of linking to a webpage of your choice once the questionnaire has been submitted.
	The option of adding cookies to the participant's computer to prevent multiple submission from the same machine
	Control over the use of cookies to allow particular machines to make multiple submissions (e.g. where a suite of computers may be used by different participants, or for data entry from paper-based versions)?

Choosing software for online questionnaire production

Introduction

Software plus hosting

Cost

Available features

Examples of services available

Software only

Open source options

Locating software

Choosing options

Examples

Commercial options

Choosing options

Examples

Activity: Creating a checklist

Create personalised checklist

View all checklist questions

General

Question creation

Design and layout

Questionnaire settings

Implementation and security

Distribution

Results and analysis