Exploring online research methods - Incorporating TRI-ORM

Choosing software for online questionnaire production

[Skip instructions]

[i] Click on the headings to open them. They will open on this page. Open the following link for further information about these headings if required.

Your browser does not support these headings. To ensure that the contents remain accessible, they have been automatically opened so that all the information on the page is displayed.

However, to take advantage of the headings and to ensure that the layout and design of this site are displayed correctly, you are recommended to upgrade to a current version of one of the following standards compliant browsers:

Glossary links are included within this page. If a word appears as a link, clicking on this link will show the definition of the word in a 'pop-up window'. Select the following link for information about these glossary links if required.

  1. Select the links see the definitions in a pop-up window.
  2. NB. If you use pop-up window blocking software, you will need to deactivate it for pop-ups on this site to use the glossary links. Alternatively, all glossary definitions can be seen on the 'Glossary' page in the 'Resources' section.
  3. Use of the glossary links is JavaScript dependent. If JavaScript is disabled, it will be necessary to open the 'Glossary' page to view the definitions. Opening this page in a new window may allow you to refer more easily to the definitions while you navigate the site.

Open/close headingIntroduction

Online questionnaire software is designed to make the process of producing the HTML and necessary scripts possible for researchers with little or no technical knowledge. It generally provides a form-style interface to allow the questionnaire to be built up through inputting the questions and response types, and selecting pre-programmed features such as required responses, skip patterns, randomised ordering of choices, and resubmission checks.

There is a huge range of options available. At the time of writing, the [External Link - opens in a new window]WebSM searchable database of online questionnaire software and services has 408 entries categorised according the following:

  1. Type (software or service);
  2. Code availability of software (open or closed source);
  3. Charges;
  4. Language;
  5. Country where offices are located.

The [External Link - opens in a new window]Association for Survey Computing searchable software register also lists 79 examples of software that can generate online questionnaires. A simple Google search reveals many more examples of online questionnaire software with a vast range of features and prices.

This page will offer examples of the different types of software available and consider the different features and levels of sophistication commonly offered. It will also explore the factors involved in making a choice of software, allowing you to develop a checklist for use when comparing different options.

NB. Any references to prices or features are correct at the time of writing, but should be seen only as a snapshot of what is on offer in April 2005. References to particular software providers are given only as examples of the options available and should not be seen as an endorsement of any particular products or services, nor as a guarantee of quality.

 

Close heading CLOSE

Open/close headingSoftware plus hosting

For researchers without suitable access to a server, or for those who prefer to avoid installing and maintaining software, a wide range of hosting services are available. In the majority of cases, the whole process of producing and implementing the questionnaire is carried out online using a form-based interface. Many companies that offer 'software only' options also tend to have hosting facilities which registered users can take advantage of.

There are a great many companies offering services targeted at all levels of users from those offering design, hosting and analysis facilities for individual researchers for free, to those offering solutions to multinational companies for thousands of pounds. Many offer a variety of products targeted at different users with special rates for small-scale individual use. Others offer educational rates or even free services for researchers who are prepared to fulfil certain conditions such as having a link to the company on the institution website, and being prepared to contribute their survey questions to libraries of question types for reuse.

If the researcher decides that this type of service is suitable for his or her study, a number of questions must be considered when choosing a service. The aim should be to find an appropriate balance between cost and features required.

Open/close headingCost

The cost of the services is often dependent on the number of respondents, questions and or surveys. Most charge a fixed monthly or annual fee, often setting limits on the number of completed surveys and charging additional fees for each survey beyond this maximum.

Many offer different levels of service, at different prices, usually with names such as 'basic edition', 'standard edition' or 'professional edition'. This generally involves differences in the maximum number of surveys allowed and differences in the levels of technical support offered. It may also involve additional features such as more advanced analysis and reporting tools, and more sophisticated question types and questionnaire options.

Given the competitiveness of the market, the majority of the services offer clear pricing information which can be linked directly from the home page. This tends to allow relatively easy comparison of different providers.

 

Close heading CLOSE

Open/close headingAvailable features

Many of the software company websites offer product demonstrations, tables of features, example surveys, and/or the creation of free demonstration surveys to 'sell their wares'. This makes it relatively easy to check the features that are available, but can also quickly lead to information overload.

A useful example survey which includes comments on the features illustrated by particular questions can be found on the demo pages of the SelectSurveyASP service at [External Link - opens in a new window]http://www.classapps.com /SelectSurveyASPDemo.asp (Note that this software does not offer a hosting service).

Surveyz is one of many options that offer a fully functional trial of their software and hosting service which allows the user to quickly get a feel for the functionality and options available. Registration is required to activate this trial and the trial is restricted to a one-month period, one questionnaire active at any one time, and a total of 25 responses. [External Link - opens in a new window]http://www.surveyz.com/

It is a good idea to create a checklist of the features that you are likely to require. This can then be compared to the features offered by different providers. It can be particularly useful in enabling a minimum level to be established where a provider offers increasing levels of service at different prices.

The 'creating a checklist' activity below will guide you through the process of developing such a checklist according to the needs of your study.

 

Close heading CLOSE

Open/close headingExamples of services available

The following examples are chosen as being representative of some of the different types of services available. In each case, a range of comparable options may be available.

Service Comment
[External Link - opens in a new window]Bristol Online Surveys Targeted at institutions requiring the option to have a number of different surveys and survey administrators. Highly customisable to the style needs of institutions including an option to run surveys on their server with an address that appears to be that of the institution.
[External Link - opens in a new window]Globalpark Provides'EFS Survey' software which offers relatively advanced features and a high degree of flexibility. Also makes the software available to academic researchers for a nominal fee through [External Link - opens in a new window] unipark providing excellent value for these users where advanced functionality is required.
[External Link - opens in a new window]SurveyConsole / QuestionPro Both are divisions of the [External Link - opens in a new window] surveyanalytics company and they use the same software and interface, but with different pricing. May offer sponsored use for academic or not-for-profit projects if certain conditions are met. Also offer a range of free resources such as articles and question templates.
[External Link - opens in a new window]surveymonkey Compares well with many of the other available services in terms of features, but is one of the cheapest commercial options.
[External Link - opens in a new window]surveywriter Relatively expensive, but unusual in that charges are not made per period of use, but per completed survey and email invitation with a minimum of 200.
[External Link - opens in a new window]Surveyz! A range of relatively sophisticated features. Targeted at individual researchers or at institutions. Offers academic pricing and free use for academic projects if certain conditions are met. Also has a range of resources such as articles on online questionnaires and copyable templates of questionnaires and questions.
[External Link - opens in a new window]websurveyor Relatively expensive, but with a wide range of features. Offers both hosting and software only solutions.
[External Link - opens in a new window]zoomerang Offers pricing for not-for-profit and educational institutions. Also offers a range of research services such as questionnaire administration, panel services and translation.

 

Close heading CLOSE

 

Close heading CLOSE

Open/close headingSoftware only

The use of software-only solutions depends on the researcher having access to a suitable server with adequate administration permission to, for example, create and access a database. It also requires the software to be installed and configured correctly and for any possible problems that might emerge to be dealt with. Where these requirements can be met, this type of software can have a number of advantages. These may potentially include:

  • Savings of cost, especially when used in the longer-term;
  • Increased control over data collection and storage;
  • More opportunities to work offline if required.

Some examples of this type are also primarily designed as survey analysis software which can offer generally more sophisticated analysis tools where required.

Open/close headingOpen source options

Open/close headingLocating software

Two of the major sources of information on Open source software are [External Link - opens in a new window]http://sourceforge.net which is a repository of open source projects, and [External Link - opens in a new window]http://freshmeat.net/ which is a listing of new software releases.

A simple search for 'survey' reveals a range of projects providing software for online questionnaire creation. These range from simple survey software with limited question types and functionality, to options with features which compare very well with the majority of commercial options.

 

Close heading CLOSE

Open/close headingChoosing options

When choosing whether or not to use an Open source solution, it is important to consider the following potential problems:

  1. They do not tend to offer hosting, and installation and configuration can be complex (though this may be less problematic than with commercial software-only alternatives);
  2. The user-interface for the software can be more complex in some cases;
  3. Documentation such as user-guides may not be as extensive;
  4. Where problems are encountered, suitable technical support may not be guaranteed.

Making a successful choice depends on identifying whether or not the software provides the features required. The 'creating a checklist' activity below will guide you through the process of developing a checklist according to the needs of your study which can be used when comparing software. In many cases the information on the features available is less extensive than commercial options, though many of the more established options offer demonstrations, examples and feature lists.

It is also necessary to ensure that the software can be installed and configured on a suitable server, and that someone with the necessary skills is available to maintain it successfully and deal with problems. An indication of what is typically involved in this is given by the administrator's manual for Lime Surveys, which can be found by following the 'Documentation' linkat [External Link - opens in a new window] http://www.limesurvey.org/ . Many of the other open source options also have similar installation guides which can be accessed through the project website or downloaded with the software.

Finally, it is important to check that the software has an adequate level of reliability. The JISC open source software advisory service, OSS Watch, at [External Link - opens in a new window]http://www.oss-watch.ac.uk/ offers useful guidance on criteria for choosing reliable software. Some key factors identified are:

  1. Reputation - Have you spoken to people with experience of a particular product?
  2. Ongoing effort - Is there clear evidence of active development of the software?
  3. Community support - Is there an active community of users on the project mailing list ready to answer questions from users experiencing problems?
  4. Version - Has the latest stable version been available for some time and is there evidence that problems have been identified and fixed?
  5. Documentation - Is this sufficient to allow you to decide whether or not the software is sufficiently well-developed for your purposes?

See the page of 'Top Tips For Selecting Open Source Software' at [External Link - opens in a new window]http://www.oss-watch.ac.uk/resources/tips.xml for further details.

 

Close heading CLOSE

Open/close headingExamples

The following are some of the main examples of open source software for online questionnaires (generally the more established and/or sophisticated options). A range of other options may be available.

Software Comment
[External Link - opens in a new window]LE Survey Designed as a tool for running questionnaires as part of longitudinal studies. Allows respondents' responses to be matched to responses to previous questionnaires while maintaining confidentiality. In early stages of development at the time of writing.
[External Link - opens in a new window]phpESP Well-established software with a working demo available allowing the features and user-interface to be tested. Offers low cost hosted services.
[External Link - opens in a new window]LimeSurvey A range of relatively sophisticated features. Well-established with useful documentation. Working demos are available allowing the features and user-interface to be tested.
[External Link - opens in a new window]Mod_Survey Well-established with sophisticated features such as dynamic content generation depending on previous answers. Requires the user to learn to use XML syntax particular to the software.
[External Link - opens in a new window]VTSurvey Easy to use and particularly useful for straightforward questionnaires as only the four main types of questions are supported (Multiple choice with radio buttons and Check boxes, and short and long text entry boxes).

 

Close heading CLOSE

 

Close heading CLOSE

Open/close headingCommercial options

Open/close headingChoosing options

Documentation and technical support for commercial options tends to be more extensive than that offered by open source software, but in many cases there are charges for a support package. They frequently offer powerful statistical analysis, and many can be used to develop and analyse questionnaires off-line. In the majority of cases, hosting services are offered for an extra cost.

A number of the commercial options were originally designed as software to aid in the production and analysis of questionnaires in paper-based and other modes, often with online questionnaire functions added later and offered at extra cost. Others offer online questionnaire functions as part of a set of 'add-ons' allowing, for example, computer-assisted telephone interviewing (CATI) or surveys for Personal Digital Assistants (PDAs) or other mobile computing technologies. This can make them a particularly effective choice when there is a need to integrate mixed-mode surveys.

It is useful to list the features required for your study which can be used when comparing software. The 'creating a checklist' activity below will guide you through the process of doing this, providing you with some of the main questions to consider.

 

Close heading CLOSE

Open/close headingExamples

The following are some examples of commercial software for online questionnaires. A range of other options are available.

Software Comment
[External Link - opens in a new window]ProntoSurvey Shareware tool from Bebosoft which can be downloaded for Mac as well as windows. Relatively inexpensive and straightforward to use with clear and extensive documentation and tutorials.
[External Link - opens in a new window]Questionmark Perception An educational assessment tool which offers many of the key features needed to create basic online questionnaires and has some of the more advanced features such as randomisation of questions and conditional branching. Potentially useful option if the software is available through the researcher's institution.
[External Link - opens in a new window]SelectSurveyASP Relatively inexpensive. Offers 'classic' and 'advanced' versions with different levels of features at different prices. Has a working online demo and a useful example survey which includes comments on the features illustrated by particular questions. Offers a free installation service and free technical support.
[External Link - opens in a new window]Snap Surveys Extensive options for mixed-mode surveys, offering a 'core product', Snap Professional, with add-ons for questionnaires via internet and PDAs, and to allow scanning and multiple data entry. Expensive example of 'high-end' options.
[External Link - opens in a new window]StatPac The online questionnaire software does not include analysis tools, but it can be purchased alongside the statistics tools offered. Has basic statistical tools or an advanced version allowing multivariate statistical techniques. Technical support and updates are available free for three months, but are chargeable via annual support/maintenance agreements thereafter. A fully-functional version of the software can be downloaded and used for free, limited to 35 respondents for each survey. Download includes tutorials and extensive user guide.

 

Close heading CLOSE

 

Close heading CLOSE

 

Close heading CLOSE

Open/close headingActivity: Creating a checklist

[?] This activity will allow you to answer a number of questions about the requirements of your study, leading to the creation of a personalised checklist for use when comparing products.

Alternatively, a checklist containing the complete list of questions can be obtained by following the link below.

Open/close headingCreate personalised checklist

[i] Answer the following 12 questions about your study.

When you have finished, press the button to see a list of options for your final checklist.

This will open in a new window, which you should close to return to this section.

 

months

4. Do you need only one questionnaire or do you need multiple questionnaires to be online at the same time?

One
Multiple
Don't know

5. Do you need a one-page questionnaire or one that spans multiple pages?

One page
Multiple pages
Don't know

6. Do you expect all the participants to complete the questionnaire online or do you need to be able to use the online questionnaire in tandem with paper-based versions?

Online only
Online and paper-based in tandem
Don't know

7. Do you intend to create a paper-based version of the questionnaire for conversion to an online format, or to create the questionnaire directly using the software?

Convert from paper-based version
Create directly using the software
Don't know

8. Are you willing / do you want to adapt the design of the questionnaire or do you just want to use templates?

Willing to adapt the design
Templates only
Don't know

9. Which of the following methods do you plan to use to distribute your questionnaire?

Email
Via web link(s)
Email and web link(s)
Don't know

10. Do you plan to use a package such as Excel or SPSS to analyse the results or to analyse them directly using the software

Analyse using a package such as Excel or SPSS
Analyse using the software
Both
Don't know

11. Do you plan to gather highly sensitive information?

Yes
No
Don't know

12. Which of the following are you likely to need?

Access control through passwords or by limiting access to those from particular IP addresses.
The option to require a response to certain questions.
Randomisation of the order of questions or responses within questions.
Extra design features on individual pages, such as images, extra text or line breaks.
Features such as images, headers or footers that can be applied to all pages.
Skip patterns so that the participant is directed to alternative questions depending on responses to particular questions.
The inclusion of multimedia stimuli such as video and animation.
The inclusion and placement of a progress bar.
The facility to enter a closing date for the questionnaire and add a customisable message to those who attempt to access it once closed.
The option of receiving the results by email alongside saving them to the database.
The option of linking to a webpage of your choice once the questionnaire has been submitted.
The option of adding cookies to the participant's computer to prevent multiple submission from the same machine
Control over the use of cookies to allow particular machines to make multiple submissions (e.g. where a suite of computers may be used by different participants, or for data entry from paper-based versions)?

 

 

 

 

Close heading CLOSE

Open/close headingView all checklist questions

[i] Select the headings below to view the complete list of checklist questions.

Open/close headingGeneral

Does the service allow you to collect the maximum number of completed questionnaires you expect without extra charges?

Does it allow the maximum number of questions you need to include in your questionnaire?

Will you be within your budget if you use the service to keep the questionnaire open for the time you need?

Do you need only one questionnaire or do you need multiple questionnaires to be online at the same time?

Do you need a one-page questionnaire or one that spans multiple pages?

Do you expect all the participants to complete the questionnaire online or do you need to be able to use the online questionnaire in tandem with paper-based versions? Can paper-based versions of the questionnaire be easily produced?

Does the interface seem to be easy enough to understand and use in the time available?

Is online and/or telephone technical support on the use of the service available if required?

 

Close heading CLOSE

Open/close headingQuestion creation

Does it produce all the question types you would like to include?

Are there any limits to the number of choices for particular question types? Is this a problem for your questionnaire?

Can questionnaires be imported from word-processor or text files?

Is the layout and positioning of questions and answers acceptable? Can this be controlled?

Can you save the questions you have input and continue creating the questionnaire at a later date?

Is there the option of requiring a response to certain questions?

Can you randomise the order of possible responses?

 

Close heading CLOSE

Open/close headingDesign and layout

Does it produce pages that have no major differences when viewed in different browsers or with different screen resolutions?

Is there a choice of templates? Are they suitable? Are the layout and colour schemes acceptable?

Do any templates allow easy customisation?

Is it possible to add extra design features to individual pages, such as images, extra text or line breaks?

Can you easily add features such as images, headers or footers to all pages?

 

Close heading CLOSE

Open/close headingQuestionnaire settings

Is the statistical information offered by the software extensive enough for your needs? (e.g. can it record IP addresses of participants, approximate time taken, response rates for each question etc.)

Can the number of questions per page be easily controlled?

Can skip patterns be established so that the participant is directed to alternative questions depending on responses to particular questions?

Can the navigation be customised? Can the text on buttons be customised and is there a choice over whether or not a respondent can navigate back to completed sections?

Can multimedia stimuli such as video and animation be included?

Does it allow randomisation of the order of questions or responses within questions?

Can cookies be added to the participant's computer to prevent multiple submission from the same machine?

Can this use of cookies be controlled to allow particular machines to make multiple submissions (e.g. where a suite of computers may be used by different participants, or for data entry from paper-based versions)?

Does it allow for the inclusion and placement of a progress bar?

 

Close heading CLOSE

Open/close headingImplementation and security

Are the welcome pages and post-submission messages fully customisable?

Can you link to a webpage of your choice once the questionnaire has been submitted?

Can you enter a closing date for the questionnaire and add a customisable message to those who attempt to access it once closed?

Does it offer the access control you need, such as the ability to set passwords or limit access to those from particular IP addresses?

Are you satisfied by any security statements made?

Is there an option to encrypt the information on a secure server?

 

Close heading CLOSE

Open/close headingDistribution

Is there an email management list?

Can you insert individual email addresses?

Can multiple email addresses be imported?

Can you select the whole list or a subgroup of the list to send a message to?

Can you send personalised emails by merging personal information into a standard email?

Can the link to your questionnaire be easily incorporated into emails?

Can you send your entire questionnaire as an email?

Can you easily incorporate a link to the questionnaire into a website?

Can you record which of several links was selected by the participant?

Can a pop-up version of the questionnaire or of a link to the questionnaire be activated when a user enters the site?

 

Close heading CLOSE

Open/close headingResults and analysis

Can results be received by email alongside saving them to the database?

Does it allow the data to be downloaded in a format that can be easily imported into the statistical analysis package you plan to use?

Are there any limits to the number of times that you can download your data? Is this a problem for your study?

Can you choose whether you would like to include all data in the download, or only data from a particular period?

Can any extra information you need be easily included in the data files (e.g. IP addresses and start and completion times)?

Is the statistical analysis and reporting offered by the software sophisticated enough for your needs?

Can you view individual results and overall results on an on-going basis?

Does it offer the option to delete or edit results?

Are there the viewing options you require (e.g. Can you view the results as percentages or in the form of a particular type of chart)?

Can you easily access the statistical information you need from within the software?

Does it offer filtering options so that results can be analysed according to the response to particular questions (e.g. focusing on gender or age)?

Is cross-tabulation of results possible, so that correlations between particular questions can be analysed?

 

Close heading CLOSE

 

You can copy or print a checklist with these questions for use when comparing products, by following the link below:

[i] The checklist will open in a new window, which you should close to return to this page.

Open checklist

 

Close heading CLOSE

 

Close heading CLOSE

  © 2004-2010  All rights reserved    |    Maintained by ReStore    |    About this website    |    Disclaimer    |    Copyright    |    Citation policy    |    Contact us