Click on the headings to open them. They will open on this page. Open the following link for further information about these headings if required.
Your browser does not support these headings. To ensure that the contents remain accessible, they have been automatically opened so that all the information on the page is displayed.
However, to take advantage of the headings and to ensure that the layout and design of this site are displayed correctly, you are recommended to upgrade to a current version of one of the following standards compliant browsers:
- Internet Explorer (http://www.microsoft.com/ windows/ie/downloads/ default.mspx)
- Mozilla Firefox (http://www.mozilla.org/ products/firefox/)
- Opera (http://www.opera.com/download/)
Glossary links are included within this page. If a word appears as a link, clicking on this link will show the definition of the word in a 'pop-up window'. Select the following link for information about these glossary links if required.
- Select the links see the definitions in a pop-up window.
- NB. If you use pop-up window blocking software, you will need to deactivate it for pop-ups on this site to use the glossary links. Alternatively, all glossary definitions can be seen on the 'Glossary' page in the 'Resources' section.
This page aims to offer an overview of the process of creating an online questionnaire and to provide an introduction to some of the key issues that must be considered when deciding on an option for implementing a questionnaire.
It will also focus on the different options available, and provide examples of the technical choices a number of researchers made to facilitate their studies, and their rationales for doing so.
Finally, it will outline some of the key skills required to successfully employ each of the different methods.
A typical procedure for implementing an online questionnaire is as follows:
- Create an HTML document with questions and form elements to collect responses.
- Ensure that the document is correctly designed and formatted for consistency and accessibility.
- Add server-side scripts to allow the information to be collected through emailing the results in a useable format or automatically populating a database or text file with them.
- Add server and/or client-side
scripting to minimise measurement error and improve the quality
of data collection. This may involve:
- Incorporating validation routines to check that the information submitted is in a suitable format and that no questions or selections have been accidentally missed.
- Establishing access control to (where needed) increase the chances that only those who are part of the sampling frame take part in the survey.
- Including routines to obtain information such as whether or not the respondent's machine has been used to submit the information previously, or how long it took to complete the form.
- Adding a facility to thank the respondent for their participation and assure them that their submission has been successful.
- Add server and/or client side scripting to improve the respondent's
experience of completing the questionnaire and thus to reduce
non-response. This may involve:
- Preserving data entered so that it is not necessary to restart either the survey or a section of the survey if the respondent makes an error or is interrupted.
- Adding effective skip mechanisms or tailoring subsequent questions in the survey according to particular responses given.
- Providing a progress bar or similar mechanism to indicate roughly how far through the survey a respondent is.
- Upload the document to a server to make it available online.
- Carry out the necessary distribution activities by, for example, emailing members of a sample list and sending follow-up emails according to responses.
A researcher who wishes to carry out each of these stages independently is likely to need a wide range of web design and web-programming skills to do this effectively. Specialist software or institutional systems are also frequently used to automate or simplify the process.
There are three main options available to researchers who wish to implement an online questionnaire. They can either take advantage of institutional systems that may be available, use an 'off-the shelf' solutions to host and/or produce and manage the questionnaire, or use and entirely self-produced questionnaire.
The choices made are likely to be influenced by issues such as the following:
- The number of likely respondents;
- The length of time the questionnaire will need to be online for;
- Whether there is a need for more than one questionnaire to be running concurrently;
- Whether there is a need to link data from particular respondents over time;
- Whether the plan is to use the questionnaire in tandem with paper-based or other types of survey;
- If so, whether the facility is required to enter data from paper-based questionnaires into the same database as the online ones, or to use machine-readable paper questionnaires.
- Whether a budget is available;
- Whether the researcher is working in an institution which provides technical support or tools for creating online questionnaires;
- Whether there is easy access to a server on which to house web-pages and scripts;
- Whether adequate administrative permissions are available to, for example, create and access a database on the server.
- Whether a simple replacement for a paper-based questionnaire is required, or whether there is a need for particular features such as the option to require a response to particular questions, the incorporation of audio and video materials, randomisation of questions, or the inclusion of branching or skip patterns
- The extent to which the researcher plans to control access to the questionnaire;
- Whether a particular sample group will be targeted;
- Whether a list of email addresses is available to enable contact with the sample;
- Whether the technological facilities are required to manage the scale of email correspondence by, for example, checking for submission and sending follow-up emails to particular groups.
- Whether particularly personal or sensitive information is to be collected that will need to be kept on a secure server;
- Whether there is a need to securely share access to the data with others;
- Whether there are particular style and layout needs;
- Whether there is a need for particular types of analysis;
- Whether there are any relevant institutional constraints such as the need for the questionnaire to be housed under the URL of the institution or for the questionnaire to meet institutional style guidelines.
- The level of experience of internet-mediated research and web design expertise;
- Whether the questionnaire has to be developed to a tight timescale, and the amount of time available to the researcher to develop skills such as the following:
For researchers working within an institution such as a university, a government department, an NHS trust, or a charitable organisation, technical support may be available which the researcher may be able to use to implement the questionnaire. In some, cases, support may extend to the bespoke conversion of a paper-based questionnaire to a web version.
1. The Association of American Geographers 'Internationalizing Geography in Higher Education' study
Michael Solem (Educational Affairs Director)
How did you implement the questionnaire?
The development process was relatively straightforward and the questionnaire was programmed by one of our technical staff.
Did you experience any problems at all?
We did experience some database issues during implementation (sometimes buttons didn't submit information properly), but nothing that couldn't be resolved fairly quickly.
Were there any issues around response rates?
We mailed a printed version of the questionnaire to everyone who didn't respond to the two online requests, which increased our response rate significantly. My experience has been that printed surveys are needed to motivate large numbers of non respondents to participate, who otherwise would ignore the Web survey.
Waverly Ray (Research Assistant)
What was the context of the research?
The research investigated factors that influence geography faculty members’ participation in activities that serve to internationalize college and university campuses, including international collaborative teaching and research.
Why did you choose to carry out the study using an online questionnaire in tandem with a paper-based version rather than a paper-based version or an online version only?
To lower data collection costs, an online instrument was developed. The response rate from the online questionnaire was not as high as anticipated, so paper-based versions of the instrument were sent to online survey non-respondents.
Was this decision made before or after the development of the questionnaire?
The online questionnaire was developed first, followed by the paper-version of the survey. From the beginning of data collection, the paper-based version of the survey was made available online for respondents who for personal preference or accessibility reasons did not want to complete the online version.
Why did you choose to have the questionnaire produced and hosted in-house?
Staff and resources were available to produce and host the questionnaire in-house.
Did you consider any other options for implementing the questionnaire (e.g. off-the shelf software and/or hosting solutions?) If so, why were they rejected?
To my knowledge, no other options for implementing the questionnaire were considered.
How involved were you in the technical implementation?
I was not involved in the technical implementation, other than as a trouble-shooter for the beta version of the online survey.
What did you actually have to do to get the questionnaire online (e.g. hand over a paper version / liaise over design and development / specify technical features required such as access control and format of resulting data)?
I provided the programmers with a paper version of the survey and the programmers took care of the design and development of the questionnaire. I did not provide the programmers with format requirements for the resulting data. As a result, I assigned numerical codes to the data prior to analysis. This step could have been eliminated if I had provided the programmers with format requirements.
Would you recommend your methods of implementing the questionnaire to other researchers? What were the advantages and disadvantages?
I would recommend my methods to other researchers, although I had the benefit of a staff of programmers that were responsible for the technical development of the online questionnaire which made an online questionnaire viable for my research. There were many advantages of an online questionnaire, including (a) the ease of contacting survey participants in far away places; (b) the low cost of using online methods versus paper/mail-based methods; (c) time for data-entry was lessened; and (d) errors resulting from data-entry were lessened, as well. The main disadvantage is that the anticipated response rate was higher than the actual response rate. So it was important that we had the resources to mail paper-versions of the questionnaire to survey non-respondents.
What advice would you give to people in working with technical support staff to get a questionnaire implemented?
In working with technical support staff, I would suggest that other questionnaires are reviewed so that there is a clear understanding of what format would best suit the aims of the research. Then, the finalized questionnaire items, the array of possible responses, and numerical codes for the resulting data should be provided to the technical support staff. Creating beta versions of the survey is important, so that the survey can undergo revisions.
Were there any issues that emerged through the process that you hadn't expected?
Technical issues arose that the support staff resolved without my involvement. The expected response rates were lower than the anticipated response rates, which resulted in delaying data analysis until paper-based versions of the questionnaire could be received from survey non-respondents.
What experience of web page production had you had prior to setting up these pages?
I had very little web page production prior to participating in this research.
Did you have to learn any new technical skills?
No, I did not have to learn any new technical skills.
2. Nicky Shaw (Lecturer in Operations Management, Leeds University Business School)
What was the context of the research?
We were a team of non-technical academics from social science and psychology disciplines. None had experience of setting up online questionnaires, though all were proficient in questionnaire design. The survey was of Leeds University academic staff (all grades) regarding work-life balance. The questionnaire comprised four sections (three research areas plus a demographic) and was quite long.
Why did you decide to use an online questionnaire?
Really, our driving force was cost (so the online bit was a relatively cheap option saving on photocopying and return postage) and ease of data entry. We were looking to have the responses migratable to a format compatible with SPSS 'by magic'!! Which worked! The questionnaire was in html and dropped all responses into a comma delineated format, which could be opened in Excel and copied across to SPSS.
When did you make the decision?
Actually, (as academics) we were more concerned with the questionnaire itself - which took around 1 year to develop - and the issues around making it available online were addressed proportionately quite late on.
Why did you use the university to host the questionnaire (as opposed to a commercial hosting service)?
Never occurred to us to pay outside people!! We had obtained a little funding, which was just enough to pay an internal computer guy within one of our departments plus incentives for recipients...
Were there many issues that emerged through the process that you hadn't expected?
All the points below came about as a result of conversation with other colleagues and the technical staff member involved in setting up the website. None were issues we had consciously considered prior to embarking upon an online questionnaire
- Password protecting the website, to prevent people completing the questionnaire from some other 'sample population'.
- We knew we wanted respondents to view the three research sections in a random order (to prevent fatigue on one particular section) and this was done automatically as people accessed the questionnaire
- We indicated the rough time duration for each section and colour coded the sections for clarity.
- To prevent people missing questions by mistake, a check was performed on missing answers for each section and the respondent informed of any omissions (highlighted in red). If they still wished to skip the question, the submit button would take them to the next page the second time around.
- Logic pathways had to be very clear for programming purposes (e.g. If answering X to question Y, go to section Z etc).
- A lot of attention had to be paid to the coding, both in SPSS and Excel, as well as the main data file and a few dry runs performed to make sure that data wouldn't be lost.
- The questionnaire was designed in Netscape and the layout appeared distorted initially when running in Explorer.
Where this kind of service is not available, there may also be standard procedures in place for implementing web forms once they have been created by the researcher. This may involve a mailing facility whereby the data is automatically formatted and delivered by email when the form is submitted. It may also involve a facility allowing the data to be downloaded in an appropriate format for importing into a statistical analysis, database or spreadsheet package.
Tim Vorley (Department of Geography, University of Leicester)
Why did you use web-based questionnaires?
I used web based questionnaires not because they are innovative, but because they were wholly appropriate. They offered a dynamic alternative to paper based questionnaires, with the prospect of a higher return rate as there is no work involved other than the responses themselves. Also the automatic coding of answers meant that responses could be interpreted quickly and saved into databases and separate files relating to each respondent.
What experience of web page production had you had prior to setting up these pages?
I had no experience of web programming before creating the first version of the online questionnaire. HTML was easy to pick up and so I was able to teach myself the majority of what I needed to know, but support was available.
Did you use a WYSIWYG editor like Macromedia Dreamweaver or Microsoft FrontPage or code them by hand?
The web-form was created using a combination of WYSIWYG and hand coding – I think there are pros and cons to simply using WSYIWYG as it is important to understand what you are doing and how you are doing it to avoid confusion further down the line!
How did you go about implementing your questionnaire?
Firstly, I read up about basic HTML programming and learnt the basics. This allowed me to develop the skills needed to produce my questionnaire. I then took advantage of the University’s facilities to host the questionnaire and to link it to server-side scripts allowing the data to be automatically emailed to me.
What kind of support did you receive?
The computer centre at the university offered a good level of support in setting up initial web-form and the server-side scripts for submitting results. Subsequently they helped iron out minor problems with the format of the questionnaire and the way in which results were received via server-based emails.
What exactly were these problems and how did you solve them?
Firstly I had problems with the frames in the webpage. By playing with the page, I managed to get the optimal viewing resolution lower so it appeared on even the lowest resolution and was easier to read. The problem with the automated replies was firstly with the coding. Initially the responses were uncoded and were received in a format that made it difficult to deal with the data. Both of these issues were resolved through trial and error and with the advice of the computer centre until I was receiving the results in the most user-friendly manner.
Why didn't you use off-the-shelf questionnaire software?
Using my limited and self taught knowledge of HTML programming I used Microsoft FrontPage to produce a clear and simple web-based form, designed using university corporate logos and colours. This was preferred over using off the shelf software because of the ability to build the web-form to my exact specification, also using widely available software meant teething problems were solved with the computer centre. At the initial time of conception I was also unaware of any appropriate software for designing questionnaires, and on further enquiry did not have the resources to purchase such software.
What did you do to check the data you received?
All of the respondents to the questionnaire were invited. With name and contact details on the questionnaire itself, responses were verified and constituted the basis for determining whether respondents would subsequently be interviewed. None of the data collected was explicitly presented within the research statistically or otherwise, but informed the interview process. The fact that the questionnaire was sent to invited respondents meant that security was not really an issue so I avoided many of the problems involved with ensuring the integrity of results and respondents.
Would you use the same system again if you were to do a similar study?
The system was easy to use and worked for me on this occasion. I am not the greatest fan of using surveys or questionnaires; however this was a lot easier to manage than a traditional paper based survey at every stage. The ability to modify questions and responses that did not work or were inappropriate after the pilot study was easy to do at any time when the initial questionnaire was set-up. If I did need to use a survey again I would use online over traditional every time!
What advice would you give to people in using systems like these?
I'd say to be persistent! If you have an idea, the chances are that it is possible but you just aren't sure how to do it at the moment! As a social scientist my forte is not designing websites or programming, but there are people out there who can help - the challenge is finding them and learning something new!
In some cases, the institution may have purchased a site licence for an off-the shelf survey creation and administration package which may be suitable for the purposes of the research project.
In educational institutions in particular, site-licences may also have been purchased for assessment software which may provide the facilities needed for the questionnaire. These are generally designed to allow students to answer a range of questions, such as multiple-choice questions or short text entry questions, online and submit their answers. The tutor can upload the questions and perhaps set passwords for each individual participant. They can then download answers and see reports on the answers of individuals or groups. Because these are the same functions as those of basic survey creation software, this can be a straightforward (and free) way of implementing the questionnaire if the software is sophisticated enough for the needs of the study (e.g. by offering an adequate range of different question types). However, it should be remembered that these tools may only be available to participants within the institution.
The procedures for using these systems are often available on the institution intranet or through contacting computer services.
A wide range of software is available that is designed to allow easy creation and administration of online questionnaires. There is a great deal of variety in the features available, and also the cost. There are a number of free open source examples which allow users to create basic forms with little knowledge of web programming. A variety of commercial software is also available, at a wide range of prices. Many of these offer different features according to the prices paid and it is common for them to offer free trials which are limited either in the number of respondents who can be surveyed or in the length of time the survey can be made available. The majority offer hosting services for those who do not have suitable access to a server, or who prefer to avoid installing and maintaining software.
The major advantage of these facilities is that they can reduce the need for web-design and programming skills. The majority of them offer a form-style interface into which the researcher adds questions and decides the type of web-form element they would like to use for answer. They also typically allow a range of formatting options and choices for validation, such as whether or not a response to particular questions will be required before submission. They can also provide server space if required and can allow a secure means of saving and retrieving data in a format such as comma-separated values (CSVs) suitable for easy input into common spreadsheet and statistical analysis applications. Other examples offer automatic analysis of data or facilities to easily cross-tabulate or filter results.
However, in some cases, they can also serve to limit the options available to researchers who may have particular design, validation or analysis needs not easily catered for by 'standard' software. Some basic knowledge of web design and programming can serve the researcher who wishes to 'tweak' the content or look of the surveys produced by these programs well.
- Examples of the different types of software available;
- An overview of the different features and levels of sophistication commonly offered by providers;
- Guidelines for choosing software and the opportunity to develop a personalised checklist for use when comparing different options;
- A general outline on how to use the software.
There are many organisations that offer consultancy services
for online questionnaire creation and administration. These
range from advice and assistance with particular aspects of
the process, to complete design, hosting, management and analysis
services. Examples of these include those provided by academic
research institutes such as the Bristol Online Survey services
offered by the University of Bristol's Institute for Learning
and Research Technology (http://www.survey.bris.ac.uk),
and those provided by commercial companies such as the survey
shop, the research services arm of the snap survey software
surveyshop/servicesweb.shtml). Of commercial market research
and Mathur (2005)'s examination of the involvement in online
surveys of the largest US-based and global market research firms
provides an extensive list of the services offered as of late
There are a number of cases in which a researcher may wish to carry out the whole process of creating and administering an online questionnaire:
- The researcher may have access to a suitable server on which to house a self-produced questionnaire, and have the web design or programming skills required to do this or be keen to control all aspects of the process.
- Technical support or institutional systems may not be available or may not be robust enough to provide an acceptable solution. (The use of an 'off-the-shelf' software solution may be more efficient in many cases).
- Where the technical requirements of the planned questionnaire are 'standard', it is frequently possible to obtain a wide range of freely available server and Client-side scripts that can be used to set up the survey. With a little effort, a researcher with basic skills is likely to be able to acquire the knowledge to do this successfully and to customise where necessary. (Again, the use of 'off-the-shelf' software may be more appropriate).
- Where the nature of the planned survey is particularly complex and involves features that are not likely to be easily available without bespoke programming.
Martin Bruder (School of Psychology, University of Cardiff)
1. Why did you decide to use a commercial internet hosting service rather than using the services offered by your university?
As I am doing my PhD at a number of different universities (Cambridge, Cardiff, Freiburg, Berlin), I did not want to depend on any university services for running my studies (I often find it hard to solve problems, when I am not actually there). I am now using a commercial provider (rather cheap) which offers php-support (I understand that universities often do not offer this support or need time to check any scripts). As long as no reaction time measures are involved, this seems to me a very good solution.
2. Why didn’t you use some of the off-the-shelf questionnaire software which is available online?
I did not use any software package to produce my studies, because I like to have full control over what is displayed, how the data is stored, etc. (often, for example, software packages rely on cookies to temporarily store the data, which I don't like). At least for questionnaires and simple experiments, the programming seems to be manageable (and I am really not a computing specialist).
3. What kind of technologies did you use to implement your questionnaires, and have they been successful from a technical perspective?
My studies use nearly exclusively php (and mysql for saving the data) and have been working perfectly well. For questionnaire-based studies without reaction time measures, I think server-side programming really is the way to go and I am confident in saying that pretty much anyone with a Web browser can see and use my pages.
4. How did you achieve this level of accessibility?
5. What did you do to check the data you received?
For fraud detection, I look at IP-addresses, cookies that I set and display a message if a cookie is set. Also, it seems a good idea to receive all the submissions by e-mail as well as saving them to a file. This way, one quickly gets a feel for "unusual" patterns of data submission (e.g. very many in a very short time) and can check the respective data sets more carefully.
6. What experience of web page production had you had prior to setting up these pages?
I had no experience before starting online studies (well, my first Website was actually a signup page for lab participants including a questionnaire measure).
7. Did you use a WYSIWYG editor like Macromedia Dreamweaver or Microsoft FrontPage, or code them by hand, and did you have to learn many new skills to do this?
I code my pages by hand using a freeware tool called HTMLkit. I had to learn to understand what html does (which is really fairly quick) and how pages can be formatted using Style Sheets. Learning this really does not take that long (maybe 15-20 hours).
8. Did you write your own php scripts to write to the database and send the email, or did you use or adapt an existing one? Again, did you have to learn many new skills?
I used and adapted existing scripts. Although I understand the basic principles of programming, I find it very hard to put it all exactly the right way and there are lots of scripts freely available online (many of the bits one needs for a questionnaire study - like setting cookies, retaining referrer URL, retaining IP-address are fairly standard and only short pieces of scripts). I have to say, though, that for one study that involves multiple randomisations and lots of different conditions, I had quite a bit of help by a friend who does computing. So if I had to do such a study all by myself, I might either have to put in more time to learn how to properly program or find a tool that I could use. But this does not apply to simple questionnaires.
9. Presumably you developed your questionnaires on a test server (apache?) on your home computer, and then uploaded them to the commercial provider (FTP?). Is this correct?
Yes, I installed foxserv (includes apache and mysql). Often, however, I upload it directly and test it online.
10. How did you choose which provider to go with?
There are lists of the 10 or so best Web providers for each country (my provider is in Germany). I knew I needed php- and mysql-support, so then I compared the prices for the ones that offered these and was either lucky or they are all good. So far, I never noticed any interruption of service.
11. Finally, would you recommend your methods to other researchers?
Yes, although, thinking about it, I might not have done things the most efficient way. The reason being that I did not mind so much investing some time - just out of curiosity.
The tables below give an indication of the knowledge and skills set that a researcher is likely to need to design and implement an online questionnaire using different methods. Table 1 shows the importance of each skill in different contexts, and table 2 outlines the reasons why these skills will be required.
Table 1. The importance of different sets of knowledge/skills when different methods are used to create an online questionnaire.
|Knowledge/skills||Institutional systems||Self-produced||'Off the shelf' solutions|
|Knowledge of how to use questionnaire software||N/A||N/A||HIGH|
|Basic knowledge of HTML (and possibly CSS)||HIGH||HIGH||MEDIUM/LOW|
Table 2: A list of the technical knowledge and skills required to produce an online questionnaire with the reasons why each skill is required.
|Understanding of the different facilities provided by different questionnaire software and hosting providers. Familiarity with the interface of chosen software and procedures for creating and managing the questionnaire.||There are a great many 'off-the-shelf' for online questionnaire software solutions available and if a researcher decides to take this route, time may be needed to become familiar with the options and facilities available. Once a choice is made, it is necessary to become familiar with how the software is used. The complexity varies greatly and options with relatively limited functionalities are often very quick and easy to learn. However, examples that offer advanced functionalities may also be relatively difficult to learn to use effectively.|
|Basic knowledge of HTML (and possibly CSS)||It will be necessary to create and appropriately format HTML pages and web forms. This can be done with a simple text editor or with 'What you see is what you get' editors such as FrontPage or dreamweaver. These can make the process of creating HTML pages more efficient, but a basic knowledge of HTML and CSS is very useful in using these packages effectively. Where form-creation software is used, knowledge of HTML and CSS will allow you to customise the look and feel of the questionnaire after it has been produced. Even with well-designed software, problems can emerge when HTML is automatically produced, and this knowledge will allow you to deal with this as effectively as possible by 'going to the source' if necessary.|
|Design issues||An awareness of design issues related to general web-design (use of colour, optimisation of graphics, design for a range of screen resolutions and browser types, accessibility issues etc) and those specifically connected to online survey design (progress bars, tables for matrices etc) will be needed to make the survey more effective and reduce measurement error. Where form-creation software is used, this allows easier customisation of templates and response to problems that the process of automatic HTML production may create.|
|Form elements||The use of common form elements will be required to collect the data for processing. Where software is used to automatically insert forms and elements, customisation will be easier.|
|Server issues||The questionnaire will need to be placed on a suitable server connected to server-side processing facilities. Knowledge of server technology and security issues will be required to do this independently and, where support is available, a sound understanding will be helpful in getting this done effectively.|
|Server-side processing||The use of suitable server-side scripts will be necessary to allow the data to be processed and delivered or automatically inserted into a database. It will also be required to provide the respondent with a suitable acknowledgement that their submission has been successful.|