Sampling issues

Accessing respondents is a key concern in online questionnaires. As Coomber (1997) has highlighted, there is little point in setting up an online questionnaire and passively 'waiting' for eligible respondents to find the site: more active enrolment is needed to encourage users to complete an online survey. Though, as Fricker (2008) reports, web-based recruitment can be fruitful, it can also be fraught with difficulties around avoiding accusations of spamming, accessing hard-to-reach users such as 'lurkers in discussion boards, and even dealing with aggressive messages or attacks from hackers. The significance of having any relevant web site providers 'on your side' cannot be underestimated. This access issue is becoming increasingly important. As the use of the internet increases in the general population, and the novelty of responding to online questionnaires is wearing off, getting online users to complete online questionnaires is becoming more problematic, while 'survey fatigue' is increasingly an issue regardless of mode (Whitte, 2009). Online users are becoming wise to the fact that they are paying for the privilege of being 'over-surveyed' (McDonald and Adam 2003). The result is that online users are intolerant of unsolicited communications and invitations to participate in research are increasingly considered 'spamming' (Harris 1997), resulting in online surveys often having lower response rates than onsite surveys. Witmer et al. (1999), for instance, report response rates of 10% or lower being common for online surveys, while Shih and Fan (2007) report response rates in a mixed-mode survey being 14% higher for mail surveys compared to web-based ones.

OPEN MY REFERENCE LIST

CLOSE

A further issue of concern when using online questionnaires is that they present serious sampling problems for a study based on the quantitative tradition. There is no access to a central registry, or master database, from which to create an accurate sampling frame, nor is there any way of discerning how many users are logging on from a particular computer or how many accounts/memberships a particular individual might have. This means random sampling or gaining a representative sample is not possible which places serious limitations on the inferential value of web surveys, especially those focusing on 'broad and diffuse populations' (Couper, 2007, p.S88). Internet surveys on the whole, therefore, attempt to select a sub-set of users to participate in the survey. This may be through attempts at non-probability sampling, or through self-selection. Coomber (1997) has suggested that online self-selection is suitable to use when researching a particular group of internet users, especially when connecting with groups that are not bound in a particular area but that share a common interest (O'Lear 1996, 210). So while self-selection may clearly limit the scope of the results where broad sample representativeness is required, it is important for reaching marginal groups or if the researcher is conducting an interpretive investigation. Moreover, it must be noted that self-selection occurs in many conventional surveying situations and is not unique to online research.

There is, however, divergent opinion as to whether the internet provides an inherently biased sample population for quantitative studies. Research has documented that in the early years of its inception, those using the internet tended to be predominately male, white, first world residents under 35 years old while those with lower educational levels, lower incomes, living in rural areas and black or Hispanic were underrepresented (Mann and Stewart 2000). Some argue that access to the internet is still highly unevenly distributed both socially and spatially (Janelle and Hodge 2000; Warf 2001). Indeed, according to Silver (2000), the digital divide has continued to grow in America, and this divide is fast becoming a 'racial ravine', suggesting a biased internet user sample population. Hewson et al. (2003) however, are more optimistic. They argue that overall the evidence suggests that the internet user population now represents a vast and diverse section of the general population and that it is rapidly moving beyond the select group of technologically-proficient male professionals who were once largely predominant. Dodd (1998, 63), for example, argues that the internet's broad scope can actually improve representativeness, as many population groups usually difficult to contact may be easier to access via the internet while Litvin and Kar (2001) show that the sample characteristics of conventional methods and electronic methods are converging, with electronically solicited samples becoming more like random paper-based samples, as technological uptake of the internet increases. Indeed, most recent research opinion suggests that as online surveys can often survey an entire population of a particular group, rather than a sample, they can reduce or eliminate the effects of sampling error altogether (Umbach, 2004). Moreover, samples can also be weighted to reduce bias so if a certain demographic group is underrepresented, its responses can be counted more heavily (Best and Krueger 2004).

However, when the online researcher is working in a more qualitative tradition, then purposive sampling of key online communities/online individuals with particular interests can be a particularly useful sampling strategy. Snowballing can also be a useful sampling method whereby one individual is contacted for a specific piece of research and they recommend further contacts for the research project. Recruitment therefore gains momentum, or 'snowballs' as the research progresses. Similarly, in these cases, occurences such as the 'pass-along effect' whereby links to a questionnaire are forwarded beyond the sampling frame outside the control of the researcher, may not be negative and may, with careful consideration intentionally be incorporated into the sampling strategy (Norman and Russell, 2006).

OPEN MY REFERENCE LIST

CLOSE

A further issue relating to online questionnaires involves verifying the identity of the participants and the reliability of their responses. Often it quite simply is not possible to verify the identity of respondents so there is the possibility that some respondents may be 'spoofs' or indeed may play with their online identity in completing the research (Roberts and Parks 2001). Online research also does not enable the researcher to assess the reliability of responses. As Hewson et al. (2003, 44) state: '…when materials are administered via a computer terminal rather than in person, the researcher is less able to judge the extent to which the responses are sincere and genuine, the conditions under which the questionnaire was answered and the state of the participants at the time of participation (for example, intoxicated, distracted, and so on)…'. While being an irresolvable sampling issue of online research at present, this is not unique to virtual methods: incorrectly completed questionnaires, unreliable responses and non-verifiable identities may also be a feature of onsite surveys. Moreover, in conducting online community research, how necessary is it to 'prove' the offline identity of the participants anyway? Taylor (1999, 443) argues that this depends on the initial research question and that '…the acceptance of online life as a thing in itself' is important. Indeed, it is increasingly recognised that online textual persona cannot be separated from the offline physical person who constructs them and they are commonly based on offline identities in any case (Valentine 2001). Additionally, recent research suggests that the anonymity of participants can play a positive role in the research process, reducing researcher bias and being particularly useful for embarrassing and sensitive topics (Hewson et al. 2003). But identity verification also raises issues for the research participants too as it may be harder for them to verify the researcher's identity. As Raghuram (personal communication, 2005) notes: 'One danger of online questionnaires is that questionnaires involve trust and it may be harder to build up trust when you are not face-to-face. Physically seeing a face can give you a sense of reassurance and in itself provides a forum for communication.' One way to overcome this problem is to have a dedicated project website in which the identity of the researchers and their institutional affiliation can be verified (see Madge and O'Connor, 2002).

OPEN MY REFERENCE LIST

CLOSE

According to Jeavons (1998) response rates show no relationship to gender, age or education level. However, response rates do drop off rapidly to online questionnaires. Crawford et al. (2001) suggest that if people are going to complete a web-survey they will do so in the first few hours or days of receiving it. However, it has also been found that increasing response rates can be achieved by follow-up reminders. Crawford et al (2001) propose that a single reminder email can double the number of respondents while Schaefer and Dillman (1998) found that four repeated contacts yielded the highest response rate. To improve response rates online questionnaire formats should be simple. Complex graphics, grid questions, open-ended questions and requests to supply email addresses all reduce response rates (Jeavons 1998; Knapp and Heidingsfelder 2001; Porter and Whitcomb 2003a) (Although Deutskens et al. (2004) suggest that enhancing online questionnaires with visual elements can lead to a higher response rate). Response rates can also be improved with carefully worded introductory letters or emails which include details of estimated time to complete the survey and a statement indicating that the respondent is part of a small group chosen to participate in the study (Porter and Whitcomb 2003b). Short surveys (maximum ten minutes) improve response rates (Crawford et al (2001) as do those that request personal information at the start of the questionnaire rather than the end (Frick et al. 2001). Type of internet connection and hardware and software used in accessing the internet will also impact on response rates, while emphasis on anonymity or confidentiality has also been found to increase response rates (Michaelidou and Dibb, 2006). Trust in the sponsor of a web survey also has a direct positive impact on response rates (Fang et. al., 2009). Best and Krueger (2004) also suggest that introducing a 'social presence' will discourage item non-response. To avoid particular questions not being completed, messages can be inserted to express gratitude, emphasise importance or describe progress. The use of missing data messages when an item has not been completed has also been found to reduce item non-response.

OPEN MY REFERENCE LIST

CLOSE

1. Send introductory letter outlining project and estimated time needed to complete the questionnaire;
2. Include an institutionally sanctioned website to validate researchers' identity;
3. Provide clear instructions on how to complete the questionnaire;
4. Request personal information at the start of the questionnaire rather than the end;
5. Use simple questionnaire format and avoid unnecessary graphics;
6. Avoid grid questions, open-ended questions and requests for email addresses;
7. Design survey so it takes approximately 10 minutes to complete;
8. Do not include more than 15 questions;
9. Send one or two follow up reminders;
10. Include 'social presence' or missing data messages to reduce item non-response;
11. Emphasise confidentiality.

CLOSE

The impact of incentives on improving response rates is mixed. Some researchers suggest that incentives have no effect on response rates (Cook et al. 2000) while others indicate some improvement of response rates when an incentive is introduced (Bosnjak and Tuten 2001). There are three main types of incentive: cash equivalents through web-based companies, gift certificates from popular retailers or lotteries that promise financial or produce reward. Deutskens et al. (2004) suggest that vouchers are the best incentives with long questionnaires while lotteries are more efficient in short surveys. Bennett (2000) proposes that the incentive must be relevant to the audience while Birnholtz et al (2004) suggest that generally cash is a superior incentive to gifts for an online survey, even with technologically sophisticated respondents. This may be due to the perceived limitations, delayed payoff or reduced visibility of online gift certificates. Bosnjak and Tuten (2003) also show that pre-paid incentives have no advantages in terms of willingness to participate and actual completion rates, when compared to postpaid incentives. Finally, it must also be remembered that introducing voucher incentives has the potential to skew the response rate towards people who like shopping in the shops where the vouchers may be used. This also raises issues with respect to internationalisation, as clearly many vouchers are place specific and cannot be used beyond national boundaries. Su et. al. (2008) found that the effects of incentives for online questionnaires are largely consistent with those for more 'traditional' surveys: for example that material incentives are most effective, that prepaid incentives are marginally more effective than promised incentives, and that conditional incentives significantly reduce response rates compared to unconditional ones with only a slight positive impact on response quality. Clearly, though, a decision on whether incentives are appropriate and what form of incentive is most suitable can only be made according to the particular context of research and the target population.

OPEN MY REFERENCE LIST

CLOSE

It is clear therefore, that although the online questionnaire has great potential in reaching specific groups difficult to access using conventional means, and in increasing the opportunity for having a very large worldwide pool of respondents, it also has the potential for supporting the views of those 'privileged' with computer access. This is especially the case if the research is represented uncritically without reference to the sampling procedure. Findings from online questionnaires are indicative, should be read with caution and analysed with acceptance of the likely relative sample bias (although the degree of this cannot be measured). Thus according to Wakeford (2000, 33): 'The quantity of information that may be generated, and the speed at which responses can be collected, can result in pleasing piles of data- but we should be wary of being seduced by sheer quantity; data is only useful if it is representative of the larger population.' This is clearly currently the case but recent research does hint that in the future the sampling issue may become a less significant issue in the virtual environment. Riva et al. (2003), for example, report no significant differences in responses gained from the same questionnaire from online participants compared to those completing a paper survey, even when the online sample is not controlled. However, it is likely that the use of online questionnaires will increase further in popularity as problems of coverage bias and unfamiliarity subside. Also, as the tools for conducting online questionnaires improve in sophistication and research on how best to employ this particular method progress, it is likely that the use of online questionnaires will proliferate.

OPEN MY REFERENCE LIST

CLOSE

Following Best and Krueger (2004), there are 5 main stages in drawing a sample for an online questionnaire:

CLOSE

The following case studies give an indication of how sampling issues were dealt with in three studies employing online methods.

CLOSE