ECONOMIC & SOCIAL RESEARCH COUNCIL
P|E|A|S FAQ   | home   | contact us   | index   | links and further resources   |
Site last updated 07/05/09
text/print version
FREQUENTLY ASKED QUESTIONS

Q1: I am analysing a weighted survey, but I don't have any special survey software. I have made sure that my weights add to the achieved sample size and then I use a simple weighted analysis in SPSS. Is this OK?

Q2: I see that in order to get the correct InfoButtondesign effects and InfoButtondesign factors in SPSS I need to make sure my weights add to the population size. But I don't have information on how large my population was. What should I do?

Q3: My superviser says I should use multi-level modelling for clustered samples. Is this the same as analysing a survey and setting a cluster option in a survey package? Is multi-level modelling better?

Q4: Obviously, it isn’t a good idea to use weighting if your values vary enormously (for example, you have 96 males and four females, and you are attempting to balance it to 50:50. At what point would you say that it is unadvisable to use weighting e.g. is it closer to a 75/25 or 50/50 spilt on values?

uparrowtop

Q1: I want to analyse a weighted survey, but I don't have any special survey software. Is it OK just to make sure that my weights add to the achieved sample size and then use a simple weighted analysis, such as you would get by setting a weight variable in SPSS?

Congratulations. You are obviously a careful analyst and you notice that your results make sense. What you propose will not be too bad, expecially if your survey is a simple one with weighting as its only design feature.

You will always get the right estimates from a weighted analysis, no matter what kind of weights you use. But to get the right standard errors you ought to use probability weights (see section 3.9 of the P|E|A|S site). The usual weights you get in packages like SPSS are frequency or analytical weights. The formula for the standard errors for probability weights is quite different from the other kind. But it will not be the wrong order of magnitude if your weights add to the sample size, as can happen if you don't scale your weights. See exemplar 1 for a practical illustration of this.

If the survey you are analysing has other features like stratification and clustering, a statistical package without these features will not make allowance for this.


uparrowtop

Q2: I see that in order to get the correct design effects and design factors in SPSS I need to make sure my weights add to the population size. But I don't have information on how large my population was.What should I do?

Unless you have a large sampling fraction (say more than 1% or 2% of your population) you don't need to worry. Make a reasonable guess as to what your population size might be and get the design effects. To convince your self you can experiment by multiplying your population by a factor of 2 (say) and you should find that the design effect hardly changes. So provided you are in the right ball-park (and that uuyally means large enough) you can be confident that your design effects are OK.

uparrowtop

Q3: My superviser says I should use multi-level modelling for clustered samples. Is this the same as analysing a survey and setting a cluster option in a survey package? Is multi-level modelling better?

Your superviser is quite right. Multilevel modelling is a possible approach to clustered data. It is not the same as using methods for clustered surveys. The two methods were devised for very different sorts of application.

Multi-level modelling was devised for situations where clusters were of interest in theor own right, for example schools, hospitals. It allows you to learn a lot about how your outcomes vary within and between clusters. In survey methodology the clusters are not usually of interest, but are only used for the convenience of the interviewers. A clustered survey analysis will not tell you about the differences between the clusters.

You should discuss with your supervisor whether the clusters in your data are of intrinsic interest, or jsut a nuisance. If they are of interest then you should use multi-level modelling. If they are just a nuisance, use one of the survey packages suggested on this site. The choice of a clustered survey analysis will have the advantage of being free from the modelling assumptions that are needed for a multi-level analysis.

uparrowtop

Q4: Obviously, it isn’t a good idea to use weighting if your values vary enormously (for example, you have 96 males and four females, and you are attempting to balance it to 50:50. When is it unadvisable to use weighting e.g. is it closer to a 75/25 or 50/50 spilt on values?

I think the situation you refer to is most likely to happen when your weights are the result of non-response, and thus not likely to have been chosen so as to improve the precision of your results. The section on non response weighting has sections that discuss the effect of weighting on precision of estimates. In response to your question I have also added a paragraph to the end of section 5.7.

peas project 2004/2005/2006