That Research is Crappy: It stinks!
Goodness Criteria in Social Research
Any topic in society or politics that garners some interest in even the tiniest of publics will these days be met with a host of social research. Needless to say, such research will be of very disparate quality. But how can we "cut through the crap" and get to the valuable findings? That is an extremely difficult task, because the disparity of goodness criteria is almost as large as the quality variance in social research. Different theoretical traditions use completely different evaluation criteria, and, what is worse, hardly ever talk about these criteria across paradigms. Scholars that work on the same substantive topics may use completely different (albeit not always incompatible) theories, depending on their paradigm affiliation. Each of these paradigms — be it constructionism, Marxism, rational choice, to name only the few most visible paradigms — has developed its own set of criteria, which decide about the "goodness" or "badness" of a theory. In this seminar we will get to know several goodness criteria stemming from important contemporary theoretical traditions. We will compare these criteria in an initial effort to distill goodness criteria that are valid for all social science research.
The seminar is divided into five sections. After a short overview on the topic, part II will cover traditional goodness criteria, namely validity, reliability, parsimony, and empirical falsifiability. Part III discusses goodness criteria that aim at the substantive content of a theory. Part IV introduces the most common statistical goodness criteria and checks, if these criteria reflect the theoretical criteria developed in the previous weeks. In the last part of the seminar we will finally apply goodness criteria to a number of existing studies.
Organization: All required readings are to be read prior to the sessions to which they are assigned.
Requirements: Students must submit weekly assignments and produce either a term paper or an oral presentation.
Grading: assignments 70%, class participation 20%, presentation 10%; when term paper is chosen: assignments 55%, class participation 10%, term paper 35%
Prerequisites: Knowledge in statistical methodology at least at the level of Statistik 2, basic knowledge of research methodology (e.g., Schein in "Einführung in die Methoden empirischer Sozialforschung"); some knowledge of German, sociological theory, quantitative methodology and epistemology a plus.
Week 1: Constitutive Session
Presentation of the Syllabus, organization of the class.
Recommended Literature (for the entire term)
- Blalock, Hubert M. 1984. Basic Dilemmas in the Social Sciences. Beverly Hills: Sage Publications.
- ---- 1982. Conceptualization and Measurement in the Social Sciences. Beverly Hills: Sage Publications.
- Bourdieu, Pierre, Jean-Claude Chamboredon, and Jean-Claude Passeron. 1991. Soziologie als Beruf: Wissenschaftstheoretische Voraussetzungen soziologischer Erkenntnis.
Berlin & New York, NY: de Gruyter.
- Flyvbjerg Bent. 2001. Making Social Science Matter : Why Social Inquiry Fails and How It Can Succeed Again. Cambridge, U.K.: Cambridge University Press.
Week 2: The State of the Art
In an entertaining and extremely well argued essay, former NORC-boss James A. Davis analyzes the strength and weaknesses of contemporary social research. The questions he raises will follow us through the course of the seminar.
- Davis, James A. 1994. "What's Wrong with Sociology?" Sociological Forum 9:179-97.
- Sociological Forum 9 (2), 1994. Special Issue on "What's Wrong With Sociology?".
II Formal Goodness Criteria
The first part of the seminar deals with the most common formal (processual) criteria for goodness. These criteria have an elective affinity with rational choice theories in the realm of social science paradigms and with liberalism in the realm of political philosophy, but can also fairly easily be adapted to other theories and philosophies.
Week 3: Validity: Theory
(Almost) all students who took a class in introductory research methods know that empirical research should rely on "valid" indicators for the measurement of theoretical concepts. In this class we will elaborate the concept of validity and learn to distinguish different types of validity (ecological validity, construct validity, etc.).
- Blalock, Hubert M. 1984. Basic Dilemmas in the Social Sciences. Beverly Hills: Sage Publications, pp. 62-71.
- Flyvbjerg Bent. 2001. Making Social Science Matter : Why Social Inquiry Fails and How It Can Succeed Again. Cambridge, U.K.: Cambridge University Press, 2001, pp. 38-52.
- Thomas D. and Donald T. Campbell. 1979. Quasi-Experimentation Design & Analysis Issues for Field Settings. Boston et al.: Houghton Mifflin Company, pp. 37-94.
Week 4: Validity: Praxis
From an epistemological standpoint, the concept of validity is fairly easily delineated. The trouble usually starts when one tries to apply the methodological guidelines that result from the validity criterion to actual research. Then it turns out that these guidelines are not sufficiently specific to select a definite methodology. In the end, a lot depends on the theoretical framework in the decision, which methodology leads to the most valid results. We will explore how these difficulties are tackled with the example of newspaper data on collective action.
- Snyder, David and William R. Kelly. 1977. "Conflict Intensity, Media Sensitivity and the Validity of Newspaper Data." American Sociological Review 42(1):105-23.
- Oliver, Pamela E. and Gregory M. Maney. 2000. "Political Processes and Local Newspaper Coverage of Protest Events: From Selection Bias to Triadic Interactions." American Journal of Sociology 106(2):463-505.
Week 5: Reliability
Research results ought to be reproducible. This demand might be fair enough for the natural sciences. In the social sciences we face two serious problems in this respect, though:
- Society is in a constant flux; what might have been an adequate research design in 1925 Bavaria may not be suitable for 2002 Burkina Faso.
- The measurement of theoretical constructs is at best indirectly. What, if the measurement theories are flawed?
How can we nevertheless achieve somewhat reliable results?
- Blalock, Hubert M. 1984. Basic Dilemmas in the Social Sciences. Beverly Hills: Sage Publications, pp. 72-80.
- Cassel, Carol A. 1984. "Issues in Measurement: The "Levels of Conceptualization" Index of Ideological Sophistication." American Journal of Political Sciences 28 (2): 418-29.
- Stuart C. and Ethal Shanas. 1943. "Operational Definitions, Operationally Defined." American Journal of Sociology 48 (4): 482-91.
Week 6: Parsimony
"You gotta differentiate!" That's one of the key phrases social science students learn in their first semester. But the Pawlowian response to any question is not always the correct one. Contrary to conventional wisdom among sociology majors that differentiation always improves a theory, it might well be the case that simplification makes more theoretical headway. In fact, scientific theories chiefly try to meaningfully simplify empirical reality. Therefore, parsimony is an important criterion for the power of a theory.
- Arrow, Kenneth J. Mathematical Models in the Social Sciences. Daniel Lerner, and Harold Dwight Lasswell, eds., The policy sciences: recent developments in scope and method. Stanford, CA: Stanford University Press; 1951; pp. 129-154.
- Hirschman, Albert O. 1984. Against Parsimony: Three Easy Ways of Complicating Some Categories of Economic Discourse. American Economic Review 74: 89-96.
- Blalock, Hubert M. 1982. Conceptualization and Measurement in the Social Sciences. Beverly Hills: Sage Publications; pp. 27-32.
- Przeworski, Adam, and Henry Teune. 1970. Generality, Parsimony, Accuracy, and Casualty, in: The Logic of Comparative Social Inquiry. New York: John Wiley, pp. 20-23.
Also, please shave with Occam's Razor!
Week 7: Testability …
In the so-called positivism dispute in German sociology, Critical Rationalists around Popper insisted on empirical falsifiabilty as the sine qua non of social scientific theories.
- Popper, Karl R. 1966. Logik der Forschung. Tübingen, FR Germany: J.C.B. Mohr, chapter 1 & 2.
- Lakatos, Imre. 1970. "Falsification and the Methodology of Scientific Research Programmes." Criticism and the Growth of Knowledge, eds. Imre Lakatos and Alan Musgrave. London, U.K.: Cambridge University Press.
III Substantive Goodness Criteria
Primarily theories somewhat connected to Neo-Marxism apply (also) some substantive content criteria to social research.
Week 8: … vs. social critism (?)
Frankfurt School emphatically rejected Popper's testability criterion, because a solemn recourse to empirical data would contain an inherently conservative effect (because empirical data inevitably are rooted in the past. Instead Frankfurt School insists that social theory should always be aware of and critical about its social context, i.e. societal totality. With respect to the latter, Popper's "facts" might conceal Adorno's "social totality."
- Adorno Theodor W. 1962. "Zur Logik der Sozialwissenschaften." Kölner Zeitschrift für Soziologie und Sozialpsychologie 14(2):249-63.
- Habermas, Jürgen. 1963. "Analytische Wissenschaftstheorie und Dialektik: Ein Nachtrag zur Kontroverse zwischen Popper und Adorno." Pp. 155-91 in Theodor W. Adorno, Ralf Dahrendorf, Jürgen Habermas, Harald Pilot, and Karl R. Popper, Der Positivismusstreit in der Deutschen Soziologie. Darmstadt, FR Germany: Luchterhand Literaturverlag, 1989.
- Adorno Theodor W. et al. 1969. Der Positivismusstreit in der Deutschen Soziologie. Neuwied; Berlin: Luchterhand, 1969.
Check out this warped reading on the differences about the role of facts between Frankfurt School and Critical Rationalism.
Week 9: Constructionism I: Theory of Category Construction
Although there are still numerous references to Frankfurt School in contemporary literature, it is safe to say that its heyday has long passed. In social theory, social constructionism/constructivism is the rage of the day in theory circles, identity politics has replaced Neo-Marxism among progressive policy makers. Even though constructionism has been en vogue more than a decade by now, there still is very little systematic criteria that could evaluate constructionist approaches. It is two grand theorists, who do not fall squarely into the constructionist camp, who have developed some fruitful goodness criteria for constructionist research. Niklas Luhmann and Pierre Bourdieu reiterate, for instance the fairly ancient, yet counterintuitive demand that constructionist categories should not be derived from everyday concepts. Even though hardly anyone disputes the utility of the their theses, the many constructionists do not follow their guidelines.
- Luhmann, Niklas. 1990. Soziologische Aufklärung 5: Konstruktivistische Perspektiven. Opladen: Westdeutscher Verlag, chapter 1.
- Bourdieu, Pierre, Jean-Claude Chamboredon, and Jean-Claude Passeron. 1991. Soziologie als Beruf Wissenschaftstheoretische Voraussetzungen soziologischer Erkenntnis. Berlin & New York, NY: de Gruyter, pp. 15-36.
Week 10: Constructionism II: Concept Construction — Praxis
Taking the concept of society as an example, it will be shown that many theoretical approaches take a fairly naive approach to social reality.
- Luhmann, Niklas. 1992. "The Concept of Society." Thesis Eleven 31: 67-80.
- Billig, Michael. 1995. Banal Nationalism. London, U.K., Thousand Oaks, CA & New Delhi, India: Sage.
- Elias, Norbert. 1991. Über den Prozeß der Zivilisation: Soziogenetische und Psychogenetische Untersuchungen - Erster Band: Wandlungen des Verhaltens in den Weltlichen Oberschichten des Abendlandes. Frankfurt a/M, FR Germany: Suhrkamp, Einleitung.
Week 11: Constructionism III: The Social Construction of What?
Over the last two decades, analyses of social construction have skyrocketed. Yet, one wonders, what does it really mean, when we speak of the social construction of "marriage", "rocks", "nations", "genders", or "doorsteps"? When is imperative, when is it tautological, when is it sensible to speak of social construction? These are the question Hacking asks in his seminal essay on the social construction of what. BTW, one chapter indeed covers the social construction of what.
- Hacking, Ian. 1999. The Social Construction of What? Cambridge, Mass: Harvard University Press, pp. 1-62.
IV Statistical Measures of Goodness
The last part of the seminar will check, if the theoretical criteria we covered in the previous weeks are applicable to quantitative data.
Week 12: Measures of Association
When it comes to association measures (e.g. Pearson's correlation coefficient), many social researchers think "the bigger, the better." That motto ignores the specific attributes of different correlation measures and tends to neglect the issue of familiarity with a measure within the scientific community. Buchanan discusses these questions in his article.
- Buchanan, William. Nominal and Ordinal Bivariate Statistics: The Practitioner's View. American Journal of Political Science. 1974; 18 (3): 625-646.
Week 13: Bayesian Criteria I
With traditional statistical measures for goodness of fit like the Χ2 Kolmogorov-Smirnov-Test one can only compare nested models. That is troublesome, because most theoretical alternatives for the explanation of social phenomena rely on entirely different factors and are, thus, non-nested. What is more, traditional goodness criteria also do not consider the complexity of a model sufficiently. These shortcomings have recently triggered a growing advocacy of Bayesian goodness criteria, which do not share these problems.
- Rudolf K. 1998. "Kolmogorov Test, http://rkb.home.cern.ch/rkb/AN16pp/node143.html, 02. Juli 2002.
- "Chi-square Test," <http://rkb.home.cern.ch/rkb/AN16pp/node32.html, 02. Juli 2002.
- Raftery, Adrian H. 1995. "Bayesian Model Selection in Social Research." Sociological Methodology 25:112-65.
- Sociological Methods and Research 27 (3).
- Western, Bruce. 1999. "Bayesian Analysis for Sociologists." Sociological Methods and Research 28 (1): 7-34.
Week 14: Bayesian Criteria II
Follow-up on last week's discussion.
- Bartels, Larry M. 1997. "Specification Uncertainty and Model Averaging." American Journal of Political Science 41(2): 641-74.
- Western, Bruce. 1996. "Vague Theory and Model Uncertainty in Macrosociology." Sociological Methodology 26:165-92.
Week 15: Round-up Discussion
Optionally, the last session will be a weekend session, in which students can present reviews of research they have evaluated with the criteria we have covered throughout the seminar.
- Flyvbjerg Bent. 2001. Making Social Science Matter : Why Social Inquiry Fails and How It Can Succeed Again. Cambridge, U.K.: Cambridge University Press, 2001, pp. 129-140.
see Weeks 1 & 2