ECONOMIC & SOCIAL RESEARCH COUNCIL
How systematic should you be?
The stages of a systematic review
1. Produce a review protocol / plan
2. Assemble a review group / advisory group
3. Formulate review question(s)
4. Conduct a thorough search
5. Select relevant studies
6. Appraise the quality of studies
7. Extract information from individual studies
8. Synthesise studies
9. Report what is known and not known
10. Inform research, policy and practice
3. Formulate review question(s) 


As with any research, it is useful to establish the focus by review questions.  The questions delimit the scope of the review as the criteria for including texts become clear. 

‘A good systematic review is based on a well-formulated, answerable question.  The question guides the review by defining which studies will be included, what the search strategy to identify the relevant primary studies should be, and which data need to be extracted from each study.  Ask a poor question and you will get a poor review.” (Counsell, 1997, p.381). 

Unfocused and unanswerable questions such as 'what is known about leadership?' or 'what is known about project teams?' should be avoided. To start the process write down the purpose of your review.  This could be in the form of a question, a problem or a topic that you wish to address. You can write in natural language, for example -  how do you effectively lead a project team?

In a review of any field there will be multiple concepts, constructs and perspectives.  It is therefore essential to define key terms during the question formulation phase.  In the example outlined above, what precisely is meant by the terms (a) effectively, (b) lead, and (c) project team?

Once the key terms have been defined, the free-form question can then be structured into a focused and answerable question. Systematic reviews in medical science are often structured according to the PICO approach:

 P – Patient or Problem - For which group is evidence required?

I – Intervention - The effects of what event, action or activity are being studied?

C – Comparison - What is the alternative to the intervention (e.g. placebo / different intervention)?

O – Outcomes - What are the effects of the intervention?

An example of a well-formulated systematic review question for a medical problem (albeit without reference to comparison) is provided in the Cochrane Handbook:

“whether a particular antiplatelet agent, such as aspirin, [intervention] is effective in decreasing the risks of a particular thrombotic event, stroke [outcome] in elderly persons with a previous history of stroke [population]” (Higgins and Green, 2006, p.62).

Clearly, this approach is less appropriate to the study of complex questions and multidisciplinary topics outside medicine. Drawing on the work of Pawson (2006), Denyer and Tranfield (2009) argue that well-formulated review questions in management and organization studies need to take into account why or how the relationship occurs and in what circumstances.  They reformulate PICO into CIMO for use in the social sciences.

C – Context - Which individuals, relationships, institutional settings or wider systems are being studied?

I – Intervention - The effects of what event, action or activity are being studied?

M - Mechanisms – What are the mechanisms that explain the relationship between interventions and outcomes? Under what circumstances are these mechanisms activated or not activated?

O – Outcomes - What are the effects of the intervention? How will the outcomes be measured? What are the intended and unintended effects?

Denyer and Tranfield (2009, p. 682) provide an example of a question framed with these components:

“under what conditions (C) does leadership style (I) influence the performance of project teams (O) and what mechanisms operate in the influence of leadership style (I) on project team performance (O)?”.

Systematic reviews can address a diverse range of questions.  Review questions do not need to follow the CIMO formula.  However, your review questions should be:

  • clear
  • focused
  • well fomulated
  • answerable

 An Example of review questions    

    Key questions:
  1.  What are the association between networking and innovation and what is the nature of the relationship?
  2. Where does the UK stands internationally in terms of business-to-business networking and its contribution to innovation, with particular reference and comparisons to and between the UK, USA, France, Germany and Japan?
    Specific questions:
  • How do formal institutional mechanisms aimed at promoting business to business networking activity operate, for example: mediated by professional associations; incubators; clusters et cetera?
  • To what extent do informal channels of networking lead to innovation, for example: communities of practice; mentoring schemes; knowledge brokerage; and entrepreneurial networks et cetera?
  • How is networking behaviour successfully translated into tangible outcomes specifically related to innovation; including a focus on different forms of innovation, such as product and process innovation?
  • What examples exist of network failure and inertia militating against innovation occurring within networks and explore why networks fail?

Source: Pittaway, L., Robertson, M., Munir, K., Denyer, D. & Neely, A., 2004. “Networking and innovation: a systematic review of the evidence.” International Journal of Management Reviews. 5-6(3-4): 137-168.

   

Specifying the review questions is the hardest part of the process.  Don't be afraid to seek advice from your supervisor (if you have one) or a colleague. 

If you are conducting a review that is commissioned by a company or government department a significant amount of time should be spent discussing, negotiating and reformulating the review questions.  This can be a time consuming but rewarding exercise. 

Although the review questions are specified a priori in a systematic review, a certain flexibility and modification of questions may be necessary as you gain a fuller understanding of the problem.

Remember that the specific questions to be addressed in a review should be supplemented with the general review questions introduced in the section - 'Generic questions for a literature review'.  

References 

Counsell, C. (1997). 'Formulating questions and locating primary studies for inclusion in systematic reviews'. Annals of Internal Medicine, Vol. 127, No. 5, 380-387.

Denyer, D. and Tranfield, D. (2009). 'Producing a Systematic Review'. In The SAGE Handbook of Organizational Research Methods. D.A. Buchanan and A. Bryman (eds). pg. 670-689. London: SAGE.

Higgins, J.P.T. and Green, S. (2006). Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Library 4.2.6

Pawson, R. (2006). 'Evidence-Based Policy: A Realist Perspective'. London: SAGE publications

The text on this page was created by Professor David Denyer, Professor of Organizational Change, Cranfield School of Management.