Every text included in a systematic review should be appraised using an explicit criteria.  In medical science reviewers a hierarchy of evidence:    

Ia: systematic review or meta-analysis of randomised controlled trials

Ib: at least one randomised controlled trial

IIa: at least one well-designed controlled study without randomisation

IIb: at least one well-designed quasi-experimental study, such as a cohort study

III: well-designed non-experimental descriptive studies, such as comparative studies, correlation studies, case–control studies and case series

IV: expert committee reports, opinions and/or clinical experience of respected authorities

V: anecdotal / opinion 

Many scholars in the social scieces believe it is unfeasible and undesirable for management research to adopt the hierarchy of evidence.  All academic fields are different and every study is different. What consitutes evidence and 'good quality research' depends entirely on its appropriateness to question being asked (Boaz and Ashby, 2003).  As noted by Briner, Rousseau and Denyer (2009, Academy of Management Perspectives - forthcoming):

 'If the question is “what effect does intervention X have on outcome Y?” then a meta-analysis of randomized trials may indeed be the best possible evidence. Similarly, if the question is about tools for personnel selection then meta-analysis of predictive validity studies is likely to be appropriate and may provide relevant evidence for practitioners (e.g., Le, Oh, Shaffer and Schmidt, 2007).  For other questions, longitudinal studies or quasi-experiments might be the best evidence available.  If, on the other hand, the question is “how do women interpret their role on male dominated boards?” then qualitative data will form the best evidence.  If the question is “why or how does goal setting result in higher team performance?” then we need theory as well as evidence from which we can infer processes.' 

Therefore, 'good quality' research can be quantitative, qualitative or theoretical.  It is possible to create a set of generic quality criteria that are applicable to all study designs.  For example,

• Was an explicit account of the theoretical framework given?
• Is there a succinct statement of objectives or research questions?
• Is there a clear description of the context?
• How was the sample chosen, is it adequate?
• Was there a clear description of data collection methods, were they appropriate?
• Was a there clear description of data analysis methods, were they appropriate?
• Does the research provide an audit trail from the raw data (numbers, quotations or examples), to an analysis and interpretation of the meaning and significance of it?
• Are the findings relevant to policy / practice, do they provide guidance for future research?

However, these should be supplemented with some specific appraisal criteria for the particular methods employed in the original studies.  We do not believe that it is possible to judge qualitative research using criteria designed to evaluate quantitative research and vice versa.

Quality checklists for different research designs are freely available on the internet, such as those produced by the Critical Appraisal Skills Programme.

Given that different subfields of management will have different perspectives on research quality, it is also important to take this into consideration.  You should investigate the guidelines for reviewers in a number of journals in your subfield of management and organisation studies. 

Some journals produce comprehensive guidance of appraising certain forms of research.  For example, the Journal of Occupational and Organisational Psychology has an extensive set of  - criteria for evaluating papers using qualitative research methods.     If you want to construct some criteria for evaluating theoretical / conceptual papers you could  refer to a journal such as the Academy of Management Review:  This journal's Guidelines for Reviewers are provided below:

• Does the paper create, extend or advance management theory in a significant way? 
• Is the topic important and interesting? Does the manuscript pass the “so what”test? 
• Are the central constructs defined clearly? Are the underlying causal mechanisms behind proposed relationships explained clearly? 
• Are underlying assumptions clearly recognized and discussed? 
• Does the manuscript contain a well-developed and articulated theoretical framework? 
• Do the propositions (if applicable) logically flow from the theory? 
• Is relevant literature cited accurately? If relevant literature is missing, can you point the authors toward that literature? 
• Does the paper have clear implications for future research? 
• Is the paper’s contribution commensurate with its length? 

By combining general quality criteria and guidance from key journals in the field it is possible to create a bespoke and practical quality appraisal tool. 

The quality of a paper is not only related to methodological issues but also its contribution to the field.  Further, some papers fail to report the methods of data collection and data analysis in sufficient detail to conduct an evaluation.  Whilst this demonstrates poor scholarship, the paper may actually provide new insights.  In such cases you may not wish to exclude papers that are deemed to be low quality, but to evaluate and report on the study and its limitations.  If studies are excluded on the basis of quality it is crucial for the reviewer to document and justify the reasons for exclusion. 

General quality criteria for all papers	Yes / No	Comments
1. e.g. Does the paper address a clearly focused issue? 2. 3. etc
Quality criteria for quantitative papers
1. 2. 3. etc
Quality criteria for qualitative papers
1. 2. 3. etc
Quality criteria for conceptual / theoretical papers
1. 2. 3. etc
Does the paper make a contribution to the field? Does it help address the review questions?
Should the paper be included in the review

 To download and use this document – click  here

 

References

Boaz, A. and Ashby, D. (2003) 'Fit for purpose? Assessing research quality for evidence based policy and practice', London: Queen Mary, University of London

Briner, R.B., Denyer, D. and Rousseau, D.M. (2009) 'Evidence-Based Management: Concept clean up time?', The Academy of Management Perspectives, Vol. 23, No. 4, 19-32.

The text on this page was created by Professor David Denyer, Professor of Organizational Change, Cranfield School of Management.