About this Resource
How systematic should you be?
The stages of a systematic review
1. Produce a review protocol / plan
2. Assemble a review group / advisory group
3. Formulate review question(s)
4. Conduct a thorough search
5. Select relevant studies
6. Appraise the quality of studies
7. Extract information from individual studies
8. Synthesise studies
9. Report what is known and not known
10. Inform research, policy and practice
Protocol driven search 


A protocol-driven search involves the following steps:

  1. identify keywords
  2. group keywords into search strings
  3. select databases
  4. run searches
  5. run additional searches for key authors
  6. save your searches
  7. document the results

Identify keywords

Make a list of keywords that relate to your topic.  Return to the maps that you produced in the Mapping your field exercise are there keywords for each of these circles?

One way that you can report the process is to create a 3x3 table.  Place words of similar meaning in each of the cells and seperate each of the words with OR (in capitals).  Read the abstracts of a few key papers in your field and identify the keywords used by the authors.  Also, refer to a thesaurus - an online version can be fast and effective (see for example, thesaurus.com).  Once you have exhausted all potential keywords insert a bracket at the start and end of the search string - see the example below which was a reviewing of the adoption of promising practices by organisations:

To download and use this document – click  here 

Group keywords into search strings

The aim is to create a small number of search strings that cover a large number of keywords.  You can then copy and paste the entire search string into a database.  You can also combine the search strings by inserting an AND (in capitals) command.  For example:

(best practice? OR management fashion? OR management fad? OR process innovation OR good practice? OR promising practice) AND (adoption OR adaptation OR implement? OR use)

It is important to run the searches on all relevant combinations.  However, some combinations are not appropriate - in the example above it would be unwise to run a search for (culture OR assimilation OR absoption) AND (perception OR relevance) as this would be too broad and would likely yield a large number of irrelevant studies. 

You will notice in the example above that truncation characters (a question mark) are also included.  The following is a quick guide to search operators. 

         Simple operators include:

        Truncation characters – ‘*’, ‘?’; e.g. team? - searches for documents which contain the term team, teams, teamwork, teamworking etc

        Word association – ‘w’ or ‘near’; e.g. high(w)reliability; high(near)reliability

        Exact phrase – ‘ ‘, “ “; e.g. ‘high reliability’; “high reliability”

         Boolean Logic includes:

        OR  e.g. team* OR group* searches for either term in a document

        AND e.g. ‘high reliability’ AND organi*ation  searches for both terms in a document

        AND NOT e.g. team* AND NOT virtual* searches for documents which contain the term team, teams, teamwork, teamworking but not virtual

         By combining Boolean operators with parentheses complex searches can be constructed:-

        ‘high reliability’ AND (team* OR group*) AND NOT virtual 

Significant time is needed to be spent on constructing the search strings, for this investment can improve the efficiency of the search.  It is important to note that different citation databases use different search conventions.  It is wise to seek advice from a librarian if you are unfamilar with database searching.

Select databases

Once you have created a set of search strings it is worthwhile running some exploratory searches in a range of different databases.  This can help you decide which databases are likely to be the most relevant.  Individual citation databases cover only a limited number of journals.  Therefore, it is crucial that you run your searches in multiple databases.  For many topics in management and organisation studies it is senible to run searches in at least three of the general databases such as EBSCO, PROQUEST or WEB OF KNOWLEDGE.  It is also important to run searches in onre or more specialist database.  For example, for an review in the field of organisational bevhavior it would be sensible to run a search in PSYCINFO.  Other fields may have their own specialist databases. Oncemore, it would be wise to seek advice from a librarian regarding the most appropriate databases for your particular field.

Run searches

The processes of running searches should be relatively straightforward.  In this example we are going to explore searching using three search strings taken from the table above:

String 1: (best practice? OR management fashion? OR management fad? OR process innovation OR good practice? OR promising practice)

String 2: (adoption OR adaptation OR implement? OR use)

String 3: (performance OR success? OR effectiveness OR impact? OR competitiveness OR failure? OR benefits OR advantages OR disadvantages OR profit? OR productivity OR cost?)

To search using search string 1 combined with 2 you simply copy the following search string into the database:

(best practice? OR management fashion? OR management fad? OR process innovation OR good practice? OR promising practice) AND (adoption OR adaptation OR implement? OR use)

 

You can see that the search was performed using the search modes - 'Boolean/Phrase'.  This was selected in the advanced search tab.  If this was not select the default search mode - 'Find all my search terms' would have been selected resulting in 4692 articles being located.  This highlights how precise and consistent you need to be when searching - failure to tick an appropriate box can lead to sigificant errors.

 The total number of articles located was 1881.  You will notice that on the left hand navigation area 'Academic Journals' has been selected, reducing the number of articles to 739.  Whether or not your restrict the search to academic articles only is of course a decision that you need to make and justify in your protocol / plan. 

A second search is then performed using by combining search strings (2 AND 3) :

(best practice? OR management fashion? OR management fad? OR process innovation OR good practice? OR promising practice) AND (performance OR success? OR effectiveness OR impact? OR competitiveness OR failure? OR benefits OR advantages OR disadvantages OR profit? OR productivity OR cost?)

This provides a further 1254 articles.  You now have to make a choice.  Do you scan the titles and abstracts of the 739 texts located using strings (1 AND 2) and then the 1254 located using (2 AND 3) or refine your search further? 

You could combine all three strings (1 AND 2 AND 3):

(best practice? OR management fashion? OR management fad? OR process innovation OR good practice? OR promising practice) AND (adoption OR adaptation OR implement? OR use) AND (performance OR success? OR effectiveness OR impact? OR competitiveness OR failure? OR benefits OR advantages OR disadvantages OR profit? OR productivity OR cost?)

This search provides 343 articles.  Whilst scanning 343 titles and abstracts sounds more appealing you need to remeber that the more precise you make your search strings the more likely you will miss important articles.  We would recommend that you scan the 739 and 1254 articles.  This might take a whole morning.  If you repeat this search across 5 databases you could still complete your entire protocol driven search in a week.   We feel that this would be a good investment of your time.  

If the results of a search provide more than 1500 articles you may need to add an additional string.  You could also add an AND NOT command to delimit the search. If you results reveal a small number of articles (perhaps less than 50) your search strings may be too restrictive. 

Remember that you will not be able to locate all of the studies relevant to the review questions through the protocol driven search alone.   Therefore, rather than trying to perfect your database search your precious time may be better spent complementing the the protocol driven search with an approach such as snowballing.  

Run additional searches for key authors

When you have completed the search string searches in all of the chosen databases you can run some additional searches for key authors in your field.  In the image above you will see that the author is 'Walker, Warren E'.  If you click on the link to the article the details of the paper will be revealed.  You can then click the author's name and the other papers writen by this author will be provided.  You can also search for names by placing the author's name in the search box and in the 'Select a Field' box choose 'Author'.   You may need to experiment with this.  For example: 'denyer, david' is the correct format in EBSCO, 'denyer, d' does not work and 'denyer' provides 222 articles as it includes papers written by authors with the same surname.   Other databases may require a different format. 

Save your searches

Many databases allow you to log in, set up a personal account and save your searches.  This enables you to retrieve and reuse your search strings.  Saved searches can also be set up to send you email alerts - an automated e-mail with any new results.  See the section on keep up to date for futher details. 

Document your searches

It is important to document your searches so that you can explain what you have done and can return to the searches at a later date to keep up to date.  The table below provides an example of how the results of searches could be reported. 

   

Interface: EBSCO host
Database: Business Source Complete
Search Screen: Advanced Search

Date of search: October 17, 2009
Source type: Academic Journals
Years: All
Search modes: Boolean/Phrase
Saved search ID or location: S1

 

 Search Strings

Results

Relevant

String 1 only

(best practice? OR management fashion? OR management fad? OR process innovation OR good practice? OR promising practice)

3156

Refine search

String 2 only

(adoption OR adaptation OR implement? OR use)

265306

Refine search

String 3 only

(performance OR success? OR effectiveness OR impact? OR competitiveness OR failure? OR benefits OR advantages OR disadvantages OR profit? OR productivity OR cost?)

402666

Refine search

Strings 1 and 2

(best practice? OR management fashion? OR management fad? OR process innovation OR good practice? OR promising practice) AND (adoption OR adaptation OR implement? OR use)

739

126

Strings 1 and 3

(best practice? OR management fashion? OR management fad? OR process innovation OR good practice? OR promising practice) AND (performance OR success? OR effectiveness OR impact? OR competitiveness OR failure? OR benefits OR advantages OR disadvantages OR profit? OR productivity OR cost?)

1254

147

Strings 2 and 3

(adoption OR adaptation OR implement? OR use) AND (performance OR success? OR effectiveness OR impact? OR competitiveness OR failure? OR benefits OR advantages OR disadvantages OR profit? OR productivity OR cost?)

78123

Refine search

Strings 1, 2 and 3

(best practice? OR management fashion? OR management fad? OR process innovation OR good practice? OR promising practice) AND (adoption OR adaptation OR implement? OR use) AND (performance OR success? OR effectiveness OR impact? OR competitiveness OR failure? OR benefits OR advantages OR disadvantages OR profit? OR productivity OR cost?)

343

N/A – since the results of strings (1 AND 2) and (2 AND 3) were scanned for relevance all articles in this set will all be duplicates.

Total

 273

  To download and use this document – click  here