literature review process selection

• University of Texas Libraries

Literature Reviews

Steps in the literature review process.

• What is a literature review?
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support
• You may need to some exploratory searching of the literature to get a sense of scope, to determine whether you need to narrow or broaden your focus
• Identify databases that provide the most relevant sources, and identify relevant terms (controlled vocabularies) to add to your search strategy
• Finalize your research question
• Think about relevant dates, geographies (and languages), methods, and conflicting points of view
• Conduct searches in the published literature via the identified databases
• Check to see if this topic has been covered in other discipline's databases
• Examine the citations of on-point articles for keywords, authors, and previous research (via references) and cited reference searching.
• Save your search results in a citation management tool (such as Zotero, Mendeley or EndNote)
• De-duplicate your search results
• Make sure that you've found the seminal pieces -- they have been cited many times, and their work is considered foundational 
• Check with your professor or a librarian to make sure your search has been comprehensive
• Evaluate the strengths and weaknesses of individual sources and evaluate for bias, methodologies, and thoroughness
• Group your results in to an organizational structure that will support why your research needs to be done, or that provides the answer to your research question  
• Develop your conclusions
• Are there gaps in the literature?
• Where has significant research taken place, and who has done it?
• Is there consensus or debate on this topic?
• Which methodological approaches work best?
• For example: Background, Current Practices, Critics and Proponents, Where/How this study will fit in 
• Organize your citations and focus on your research question and pertinent studies
• Compile your bibliography

Note: The first four steps are the best points at which to contact a librarian. Your librarian can help you determine the best databases to use for your topic, assess scope, and formulate a search strategy.

Videos Tutorials about Literature Reviews

This 4.5 minute video from Academic Education Materials has a Creative Commons License and a British narrator.

Recommended Reading

• Last Updated: Oct 26, 2022 2:49 PM
• URL: https://guides.lib.utexas.edu/literaturereviews

Research Process :: Step by Step

• Introduction
• Select Topic
• Identify Keywords
• Background Information
• Develop Research Questions
• Refine Topic
• Search Strategy
• Popular Databases
• Evaluate Sources
• Types of Periodicals
• Reading Scholarly Articles
• Primary & Secondary Sources
• Organize / Take Notes
• Writing & Grammar Resources
• Annotated Bibliography
• Literature Review
• Citation Styles
• Paraphrasing
• Privacy / Confidentiality
• Research Process
• Selecting Your Topic
• Identifying Keywords
• Gathering Background Info
• Evaluating Sources

Organize the literature review into sections that present themes or identify trends, including relevant theory. You are not trying to list all the material published, but to synthesize and evaluate it according to the guiding concept of your thesis or research question.  

What is a literature review?

A literature review is an account of what has been published on a topic by accredited scholars and researchers. Occasionally you will be asked to write one as a separate assignment, but more often it is part of the introduction to an essay, research report, or thesis. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries

A literature review must do these things:

• be organized around and related directly to the thesis or research question you are developing
• synthesize results into a summary of what is and is not known
• identify areas of controversy in the literature
• formulate questions that need further research

Ask yourself questions like these:

• What is the specific thesis, problem, or research question that my literature review helps to define?
• What type of literature review am I conducting? Am I looking at issues of theory? methodology? policy? quantitative research (e.g. on the effectiveness of a new procedure)? qualitative research (e.g., studies of loneliness among migrant workers)?
• What is the scope of my literature review? What types of publications am I using (e.g., journals, books, government documents, popular media)? What discipline am I working in (e.g., nursing psychology, sociology, medicine)?
• How good was my information seeking? Has my search been wide enough to ensure I've found all the relevant material? Has it been narrow enough to exclude irrelevant material? Is the number of sources I've used appropriate for the length of my paper?
• Have I critically analyzed the literature I use? Do I follow through a set of concepts and questions, comparing items to each other in the ways they deal with them? Instead of just listing and summarizing items, do I assess them, discussing strengths and weaknesses?
• Have I cited and discussed studies contrary to my perspective?
• Will the reader find my literature review relevant, appropriate, and useful?

Ask yourself questions like these about each book or article you include:

• Has the author formulated a problem/issue?
• Is it clearly defined? Is its significance (scope, severity, relevance) clearly established?
• Could the problem have been approached more effectively from another perspective?
• What is the author's research orientation (e.g., interpretive, critical science, combination)?
• What is the author's theoretical framework (e.g., psychological, developmental, feminist)?
• What is the relationship between the theoretical and research perspectives?
• Has the author evaluated the literature relevant to the problem/issue? Does the author include literature taking positions she or he does not agree with?
• In a research study, how good are the basic components of the study design (e.g., population, intervention, outcome)? How accurate and valid are the measurements? Is the analysis of the data accurate and relevant to the research question? Are the conclusions validly based upon the data and analysis?
• In material written for a popular readership, does the author use appeals to emotion, one-sided examples, or rhetorically-charged language and tone? Is there an objective basis to the reasoning, or is the author merely "proving" what he or she already believes?
• How does the author structure the argument? Can you "deconstruct" the flow of the argument to see whether or where it breaks down logically (e.g., in establishing cause-effect relationships)?
• In what ways does this book or article contribute to our understanding of the problem under study, and in what ways is it useful for practice? What are the strengths and limitations?
• How does this book or article relate to the specific thesis or question I am developing?

Text written by Dena Taylor, Health Sciences Writing Centre, University of Toronto

http://www.writing.utoronto.ca/advice/specific-types-of-writing/literature-review

• << Previous: Annotated Bibliography
• Next: Step 5: Cite Sources >>
• Last Updated: Apr 19, 2024 12:43 PM
• URL: https://libguides.uta.edu/researchprocess

University of Texas Arlington Libraries 702 Planetarium Place · Arlington, TX 76019 · 817-272-3000

• Internet Privacy
• Accessibility
• Problems with a guide? Contact Us.

• Reserve a study room
• Library Account
• Undergraduate Students
• Graduate Students
• Faculty & Staff

How to Conduct a Literature Review (Health Sciences and Beyond)

• What is a Literature Review?
• Developing a Research Question

Selection Criteria

Inclusion criteria, exclusion criteria.

• Database Search
• Documenting Your Search
• Organize Key Findings
• Reference Management

You may want to think about criteria that will be used to select articles for your literature review based on your research question.  These are commonly known as  inclusion criteria  and  exclusion criteria .  Be aware that you may introduce bias into the final review if these are not used thoughtfully.

Inclusion criteria are the elements of an article that must be present in order for it to be eligible for inclusion in a literature review.  Some examples are:

• Included studies must have compared certain treatments
• Included studies must be experimental
• Included studies must have been published in the last 5 years

Exclusion criteria are the elements of an article that disqualify the study from inclusion in a literature review.  Some examples are:

• Study used an observational design
• Study used a qualitative methodology
• Study was published more than 5 years ago
• Study was published in a language other than English
• << Previous: Developing a Research Question
• Next: Databases >>
• Last Updated: Mar 15, 2024 12:22 PM
• URL: https://guides.library.vcu.edu/health-sciences-lit-review

• University of Michigan Library
• Research Guides

Systematic Reviews

• Selection Process
• Work with a Search Expert
• Covidence Review Software
• Types of Reviews
• Evidence in a Systematic Review
• Information Sources
• Search Strategy
• Managing Records

Study Selection Guidance

Typical screening and selection process, systematic review tools.

• Data Collection Process
• Study Risk of Bias Assessment
• Reporting Results
• For Search Professionals

The eligibility criteria (also referred to as inclusion and exclusion criteria) for your systematic review should be developed before you begin screening articles. The PRISMA-p Protocols extension and systematic review protocol templates include eligibility criteria among the methods to plan.

Best practices for study selection

• Use explicit, pre-defined eligibility criteria to guide screening
• Have two independent reviewers select eligible studies to reduce the risk of mistakes and personal biases

Related sections of the Cochrane Handbook

As you develop your protocol and methodology, the following sections of the online version of the  Cochrane Handbook for Systematic Reviews of Interventions  provide detailed guidance on study selection:

• Chapter 3: Defining the criteria for including studies and how the will be grouped for the synthesis
• Section 4.6 Selecting Studies  of Chapter 4: Searching for and selecting studies 

For most evidence synthesis projects, study selection will occur in two phases: title and abstract screening and full text review.

Title and Abstract Screening

Review the title and abstract for each record retrieved to determine whether it should be excluded based on the eligibility criteria. If the relevance is unclear from the title and abstract, it is best to move the record forward for full text review.

During title and abstract screening, reviewers will document their overall decision (for example, include in full text review, exclude from project, or maybe). The title and abstract screening form can also include sub-questions related to the inclusion and exclusion criteria. Responses to these prompts are useful reminders when discussing conflicts. 

Full Text Review

Retrieve and review the full text of all records advanced from the title and abstract screening process. Use the pre-defined eligibility criteria to select studies for inclusion in the review.

During full text review, document the overall decision and exclusion reason for each excluded study.

Systematic review tools have been designed to help streamline systematic review processes. The Systematic Review Toolbox catalogs available tools; the entry for each tool provides a description, supported processes, related papers, and basic costs (i.e., free vs. subscription required). Use the filters under "Advanced Search" to view tools that support specific systematic review processes. 

Popular tools with screening features include:

• DistillerSR (subscription required)
• Covidence (subscription required)
• Rayyan (free and subscription options available)
• Abstrakr (free)

Select the best systematic review tool for your team by considering your team's needs and comparing features, usability, and costs of various tools.

Recent Tool Evaluations

• Harrison H, Griffin SJ, Kuhn I, Usher-Smith JA. Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation.  BMC Med Res Methodol . 2020;20(1):7. Published 2020 Jan 13. doi: 10.1186/s12874-020-0897-3
• Kohl C, McIntosh EJ, Unger S, Haddaway NR, Kecke S, Schiemann J, Wilhelm R. Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools. Environmental Evidence . 2018 Dec;7(1):1-7. doi: 10.1186/s13750-018-0115-5

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• << Previous: Theoretical Framework
• Next: Citation Tracking >>
• Last Updated: Apr 29, 2024 1:49 PM
• URL: https://libguides.usc.edu/writingguide

• Open access
• Published: 14 August 2018

Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

• Chris Cooper   ORCID: orcid.org/0000-0003-0864-5607 1 ,
• Andrew Booth 2 ,
• Jo Varley-Campbell 1 ,
• Nicky Britten 3 &
• Ruth Garside 4  

BMC Medical Research Methodology volume  18 , Article number:  85 ( 2018 ) Cite this article

202k Accesses

203 Citations

118 Altmetric

Metrics details

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving readers clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before.

The purpose of this review is to determine if a shared model of the literature searching process can be detected across systematic review guidance documents and, if so, how this process is reported in the guidance and supported by published studies.

A literature review.

Two types of literature were reviewed: guidance and published studies. Nine guidance documents were identified, including: The Cochrane and Campbell Handbooks. Published studies were identified through ‘pearl growing’, citation chasing, a search of PubMed using the systematic review methods filter, and the authors’ topic knowledge.

The relevant sections within each guidance document were then read and re-read, with the aim of determining key methodological stages. Methodological stages were identified and defined. This data was reviewed to identify agreements and areas of unique guidance between guidance documents. Consensus across multiple guidance documents was used to inform selection of ‘key stages’ in the process of literature searching.

Eight key stages were determined relating specifically to literature searching in systematic reviews. They were: who should literature search, aims and purpose of literature searching, preparation, the search strategy, searching databases, supplementary searching, managing references and reporting the search process.

Conclusions

Eight key stages to the process of literature searching in systematic reviews were identified. These key stages are consistently reported in the nine guidance documents, suggesting consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews. Further research to determine the suitability of using the same process of literature searching for all types of systematic review is indicated.

Peer Review reports

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving review stakeholders clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before. This is in contrast to the information science literature, which has developed information processing models as an explicit basis for dialogue and empirical testing. Without an explicit model, research in the process of systematic literature searching will remain immature and potentially uneven, and the development of shared information models will be assumed but never articulated.

One way of developing such a conceptual model is by formally examining the implicit “programme theory” as embodied in key methodological texts. The aim of this review is therefore to determine if a shared model of the literature searching process in systematic reviews can be detected across guidance documents and, if so, how this process is reported and supported.

Identifying guidance

Key texts (henceforth referred to as “guidance”) were identified based upon their accessibility to, and prominence within, United Kingdom systematic reviewing practice. The United Kingdom occupies a prominent position in the science of health information retrieval, as quantified by such objective measures as the authorship of papers, the number of Cochrane groups based in the UK, membership and leadership of groups such as the Cochrane Information Retrieval Methods Group, the HTA-I Information Specialists’ Group and historic association with such centres as the UK Cochrane Centre, the NHS Centre for Reviews and Dissemination, the Centre for Evidence Based Medicine and the National Institute for Clinical Excellence (NICE). Coupled with the linguistic dominance of English within medical and health science and the science of systematic reviews more generally, this offers a justification for a purposive sample that favours UK, European and Australian guidance documents.

Nine guidance documents were identified. These documents provide guidance for different types of reviews, namely: reviews of interventions, reviews of health technologies, reviews of qualitative research studies, reviews of social science topics, and reviews to inform guidance.

Whilst these guidance documents occasionally offer additional guidance on other types of systematic reviews, we have focused on the core and stated aims of these documents as they relate to literature searching. Table  1 sets out: the guidance document, the version audited, their core stated focus, and a bibliographical pointer to the main guidance relating to literature searching.

Once a list of key guidance documents was determined, it was checked by six senior information professionals based in the UK for relevance to current literature searching in systematic reviews.

Identifying supporting studies

In addition to identifying guidance, the authors sought to populate an evidence base of supporting studies (henceforth referred to as “studies”) that contribute to existing search practice. Studies were first identified by the authors from their knowledge on this topic area and, subsequently, through systematic citation chasing key studies (‘pearls’ [ 1 ]) located within each key stage of the search process. These studies are identified in Additional file  1 : Appendix Table 1. Citation chasing was conducted by analysing the bibliography of references for each study (backwards citation chasing) and through Google Scholar (forward citation chasing). A search of PubMed using the systematic review methods filter was undertaken in August 2017 (see Additional file 1 ). The search terms used were: (literature search*[Title/Abstract]) AND sysrev_methods[sb] and 586 results were returned. These results were sifted for relevance to the key stages in Fig.  1 by CC.

The key stages of literature search guidance as identified from nine key texts

Extracting the data

To reveal the implicit process of literature searching within each guidance document, the relevant sections (chapters) on literature searching were read and re-read, with the aim of determining key methodological stages. We defined a key methodological stage as a distinct step in the overall process for which specific guidance is reported, and action is taken, that collectively would result in a completed literature search.

The chapter or section sub-heading for each methodological stage was extracted into a table using the exact language as reported in each guidance document. The lead author (CC) then read and re-read these data, and the paragraphs of the document to which the headings referred, summarising section details. This table was then reviewed, using comparison and contrast to identify agreements and areas of unique guidance. Consensus across multiple guidelines was used to inform selection of ‘key stages’ in the process of literature searching.

Having determined the key stages to literature searching, we then read and re-read the sections relating to literature searching again, extracting specific detail relating to the methodological process of literature searching within each key stage. Again, the guidance was then read and re-read, first on a document-by-document-basis and, secondly, across all the documents above, to identify both commonalities and areas of unique guidance.

Results and discussion

Our findings.

We were able to identify consensus across the guidance on literature searching for systematic reviews suggesting a shared implicit model within the information retrieval community. Whilst the structure of the guidance varies between documents, the same key stages are reported, even where the core focus of each document is different. We were able to identify specific areas of unique guidance, where a document reported guidance not summarised in other documents, together with areas of consensus across guidance.

Unique guidance

Only one document provided guidance on the topic of when to stop searching [ 2 ]. This guidance from 2005 anticipates a topic of increasing importance with the current interest in time-limited (i.e. “rapid”) reviews. Quality assurance (or peer review) of literature searches was only covered in two guidance documents [ 3 , 4 ]. This topic has emerged as increasingly important as indicated by the development of the PRESS instrument [ 5 ]. Text mining was discussed in four guidance documents [ 4 , 6 , 7 , 8 ] where the automation of some manual review work may offer efficiencies in literature searching [ 8 ].

Agreement between guidance: Defining the key stages of literature searching

Where there was agreement on the process, we determined that this constituted a key stage in the process of literature searching to inform systematic reviews.

From the guidance, we determined eight key stages that relate specifically to literature searching in systematic reviews. These are summarised at Fig. 1 . The data extraction table to inform Fig. 1 is reported in Table  2 . Table 2 reports the areas of common agreement and it demonstrates that the language used to describe key stages and processes varies significantly between guidance documents.

For each key stage, we set out the specific guidance, followed by discussion on how this guidance is situated within the wider literature.

Key stage one: Deciding who should undertake the literature search

The guidance.

Eight documents provided guidance on who should undertake literature searching in systematic reviews [ 2 , 4 , 6 , 7 , 8 , 9 , 10 , 11 ]. The guidance affirms that people with relevant expertise of literature searching should ‘ideally’ be included within the review team [ 6 ]. Information specialists (or information scientists), librarians or trial search co-ordinators (TSCs) are indicated as appropriate researchers in six guidance documents [ 2 , 7 , 8 , 9 , 10 , 11 ].

How the guidance corresponds to the published studies

The guidance is consistent with studies that call for the involvement of information specialists and librarians in systematic reviews [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ] and which demonstrate how their training as ‘expert searchers’ and ‘analysers and organisers of data’ can be put to good use [ 13 ] in a variety of roles [ 12 , 16 , 20 , 21 , 24 , 25 , 26 ]. These arguments make sense in the context of the aims and purposes of literature searching in systematic reviews, explored below. The need for ‘thorough’ and ‘replicable’ literature searches was fundamental to the guidance and recurs in key stage two. Studies have found poor reporting, and a lack of replicable literature searches, to be a weakness in systematic reviews [ 17 , 18 , 27 , 28 ] and they argue that involvement of information specialists/ librarians would be associated with better reporting and better quality literature searching. Indeed, Meert et al. [ 29 ] demonstrated that involving a librarian as a co-author to a systematic review correlated with a higher score in the literature searching component of a systematic review [ 29 ]. As ‘new styles’ of rapid and scoping reviews emerge, where decisions on how to search are more iterative and creative, a clear role is made here too [ 30 ].

Knowing where to search for studies was noted as important in the guidance, with no agreement as to the appropriate number of databases to be searched [ 2 , 6 ]. Database (and resource selection more broadly) is acknowledged as a relevant key skill of information specialists and librarians [ 12 , 15 , 16 , 31 ].

Whilst arguments for including information specialists and librarians in the process of systematic review might be considered self-evident, Koffel and Rethlefsen [ 31 ] have questioned if the necessary involvement is actually happening [ 31 ].

Key stage two: Determining the aim and purpose of a literature search

The aim: Five of the nine guidance documents use adjectives such as ‘thorough’, ‘comprehensive’, ‘transparent’ and ‘reproducible’ to define the aim of literature searching [ 6 , 7 , 8 , 9 , 10 ]. Analogous phrases were present in a further three guidance documents, namely: ‘to identify the best available evidence’ [ 4 ] or ‘the aim of the literature search is not to retrieve everything. It is to retrieve everything of relevance’ [ 2 ] or ‘A systematic literature search aims to identify all publications relevant to the particular research question’ [ 3 ]. The Joanna Briggs Institute reviewers’ manual was the only guidance document where a clear statement on the aim of literature searching could not be identified. The purpose of literature searching was defined in three guidance documents, namely to minimise bias in the resultant review [ 6 , 8 , 10 ]. Accordingly, eight of nine documents clearly asserted that thorough and comprehensive literature searches are required as a potential mechanism for minimising bias.

The need for thorough and comprehensive literature searches appears as uniform within the eight guidance documents that describe approaches to literature searching in systematic reviews of effectiveness. Reviews of effectiveness (of intervention or cost), accuracy and prognosis, require thorough and comprehensive literature searches to transparently produce a reliable estimate of intervention effect. The belief that all relevant studies have been ‘comprehensively’ identified, and that this process has been ‘transparently’ reported, increases confidence in the estimate of effect and the conclusions that can be drawn [ 32 ]. The supporting literature exploring the need for comprehensive literature searches focuses almost exclusively on reviews of intervention effectiveness and meta-analysis. Different ‘styles’ of review may have different standards however; the alternative, offered by purposive sampling, has been suggested in the specific context of qualitative evidence syntheses [ 33 ].

What is a comprehensive literature search?

Whilst the guidance calls for thorough and comprehensive literature searches, it lacks clarity on what constitutes a thorough and comprehensive literature search, beyond the implication that all of the literature search methods in Table 2 should be used to identify studies. Egger et al. [ 34 ], in an empirical study evaluating the importance of comprehensive literature searches for trials in systematic reviews, defined a comprehensive search for trials as:

a search not restricted to English language;

where Cochrane CENTRAL or at least two other electronic databases had been searched (such as MEDLINE or EMBASE); and

at least one of the following search methods has been used to identify unpublished trials: searches for (I) conference abstracts, (ii) theses, (iii) trials registers; and (iv) contacts with experts in the field [ 34 ].

Tricco et al. (2008) used a similar threshold of bibliographic database searching AND a supplementary search method in a review when examining the risk of bias in systematic reviews. Their criteria were: one database (limited using the Cochrane Highly Sensitive Search Strategy (HSSS)) and handsearching [ 35 ].

Together with the guidance, this would suggest that comprehensive literature searching requires the use of BOTH bibliographic database searching AND supplementary search methods.

Comprehensiveness in literature searching, in the sense of how much searching should be undertaken, remains unclear. Egger et al. recommend that ‘investigators should consider the type of literature search and degree of comprehension that is appropriate for the review in question, taking into account budget and time constraints’ [ 34 ]. This view tallies with the Cochrane Handbook, which stipulates clearly, that study identification should be undertaken ‘within resource limits’ [ 9 ]. This would suggest that the limitations to comprehension are recognised but it raises questions on how this is decided and reported [ 36 ].

What is the point of comprehensive literature searching?

The purpose of thorough and comprehensive literature searches is to avoid missing key studies and to minimize bias [ 6 , 8 , 10 , 34 , 37 , 38 , 39 ] since a systematic review based only on published (or easily accessible) studies may have an exaggerated effect size [ 35 ]. Felson (1992) sets out potential biases that could affect the estimate of effect in a meta-analysis [ 40 ] and Tricco et al. summarize the evidence concerning bias and confounding in systematic reviews [ 35 ]. Egger et al. point to non-publication of studies, publication bias, language bias and MEDLINE bias, as key biases [ 34 , 35 , 40 , 41 , 42 , 43 , 44 , 45 , 46 ]. Comprehensive searches are not the sole factor to mitigate these biases but their contribution is thought to be significant [ 2 , 32 , 34 ]. Fehrmann (2011) suggests that ‘the search process being described in detail’ and that, where standard comprehensive search techniques have been applied, increases confidence in the search results [ 32 ].

Does comprehensive literature searching work?

Egger et al., and other study authors, have demonstrated a change in the estimate of intervention effectiveness where relevant studies were excluded from meta-analysis [ 34 , 47 ]. This would suggest that missing studies in literature searching alters the reliability of effectiveness estimates. This is an argument for comprehensive literature searching. Conversely, Egger et al. found that ‘comprehensive’ searches still missed studies and that comprehensive searches could, in fact, introduce bias into a review rather than preventing it, through the identification of low quality studies then being included in the meta-analysis [ 34 ]. Studies query if identifying and including low quality or grey literature studies changes the estimate of effect [ 43 , 48 ] and question if time is better invested updating systematic reviews rather than searching for unpublished studies [ 49 ], or mapping studies for review as opposed to aiming for high sensitivity in literature searching [ 50 ].

Aim and purpose beyond reviews of effectiveness

The need for comprehensive literature searches is less certain in reviews of qualitative studies, and for reviews where a comprehensive identification of studies is difficult to achieve (for example, in Public health) [ 33 , 51 , 52 , 53 , 54 , 55 ]. Literature searching for qualitative studies, and in public health topics, typically generates a greater number of studies to sift than in reviews of effectiveness [ 39 ] and demonstrating the ‘value’ of studies identified or missed is harder [ 56 ], since the study data do not typically support meta-analysis. Nussbaumer-Streit et al. (2016) have registered a review protocol to assess whether abbreviated literature searches (as opposed to comprehensive literature searches) has an impact on conclusions across multiple bodies of evidence, not only on effect estimates [ 57 ] which may develop this understanding. It may be that decision makers and users of systematic reviews are willing to trade the certainty from a comprehensive literature search and systematic review in exchange for different approaches to evidence synthesis [ 58 ], and that comprehensive literature searches are not necessarily a marker of literature search quality, as previously thought [ 36 ]. Different approaches to literature searching [ 37 , 38 , 59 , 60 , 61 , 62 ] and developing the concept of when to stop searching are important areas for further study [ 36 , 59 ].

The study by Nussbaumer-Streit et al. has been published since the submission of this literature review [ 63 ]. Nussbaumer-Streit et al. (2018) conclude that abbreviated literature searches are viable options for rapid evidence syntheses, if decision-makers are willing to trade the certainty from a comprehensive literature search and systematic review, but that decision-making which demands detailed scrutiny should still be based on comprehensive literature searches [ 63 ].

Key stage three: Preparing for the literature search

Six documents provided guidance on preparing for a literature search [ 2 , 3 , 6 , 7 , 9 , 10 ]. The Cochrane Handbook clearly stated that Cochrane authors (i.e. researchers) should seek advice from a trial search co-ordinator (i.e. a person with specific skills in literature searching) ‘before’ starting a literature search [ 9 ].

Two key tasks were perceptible in preparing for a literature searching [ 2 , 6 , 7 , 10 , 11 ]. First, to determine if there are any existing or on-going reviews, or if a new review is justified [ 6 , 11 ]; and, secondly, to develop an initial literature search strategy to estimate the volume of relevant literature (and quality of a small sample of relevant studies [ 10 ]) and indicate the resources required for literature searching and the review of the studies that follows [ 7 , 10 ].

Three documents summarised guidance on where to search to determine if a new review was justified [ 2 , 6 , 11 ]. These focused on searching databases of systematic reviews (The Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE)), institutional registries (including PROSPERO), and MEDLINE [ 6 , 11 ]. It is worth noting, however, that as of 2015, DARE (and NHS EEDs) are no longer being updated and so the relevance of this (these) resource(s) will diminish over-time [ 64 ]. One guidance document, ‘Systematic reviews in the Social Sciences’, noted, however, that databases are not the only source of information and unpublished reports, conference proceeding and grey literature may also be required, depending on the nature of the review question [ 2 ].

Two documents reported clearly that this preparation (or ‘scoping’) exercise should be undertaken before the actual search strategy is developed [ 7 , 10 ]).

The guidance offers the best available source on preparing the literature search with the published studies not typically reporting how their scoping informed the development of their search strategies nor how their search approaches were developed. Text mining has been proposed as a technique to develop search strategies in the scoping stages of a review although this work is still exploratory [ 65 ]. ‘Clustering documents’ and word frequency analysis have also been tested to identify search terms and studies for review [ 66 , 67 ]. Preparing for literature searches and scoping constitutes an area for future research.

Key stage four: Designing the search strategy

The Population, Intervention, Comparator, Outcome (PICO) structure was the commonly reported structure promoted to design a literature search strategy. Five documents suggested that the eligibility criteria or review question will determine which concepts of PICO will be populated to develop the search strategy [ 1 , 4 , 7 , 8 , 9 ]. The NICE handbook promoted multiple structures, namely PICO, SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) and multi-stranded approaches [ 4 ].

With the exclusion of The Joanna Briggs Institute reviewers’ manual, the guidance offered detail on selecting key search terms, synonyms, Boolean language, selecting database indexing terms and combining search terms. The CEE handbook suggested that ‘search terms may be compiled with the help of the commissioning organisation and stakeholders’ [ 10 ].

The use of limits, such as language or date limits, were discussed in all documents [ 2 , 3 , 4 , 6 , 7 , 8 , 9 , 10 , 11 ].

Search strategy structure

The guidance typically relates to reviews of intervention effectiveness so PICO – with its focus on intervention and comparator - is the dominant model used to structure literature search strategies [ 68 ]. PICOs – where the S denotes study design - is also commonly used in effectiveness reviews [ 6 , 68 ]. As the NICE handbook notes, alternative models to structure literature search strategies have been developed and tested. Booth provides an overview on formulating questions for evidence based practice [ 69 ] and has developed a number of alternatives to the PICO structure, namely: BeHEMoTh (Behaviour of interest; Health context; Exclusions; Models or Theories) for use when systematically identifying theory [ 55 ]; SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) for identification of social science and evaluation studies [ 69 ] and, working with Cooke and colleagues, SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) [ 70 ]. SPIDER has been compared to PICO and PICOs in a study by Methley et al. [ 68 ].

The NICE handbook also suggests the use of multi-stranded approaches to developing literature search strategies [ 4 ]. Glanville developed this idea in a study by Whitting et al. [ 71 ] and a worked example of this approach is included in the development of a search filter by Cooper et al. [ 72 ].

Writing search strategies: Conceptual and objective approaches

Hausner et al. [ 73 ] provide guidance on writing literature search strategies, delineating between conceptually and objectively derived approaches. The conceptual approach, advocated by and explained in the guidance documents, relies on the expertise of the literature searcher to identify key search terms and then develop key terms to include synonyms and controlled syntax. Hausner and colleagues set out the objective approach [ 73 ] and describe what may be done to validate it [ 74 ].

The use of limits

The guidance documents offer direction on the use of limits within a literature search. Limits can be used to focus literature searching to specific study designs or by other markers (such as by date) which limits the number of studies returned by a literature search. The use of limits should be described and the implications explored [ 34 ] since limiting literature searching can introduce bias (explored above). Craven et al. have suggested the use of a supporting narrative to explain decisions made in the process of developing literature searches and this advice would usefully capture decisions on the use of search limits [ 75 ].

Key stage five: Determining the process of literature searching and deciding where to search (bibliographic database searching)

Table 2 summarises the process of literature searching as reported in each guidance document. Searching bibliographic databases was consistently reported as the ‘first step’ to literature searching in all nine guidance documents.

Three documents reported specific guidance on where to search, in each case specific to the type of review their guidance informed, and as a minimum requirement [ 4 , 9 , 11 ]. Seven of the key guidance documents suggest that the selection of bibliographic databases depends on the topic of review [ 2 , 3 , 4 , 6 , 7 , 8 , 10 ], with two documents noting the absence of an agreed standard on what constitutes an acceptable number of databases searched [ 2 , 6 ].

The guidance documents summarise ‘how to’ search bibliographic databases in detail and this guidance is further contextualised above in terms of developing the search strategy. The documents provide guidance of selecting bibliographic databases, in some cases stating acceptable minima (i.e. The Cochrane Handbook states Cochrane CENTRAL, MEDLINE and EMBASE), and in other cases simply listing bibliographic database available to search. Studies have explored the value in searching specific bibliographic databases, with Wright et al. (2015) noting the contribution of CINAHL in identifying qualitative studies [ 76 ], Beckles et al. (2013) questioning the contribution of CINAHL to identifying clinical studies for guideline development [ 77 ], and Cooper et al. (2015) exploring the role of UK-focused bibliographic databases to identify UK-relevant studies [ 78 ]. The host of the database (e.g. OVID or ProQuest) has been shown to alter the search returns offered. Younger and Boddy [ 79 ] report differing search returns from the same database (AMED) but where the ‘host’ was different [ 79 ].

The average number of bibliographic database searched in systematic reviews has risen in the period 1994–2014 (from 1 to 4) [ 80 ] but there remains (as attested to by the guidance) no consensus on what constitutes an acceptable number of databases searched [ 48 ]. This is perhaps because thinking about the number of databases searched is the wrong question, researchers should be focused on which databases were searched and why, and which databases were not searched and why. The discussion should re-orientate to the differential value of sources but researchers need to think about how to report this in studies to allow findings to be generalised. Bethel (2017) has proposed ‘search summaries’, completed by the literature searcher, to record where included studies were identified, whether from database (and which databases specifically) or supplementary search methods [ 81 ]. Search summaries document both yield and accuracy of searches, which could prospectively inform resource use and decisions to search or not to search specific databases in topic areas. The prospective use of such data presupposes, however, that past searches are a potential predictor of future search performance (i.e. that each topic is to be considered representative and not unique). In offering a body of practice, this data would be of greater practicable use than current studies which are considered as little more than individual case studies [ 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 ].

When to database search is another question posed in the literature. Beyer et al. [ 91 ] report that databases can be prioritised for literature searching which, whilst not addressing the question of which databases to search, may at least bring clarity as to which databases to search first [ 91 ]. Paradoxically, this links to studies that suggest PubMed should be searched in addition to MEDLINE (OVID interface) since this improves the currency of systematic reviews [ 92 , 93 ]. Cooper et al. (2017) have tested the idea of database searching not as a primary search method (as suggested in the guidance) but as a supplementary search method in order to manage the volume of studies identified for an environmental effectiveness systematic review. Their case study compared the effectiveness of database searching versus a protocol using supplementary search methods and found that the latter identified more relevant studies for review than searching bibliographic databases [ 94 ].

Key stage six: Determining the process of literature searching and deciding where to search (supplementary search methods)

Table 2 also summaries the process of literature searching which follows bibliographic database searching. As Table 2 sets out, guidance that supplementary literature search methods should be used in systematic reviews recurs across documents, but the order in which these methods are used, and the extent to which they are used, varies. We noted inconsistency in the labelling of supplementary search methods between guidance documents.

Rather than focus on the guidance on how to use the methods (which has been summarised in a recent review [ 95 ]), we focus on the aim or purpose of supplementary search methods.

The Cochrane Handbook reported that ‘efforts’ to identify unpublished studies should be made [ 9 ]. Four guidance documents [ 2 , 3 , 6 , 9 ] acknowledged that searching beyond bibliographic databases was necessary since ‘databases are not the only source of literature’ [ 2 ]. Only one document reported any guidance on determining when to use supplementary methods. The IQWiG handbook reported that the use of handsearching (in their example) could be determined on a ‘case-by-case basis’ which implies that the use of these methods is optional rather than mandatory. This is in contrast to the guidance (above) on bibliographic database searching.

The issue for supplementary search methods is similar in many ways to the issue of searching bibliographic databases: demonstrating value. The purpose and contribution of supplementary search methods in systematic reviews is increasingly acknowledged [ 37 , 61 , 62 , 96 , 97 , 98 , 99 , 100 , 101 ] but understanding the value of the search methods to identify studies and data is unclear. In a recently published review, Cooper et al. (2017) reviewed the literature on supplementary search methods looking to determine the advantages, disadvantages and resource implications of using supplementary search methods [ 95 ]. This review also summarises the key guidance and empirical studies and seeks to address the question on when to use these search methods and when not to [ 95 ]. The guidance is limited in this regard and, as Table 2 demonstrates, offers conflicting advice on the order of searching, and the extent to which these search methods should be used in systematic reviews.

Key stage seven: Managing the references

Five of the documents provided guidance on managing references, for example downloading, de-duplicating and managing the output of literature searches [ 2 , 4 , 6 , 8 , 10 ]. This guidance typically itemised available bibliographic management tools rather than offering guidance on how to use them specifically [ 2 , 4 , 6 , 8 ]. The CEE handbook provided guidance on importing data where no direct export option is available (e.g. web-searching) [ 10 ].

The literature on using bibliographic management tools is not large relative to the number of ‘how to’ videos on platforms such as YouTube (see for example [ 102 ]). These YouTube videos confirm the overall lack of ‘how to’ guidance identified in this study and offer useful instruction on managing references. Bramer et al. set out methods for de-duplicating data and reviewing references in Endnote [ 103 , 104 ] and Gall tests the direct search function within Endnote to access databases such as PubMed, finding a number of limitations [ 105 ]. Coar et al. and Ahmed et al. consider the role of the free-source tool, Zotero [ 106 , 107 ]. Managing references is a key administrative function in the process of review particularly for documenting searches in PRISMA guidance.

Key stage eight: Documenting the search

The Cochrane Handbook was the only guidance document to recommend a specific reporting guideline: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 9 ]. Six documents provided guidance on reporting the process of literature searching with specific criteria to report [ 3 , 4 , 6 , 8 , 9 , 10 ]. There was consensus on reporting: the databases searched (and the host searched by), the search strategies used, and any use of limits (e.g. date, language, search filters (The CRD handbook called for these limits to be justified [ 6 ])). Three guidance documents reported that the number of studies identified should be recorded [ 3 , 6 , 10 ]. The number of duplicates identified [ 10 ], the screening decisions [ 3 ], a comprehensive list of grey literature sources searched (and full detail for other supplementary search methods) [ 8 ], and an annotation of search terms tested but not used [ 4 ] were identified as unique items in four documents.

The Cochrane Handbook was the only guidance document to note that the full search strategies for each database should be included in the Additional file 1 of the review [ 9 ].

All guidance documents should ultimately deliver completed systematic reviews that fulfil the requirements of the PRISMA reporting guidelines [ 108 ]. The guidance broadly requires the reporting of data that corresponds with the requirements of the PRISMA statement although documents typically ask for diverse and additional items [ 108 ]. In 2008, Sampson et al. observed a lack of consensus on reporting search methods in systematic reviews [ 109 ] and this remains the case as of 2017, as evidenced in the guidance documents, and in spite of the publication of the PRISMA guidelines in 2009 [ 110 ]. It is unclear why the collective guidance does not more explicitly endorse adherence to the PRISMA guidance.

Reporting of literature searching is a key area in systematic reviews since it sets out clearly what was done and how the conclusions of the review can be believed [ 52 , 109 ]. Despite strong endorsement in the guidance documents, specifically supported in PRISMA guidance, and other related reporting standards too (such as ENTREQ for qualitative evidence synthesis, STROBE for reviews of observational studies), authors still highlight the prevalence of poor standards of literature search reporting [ 31 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 ]. To explore issues experienced by authors in reporting literature searches, and look at uptake of PRISMA, Radar et al. [ 120 ] surveyed over 260 review authors to determine common problems and their work summaries the practical aspects of reporting literature searching [ 120 ]. Atkinson et al. [ 121 ] have also analysed reporting standards for literature searching, summarising recommendations and gaps for reporting search strategies [ 121 ].

One area that is less well covered by the guidance, but nevertheless appears in this literature, is the quality appraisal or peer review of literature search strategies. The PRESS checklist is the most prominent and it aims to develop evidence-based guidelines to peer review of electronic search strategies [ 5 , 122 , 123 ]. A corresponding guideline for documentation of supplementary search methods does not yet exist although this idea is currently being explored.

How the reporting of the literature searching process corresponds to critical appraisal tools is an area for further research. In the survey undertaken by Radar et al. (2014), 86% of survey respondents (153/178) identified a need for further guidance on what aspects of the literature search process to report [ 120 ]. The PRISMA statement offers a brief summary of what to report but little practical guidance on how to report it [ 108 ]. Critical appraisal tools for systematic reviews, such as AMSTAR 2 (Shea et al. [ 124 ]) and ROBIS (Whiting et al. [ 125 ]), can usefully be read alongside PRISMA guidance, since they offer greater detail on how the reporting of the literature search will be appraised and, therefore, they offer a proxy on what to report [ 124 , 125 ]. Further research in the form of a study which undertakes a comparison between PRISMA and quality appraisal checklists for systematic reviews would seem to begin addressing the call, identified by Radar et al., for further guidance on what to report [ 120 ].

Limitations

Other handbooks exist.

A potential limitation of this literature review is the focus on guidance produced in Europe (the UK specifically) and Australia. We justify the decision for our selection of the nine guidance documents reviewed in this literature review in section “ Identifying guidance ”. In brief, these nine guidance documents were selected as the most relevant health care guidance that inform UK systematic reviewing practice, given that the UK occupies a prominent position in the science of health information retrieval. We acknowledge the existence of other guidance documents, such as those from North America (e.g. the Agency for Healthcare Research and Quality (AHRQ) [ 126 ], The Institute of Medicine [ 127 ] and the guidance and resources produced by the Canadian Agency for Drugs and Technologies in Health (CADTH) [ 128 ]). We comment further on this directly below.

The handbooks are potentially linked to one another

What is not clear is the extent to which the guidance documents inter-relate or provide guidance uniquely. The Cochrane Handbook, first published in 1994, is notably a key source of reference in guidance and systematic reviews beyond Cochrane reviews. It is not clear to what extent broadening the sample of guidance handbooks to include North American handbooks, and guidance handbooks from other relevant countries too, would alter the findings of this literature review or develop further support for the process model. Since we cannot be clear, we raise this as a potential limitation of this literature review. On our initial review of a sample of North American, and other, guidance documents (before selecting the guidance documents considered in this review), however, we do not consider that the inclusion of these further handbooks would alter significantly the findings of this literature review.

This is a literature review

A further limitation of this review was that the review of published studies is not a systematic review of the evidence for each key stage. It is possible that other relevant studies could help contribute to the exploration and development of the key stages identified in this review.

This literature review would appear to demonstrate the existence of a shared model of the literature searching process in systematic reviews. We call this model ‘the conventional approach’, since it appears to be common convention in nine different guidance documents.

The findings reported above reveal eight key stages in the process of literature searching for systematic reviews. These key stages are consistently reported in the nine guidance documents which suggests consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews.

In Table 2 , we demonstrate consensus regarding the application of literature search methods. All guidance documents distinguish between primary and supplementary search methods. Bibliographic database searching is consistently the first method of literature searching referenced in each guidance document. Whilst the guidance uniformly supports the use of supplementary search methods, there is little evidence for a consistent process with diverse guidance across documents. This may reflect differences in the core focus across each document, linked to differences in identifying effectiveness studies or qualitative studies, for instance.

Eight of the nine guidance documents reported on the aims of literature searching. The shared understanding was that literature searching should be thorough and comprehensive in its aim and that this process should be reported transparently so that that it could be reproduced. Whilst only three documents explicitly link this understanding to minimising bias, it is clear that comprehensive literature searching is implicitly linked to ‘not missing relevant studies’ which is approximately the same point.

Defining the key stages in this review helps categorise the scholarship available, and it prioritises areas for development or further study. The supporting studies on preparing for literature searching (key stage three, ‘preparation’) were, for example, comparatively few, and yet this key stage represents a decisive moment in literature searching for systematic reviews. It is where search strategy structure is determined, search terms are chosen or discarded, and the resources to be searched are selected. Information specialists, librarians and researchers, are well placed to develop these and other areas within the key stages we identify.

This review calls for further research to determine the suitability of using the conventional approach. The publication dates of the guidance documents which underpin the conventional approach may raise questions as to whether the process which they each report remains valid for current systematic literature searching. In addition, it may be useful to test whether it is desirable to use the same process model of literature searching for qualitative evidence synthesis as that for reviews of intervention effectiveness, which this literature review demonstrates is presently recommended best practice.

Abbreviations

Behaviour of interest; Health context; Exclusions; Models or Theories

Cochrane Database of Systematic Reviews

The Cochrane Central Register of Controlled Trials

Database of Abstracts of Reviews of Effects

Enhancing transparency in reporting the synthesis of qualitative research

Institute for Quality and Efficiency in Healthcare

National Institute for Clinical Excellence

Population, Intervention, Comparator, Outcome

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Setting, Perspective, Intervention, Comparison, Evaluation

Sample, Phenomenon of Interest, Design, Evaluation, Research type

STrengthening the Reporting of OBservational studies in Epidemiology

Trial Search Co-ordinators

Booth A. Unpacking your literature search toolbox: on search styles and tactics. Health Information & Libraries Journal. 2008;25(4):313–7.

Article   Google Scholar  

Petticrew M, Roberts H. Systematic reviews in the social sciences: a practical guide. Oxford: Blackwell Publishing Ltd; 2006.

Book   Google Scholar  

Institute for Quality and Efficiency in Health Care (IQWiG). IQWiG Methods Resources. 7 Information retrieval 2014 [Available from: https://www.ncbi.nlm.nih.gov/books/NBK385787/ .

NICE: National Institute for Health and Care Excellence. Developing NICE guidelines: the manual 2014. Available from: https://www.nice.org.uk/media/default/about/what-we-do/our-programmes/developing-nice-guidelines-the-manual.pdf .

Sampson M. MJ, Lefebvre C, Moher D, Grimshaw J. Peer Review of Electronic Search Strategies: PRESS; 2008.

Google Scholar  

Centre for Reviews & Dissemination. Systematic reviews – CRD’s guidance for undertaking reviews in healthcare. York: Centre for Reviews and Dissemination, University of York; 2009.

eunetha: European Network for Health Technology Assesment Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness 2016. Available from: http://www.eunethta.eu/sites/default/files/Guideline_Information_Retrieval_V1-1.pdf .

Kugley SWA, Thomas J, Mahood Q, Jørgensen AMK, Hammerstrøm K, Sathe N. Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: Campbell Collaboration. 2017; Available from: https://www.campbellcollaboration.org/library/searching-for-studies-information-retrieval-guide-campbell-reviews.html

Lefebvre C, Manheimer E, Glanville J. Chapter 6: searching for studies. In: JPT H, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions; 2011.

Collaboration for Environmental Evidence. Guidelines for Systematic Review and Evidence Synthesis in Environmental Management.: Environmental Evidence:; 2013. Available from: http://www.environmentalevidence.org/wp-content/uploads/2017/01/Review-guidelines-version-4.2-final-update.pdf .

The Joanna Briggs Institute. Joanna Briggs institute reviewers’ manual. 2014th ed: the Joanna Briggs institute; 2014. Available from: https://joannabriggs.org/assets/docs/sumari/ReviewersManual-2014.pdf

Beverley CA, Booth A, Bath PA. The role of the information specialist in the systematic review process: a health information case study. Health Inf Libr J. 2003;20(2):65–74.

Article   CAS   Google Scholar  

Harris MR. The librarian's roles in the systematic review process: a case study. Journal of the Medical Library Association. 2005;93(1):81–7.

PubMed   PubMed Central   Google Scholar  

Egger JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PLoS One. 2015;10(5):e0125931.

Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014;67(9):1001–7.

Article   PubMed   Google Scholar  

McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc. 2005;93(1):74–80.

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol. 2015;68(6):617–26.

Weller AC. Mounting evidence that librarians are essential for comprehensive literature searches for meta-analyses and Cochrane reports. J Med Libr Assoc. 2004;92(2):163–4.

Swinkels A, Briddon J, Hall J. Two physiotherapists, one librarian and a systematic literature review: collaboration in action. Health Info Libr J. 2006;23(4):248–56.

Foster M. An overview of the role of librarians in systematic reviews: from expert search to project manager. EAHIL. 2015;11(3):3–7.

Lawson L. OPERATING OUTSIDE LIBRARY WALLS 2004.

Vassar M, Yerokhin V, Sinnett PM, Weiher M, Muckelrath H, Carr B, et al. Database selection in systematic reviews: an insight through clinical neurology. Health Inf Libr J. 2017;34(2):156–64.

Townsend WA, Anderson PF, Ginier EC, MacEachern MP, Saylor KM, Shipman BL, et al. A competency framework for librarians involved in systematic reviews. Journal of the Medical Library Association : JMLA. 2017;105(3):268–75.

Cooper ID, Crum JA. New activities and changing roles of health sciences librarians: a systematic review, 1990-2012. Journal of the Medical Library Association : JMLA. 2013;101(4):268–77.

Crum JA, Cooper ID. Emerging roles for biomedical librarians: a survey of current practice, challenges, and changes. Journal of the Medical Library Association : JMLA. 2013;101(4):278–86.

Dudden RF, Protzko SL. The systematic review team: contributions of the health sciences librarian. Med Ref Serv Q. 2011;30(3):301–15.

Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008;61(5):440–8.

Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Academic medicine : journal of the Association of American Medical Colleges. 2011;86(8):1049–54.

Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. Journal of the Medical Library Association : JMLA. 2016;104(4):267–77.

Morris M, Boruff JT, Gore GC. Scoping reviews: establishing the role of the librarian. Journal of the Medical Library Association : JMLA. 2016;104(4):346–54.

Koffel JB, Rethlefsen ML. Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. PLoS One. 2016;11(9):e0163309.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Fehrmann P, Thomas J. Comprehensive computer searches and reporting in systematic reviews. Research Synthesis Methods. 2011;2(1):15–32.

Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic Reviews. 2016;5(1):74.

Article   PubMed   PubMed Central   Google Scholar  

Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health technology assessment (Winchester, England). 2003;7(1):1–76.

Tricco AC, Tetzlaff J, Sampson M, Fergusson D, Cogo E, Horsley T, et al. Few systematic reviews exist documenting the extent of bias: a systematic review. J Clin Epidemiol. 2008;61(5):422–34.

Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. Int J Technol Assess Health Care. 2010;26(4):431–5.

Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Inf Libr J. 2010;27(2):114–22.

Petticrew M. Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’. Systematic Reviews. 2015;4(1):36.

Betrán AP, Say L, Gülmezoglu AM, Allen T, Hampson L. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol. 2005;5

Felson DT. Bias in meta-analytic research. J Clin Epidemiol. 1992;45(8):885–92.

Article   PubMed   CAS   Google Scholar  

Franco A, Malhotra N, Simonovits G. Publication bias in the social sciences: unlocking the file drawer. Science. 2014;345(6203):1502–5.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol. 2017;17(1):64.

Schmucker CM, Blümle A, Schell LK, Schwarzer G, Oeller P, Cabrera L, et al. Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research. PLoS One. 2017;12(4):e0176210.

Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet (London, England). 1997;350(9074):326–9.

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health technology assessment (Winchester, England). 2003;7(41):1–90.

Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005;58(8):769–76.

Mills EJ, Kanters S, Thorlund K, Chaimani A, Veroniki A-A, Ioannidis JPA. The effects of excluding treatments from network meta-analyses: survey. BMJ : British Medical Journal. 2013;347

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. 2016;16(1):127.

van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for unpublished trials in Cochrane reviews may not be worth the effort. J Clin Epidemiol. 2009;62(8):838–44.e3.

Buchberger B, Krabbe L, Lux B, Mattivi JT. Evidence mapping for decision making: feasibility versus accuracy - when to abandon high sensitivity in electronic searches. German medical science : GMS e-journal. 2016;14:Doc09.

Lorenc T, Pearson M, Jamal F, Cooper C, Garside R. The role of systematic reviews of qualitative evidence in evaluating interventions: a case study. Research Synthesis Methods. 2012;3(1):1–10.

Gough D. Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. Res Pap Educ. 2007;22(2):213–28.

Barroso J, Gollop CJ, Sandelowski M, Meynell J, Pearce PF, Collins LJ. The challenges of searching for and retrieving qualitative studies. West J Nurs Res. 2003;25(2):153–78.

Britten N, Garside R, Pope C, Frost J, Cooper C. Asking more of qualitative synthesis: a response to Sally Thorne. Qual Health Res. 2017;27(9):1370–6.

Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J. 2015;32(3):220–35.

Kwon Y, Powelson SE, Wong H, Ghali WA, Conly JM. An assessment of the efficacy of searching in biomedical databases beyond MEDLINE in identifying studies for a systematic review on ward closures as an infection control intervention to control outbreaks. Syst Rev. 2014;3:135.

Nussbaumer-Streit B, Klerings I, Wagner G, Titscher V, Gartlehner G. Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study. Systematic Reviews. 2016;5:197.

Wagner G, Nussbaumer-Streit B, Greimel J, Ciapponi A, Gartlehner G. Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey. BMC Med Res Methodol. 2017;17(1):121.

Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health. 2005;59(9):804–8.

Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. Int J Technol Assess Health Care. 2003;19(4):591–603.

Pearson M, Moxham T, Ashton K. Effectiveness of search strategies for qualitative research about barriers and facilitators of program delivery. Eval Health Prof. 2011;34(3):297–308.

Levay P, Raynor M, Tuvey D. The Contributions of MEDLINE, Other Bibliographic Databases and Various Search Techniques to NICE Public Health Guidance. 2015. 2015;10(1):19.

Nussbaumer-Streit B, Klerings I, Wagner G, Heise TL, Dobrescu AI, Armijo-Olivo S, et al. Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study. J Clin Epidemiol. 2018;102:1–11.

Briscoe S, Cooper C, Glanville J, Lefebvre C. The loss of the NHS EED and DARE databases and the effect on evidence synthesis and evaluation. Res Synth Methods. 2017;8(3):256–7.

Stansfield C, O'Mara-Eves A, Thomas J. Text mining for search term development in systematic reviewing: A discussion of some methods and challenges. Research Synthesis Methods.n/a-n/a.

Petrova M, Sutcliffe P, Fulford KW, Dale J. Search terms and a validated brief search filter to retrieve publications on health-related values in Medline: a word frequency analysis study. Journal of the American Medical Informatics Association : JAMIA. 2012;19(3):479–88.

Stansfield C, Thomas J, Kavanagh J. 'Clustering' documents automatically to support scoping reviews of research: a case study. Res Synth Methods. 2013;4(3):230–41.

PubMed   Google Scholar  

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Andrew B. Clear and present questions: formulating questions for evidence based practice. Library Hi Tech. 2006;24(3):355–68.

Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012;22(10):1435–43.

Whiting P, Westwood M, Bojke L, Palmer S, Richardson G, Cooper J, et al. Clinical effectiveness and cost-effectiveness of tests for the diagnosis and investigation of urinary tract infection in children: a systematic review and economic model. Health technology assessment (Winchester, England). 2006;10(36):iii-iv, xi-xiii, 1–154.

Cooper C, Levay P, Lorenc T, Craig GM. A population search filter for hard-to-reach populations increased search efficiency for a systematic review. J Clin Epidemiol. 2014;67(5):554–9.

Hausner E, Waffenschmidt S, Kaiser T, Simon M. Routine development of objectively derived search strategies. Systematic Reviews. 2012;1(1):19.

Hausner E, Guddat C, Hermanns T, Lampert U, Waffenschmidt S. Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one. J Clin Epidemiol. 2016;77:118–24.

Craven J, Levay P. Recording database searches for systematic reviews - what is the value of adding a narrative to peer-review checklists? A case study of nice interventional procedures guidance. Evid Based Libr Inf Pract. 2011;6(4):72–87.

Wright K, Golder S, Lewis-Light K. What value is the CINAHL database when searching for systematic reviews of qualitative studies? Syst Rev. 2015;4:104.

Beckles Z, Glover S, Ashe J, Stockton S, Boynton J, Lai R, et al. Searching CINAHL did not add value to clinical questions posed in NICE guidelines. J Clin Epidemiol. 2013;66(9):1051–7.

Cooper C, Rogers M, Bethel A, Briscoe S, Lowe J. A mapping review of the literature on UK-focused health and social care databases. Health Inf Libr J. 2015;32(1):5–22.

Younger P, Boddy K. When is a search not a search? A comparison of searching the AMED complementary health database via EBSCOhost, OVID and DIALOG. Health Inf Libr J. 2009;26(2):126–35.

Lam MT, McDiarmid M. Increasing number of databases searched in systematic reviews and meta-analyses between 1994 and 2014. Journal of the Medical Library Association : JMLA. 2016;104(4):284–9.

Bethel A, editor Search summary tables for systematic reviews: results and findings. HLC Conference 2017a.

Aagaard T, Lund H, Juhl C. Optimizing literature search in systematic reviews - are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders? BMC Med Res Methodol. 2016;16(1):161.

Adams CE, Frederick K. An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care. Psychol Med. 1994;24(3):741–8.

Kelly L, St Pierre-Hansen N. So many databases, such little clarity: searching the literature for the topic aboriginal. Canadian family physician Medecin de famille canadien. 2008;54(11):1572–3.

Lawrence DW. What is lost when searching only one literature database for articles relevant to injury prevention and safety promotion? Injury Prevention. 2008;14(6):401–4.

Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA. Searching one or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol. 2005;58(9):867–73.

Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, et al. Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol. 2003;56(10):943–55.

Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complementary Therapies in Medicine. 2004;12(4):228–32.

Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials. 2000;21(5):476–87.

Taylor B, Wylie E, Dempster M, Donnelly M. Systematically retrieving research: a case study evaluating seven databases. Res Soc Work Pract. 2007;17(6):697–706.

Beyer FR, Wright K. Can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. Health Info Libr J. 2013;30(1):49–58.

Duffy S, de Kock S, Misso K, Noake C, Ross J, Stirk L. Supplementary searches of PubMed to improve currency of MEDLINE and MEDLINE in-process searches via Ovid. Journal of the Medical Library Association : JMLA. 2016;104(4):309–12.

Katchamart W, Faulkner A, Feldman B, Tomlinson G, Bombardier C. PubMed had a higher sensitivity than Ovid-MEDLINE in the search for systematic reviews. J Clin Epidemiol. 2011;64(7):805–7.

Cooper C, Lovell R, Husk K, Booth A, Garside R. Supplementary search methods were more effective and offered better value than bibliographic database searching: a case study from public health and environmental enhancement (in Press). Research Synthesis Methods. 2017;

Cooper C, Booth, A., Britten, N., Garside, R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: A methodological review. (In Press). BMC Systematic Reviews. 2017.

Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ (Clinical research ed). 2005;331(7524):1064–5.

Article   PubMed Central   Google Scholar  

Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. PharmacoEconomics. 2015;33(1):5–11.

Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with web of science and Google scholar. Res Synth Methods. 2016;7(1):34–45.

McManus RJ, Wilson S, Delaney BC, Fitzmaurice DA, Hyde CJ, Tobias RS, et al. Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. BMJ (Clinical research ed). 1998;317(7172):1562–3.

Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. Journal of public health research. 2014;3(2):177.

Matthews EJ, Edwards AG, Barker J, Bloor M, Covey J, Hood K, et al. Efficient literature searching in diffuse topics: lessons from a systematic review of research on communicating risk to patients in primary care. Health Libr Rev. 1999;16(2):112–20.

Bethel A. Endnote Training (YouTube Videos) 2017b [Available from: http://medicine.exeter.ac.uk/esmi/workstreams/informationscience/is_resources,_guidance_&_advice/ .

Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association : JMLA. 2016;104(3):240–3.

Bramer WM, Milic J, Mast F. Reviewing retrieved references for inclusion in systematic reviews using EndNote. Journal of the Medical Library Association : JMLA. 2017;105(1):84–7.

Gall C, Brahmi FA. Retrieval comparison of EndNote to search MEDLINE (Ovid and PubMed) versus searching them directly. Medical reference services quarterly. 2004;23(3):25–32.

Ahmed KK, Al Dhubaib BE. Zotero: a bibliographic assistant to researcher. J Pharmacol Pharmacother. 2011;2(4):303–5.

Coar JT, Sewell JP. Zotero: harnessing the power of a personal bibliographic manager. Nurse Educ. 2010;35(5):205–7.

Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008;61(8):748–54.

Toews LC. Compliance of systematic reviews in veterinary journals with preferred reporting items for systematic reviews and meta-analysis (PRISMA) literature search reporting guidelines. Journal of the Medical Library Association : JMLA. 2017;105(3):233–9.

Booth A. "brimful of STARLITE": toward standards for reporting literature searches. Journal of the Medical Library Association : JMLA. 2006;94(4):421–9. e205

Faggion CM Jr, Wu YC, Tu YK, Wasiak J. Quality of search strategies reported in systematic reviews published in stereotactic radiosurgery. Br J Radiol. 2016;89(1062):20150878.

Mullins MM, DeLuca JB, Crepaz N, Lyles CM. Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000–2010): are the searches clearly explained, systematic and reproducible? Research Synthesis Methods. 2014;5(2):116–30.

Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association : JMLA. 2009;97(1):21–9.

Bigna JJ, Um LN, Nansseu JR. A comparison of quality of abstracts of systematic reviews including meta-analysis of randomized controlled trials in high-impact general medicine journals before and after the publication of PRISMA extension for abstracts: a systematic review and meta-analysis. Syst Rev. 2016;5(1):174.

Akhigbe T, Zolnourian A, Bulters D. Compliance of systematic reviews articles in brain arteriovenous malformation with PRISMA statement guidelines: review of literature. Journal of clinical neuroscience : official journal of the Neurosurgical Society of Australasia. 2017;39:45–8.

Tao KM, Li XQ, Zhou QH, Moher D, Ling CQ, Yu WF. From QUOROM to PRISMA: a survey of high-impact medical journals' instructions to authors and a review of systematic reviews in anesthesia literature. PLoS One. 2011;6(11):e27611.

Wasiak J, Tyack Z, Ware R. Goodwin N. Jr. Poor methodological quality and reporting standards of systematic reviews in burn care management. International wound journal: Faggion CM; 2016.

Tam WW, Lo KK, Khalechelvam P. Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study. BMJ Open. 2017;7(2):e013905.

Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Methods. 2014;5(2):98–115.

Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Res Synth Methods. 2015;6(1):87–95.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6.

Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62(9):944–52.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Clinical research ed). 2017;358.

Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.

Relevo R, Balshem H. Finding evidence for comparing medical interventions: AHRQ and the effective health care program. J Clin Epidemiol. 2011;64(11):1168–77.

Medicine Io. Standards for Systematic Reviews 2011 [Available from: http://www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx .

CADTH: Resources 2018.

Download references

Acknowledgements

CC acknowledges the supervision offered by Professor Chris Hyde.

This publication forms a part of CC’s PhD. CC’s PhD was funded through the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Project Number 16/54/11). The open access fee for this publication was paid for by Exeter Medical School.

RG and NB were partially supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula.

The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Author information

Authors and affiliations.

Institute of Health Research, University of Exeter Medical School, Exeter, UK

Chris Cooper & Jo Varley-Campbell

HEDS, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK

Andrew Booth

Nicky Britten

European Centre for Environment and Human Health, University of Exeter Medical School, Truro, UK

Ruth Garside

You can also search for this author in PubMed   Google Scholar

Contributions

CC conceived the idea for this study and wrote the first draft of the manuscript. CC discussed this publication in PhD supervision with AB and separately with JVC. CC revised the publication with input and comments from AB, JVC, RG and NB. All authors revised the manuscript prior to submission. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Chris Cooper .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Appendix tables and PubMed search strategy. Key studies used for pearl growing per key stage, working data extraction tables and the PubMed search strategy. (DOCX 30 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Cooper, C., Booth, A., Varley-Campbell, J. et al. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol 18 , 85 (2018). https://doi.org/10.1186/s12874-018-0545-3

Download citation

Received : 20 September 2017

Accepted : 06 August 2018

Published : 14 August 2018

DOI : https://doi.org/10.1186/s12874-018-0545-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Literature Search Process
• Citation Chasing
• Tacit Models
• Unique Guidance
• Information Specialists

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

Systematic and systematic-like review toolkit: Step 3: Screening and selection of articles

Systematic and systematic-like review toolkit.

• Systematic and systematic-like reviews overview
• Step 1: Formulating the research question
• Step 2: Developing the search
• Step 3: Screening and selection of articles
• Step 4: Appraisal of articles
• Step 5: Writing and publishing
• Filters and complex search examples
• Evidence synthesis support services

Tip: Look for these icons for guidance on which technique is required

Email your Librarians

Step 3: Screening tools

Screening is the process of identifying studies from the literature search for inclusion in the review.

PRISMA  (Preferred Reporting Items for Systematic reviews and Meta-Analyses) is fast becoming a standard for reporting of systematic reviews and meta-analyses, but is also used with other review types.

It includes a procedural checklist, and a flow diagram to illustrate the screening process.

Tools to manage the screening process

Consider using one of the tools below to manage the screening process.

• EndNote  (Deakin supported)  Export searches to an EndNote library.  Create groups to screen citations against inclusion and exclusion criteria, then populate the PRISMA flow diagram.
• Covidence  (Deakin supported) Cochrane-recommended web-based software that streamlines the production of systematic reviews. Allows importing of citations, management of screening by multiple reviewers, data extraction and data export.  Deakin has a license for Covidence and for more information go to the Covidence page .
• Rayyan  (free) Web based collaborative application facilitates team screening, including the upload of citations and  recording of the decisions behind the screening process. A mobile app allows you to screening articles any place. Read the Rayyan for Systematic Reviews guide for further advice on this tool. When exporting from EndNote, choose the RIS format to import into Rayyan.  More instructions are in this EndNote guide .

When working with Deakin subscribed material (for example, PDFs of journal articles), please ensure that you comply with all licences, terms and conditions.  This applies to all screening and reference management tools, particularly when storing copies of articles.  Articles cannot be shared with non-Deakin staff or students.

The screening process

Your review protocol developed at the beginning of your review will have outlined inclusion and exclusion criteria.  These will form the basis of the screening process.

Begin by screening titles and abstracts to remove obviously irrelevant material. At this stage you may not need to provide justification for your exclusions.

You will then need to examine the full text of an article for more detailed screening against your eligibility criteria. At this stage you must provide reasons why you exclude documents.

• appropriate study population (age, geography, illness)
• appropriate intervention/method/measurements
• comparable environment/ population 
• language - can the article be sourced/translated in the language required?

When reviewing the full text of the article, consider:

• appropriate method/measurement
• appropriate sample size
• duplication of data  (avoid counting the same data twice)
• access to data not included in the article if required

Using review teams to reduce selection bias

A team of at least two or three reviewers is important for the screening process to reduce the risk of selection errors and selection bias.

Reviewer teams should:

• have a good knowledge of the topic for fast and accurate screening
• prepare for screening by doing a pilot screen to establish themes or possible difficulties          
• work independently during the screening process to avoid influencing other's decisions

The review paper should detail how many reviewers screened and the process used for resolving any disagreement.

Discover more

• Implementation of the selection process
• Selection of reviews
• Who should extract data?
• Joanna Briggs Institute Manual for Evidence Synthesis
• Systematic reviews: CRD’s guidance for undertaking reviews in health care (PDF)

Further Readings

• Cochrane Handbook, part 2, chapter 4.6: Selecting studies This chapter details the requirements for authors performing screening for SRs in the health sciences.  
• Harrison, H., Griffin, S.J., Kuhn, I. et al. Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation . BMC Med Res Methodol 20, 7 (2020).  
• Micah D. J. Peters (2017) Managing and Coding References for Systematic Reviews and Scoping Reviews in EndNote , Medical Reference Services Quarterly, 36:1, 19-31, DOI:http://dx.doi.org/10.1080/02763869.2017.1259891
• Robin King, Barbara Hooper & Wendy Wood (2011) Using bibliographic software to appraise and code data in educational systematic review research , Medical Teacher, 33:9, 719-723, DOI: 10.3109/0142159X.2011.558138
• << Previous: Step 2: Developing the search
• Next: Step 4: Appraisal of articles >>
• Last Updated: Apr 24, 2024 9:22 AM
• URL: https://deakin.libguides.com/systematicreview

University Libraries

Literature review process.

• Introduction
• Video Tutorials

How to Select a Topic

Refine your topic.

• Select a Review Type
• Search the Literature
• Plan Before Reviewing
• Review the Literature
• Write the Review

The way to select a topic depends on whether you are writing a literature review for an assignment, or your own research. Here are some suggestions:

• If an assignment, read the instructions carefully and talk to the professor for clarification if necessary
• Scan academic journals for recent research trends
• Monitor professional blogs and listservs for hot topics
• Attend conferences to learn about current research
• Talk to classmates, colleagues, and professors about your ideas to help focus them. Even an interested friend outside the field can help you clarify your topic and why it's important.

The Libraries' subject guides are a good place to find information about journals and professional organizations in your field. On the Subject and Course Guides page, use the Subjects list or the search box at the top right to find the best subject guide for you.

Perhaps the most important point is to select a topic you find fascinating because you will be living with it for months or years!

You will need to do some initial literature searching and reading on your potential topic for two reasons:

• to clarify and solidify your research question because you can't test a vague research question
• determine if someone else has already asked and answered your research question

The next tab, Search the Literature, covers where and how to search. While doing your initial search, be looking for the niche where you can do your research:

• gaps in the knowledge on a subject area
• questions about your topic that remain unanswered
• areas of disagreement in your subject area that need to be settled 

After you've refined your topic, then you can go back and do further searching to find the most relevant sources for your research question, which you'll want to discuss in the literature review. Referring to the Literature Review Process image at the right, you are going to make a feedback loop between Select a Topic and Search the Literature. An initial, broad search will define your topic and then a second, narrower search will identify the sources most relevant to your refined topic.

Even in this initial searching, it's important to keep track of what you read and which articles are important in defining your topic. A bibliographic management software is an excellent tool for managing the literature. The UNT Libraries make RefWorks available to all UNT students, staff, and faculty, and its advantage is you have already paid for it through library fees. See the library's RefWorks guide for details on creating your account and using RefWorks.  Zotero  and Mendeley are free web-based reference managers you may want to explore; you will have to purchase software/online access for other reference managers, e.g., EndNote .

Need help? Then use the library's  Ask Us service. Get help from real people face-to-face, by phone, or by email.

• << Previous: Video Tutorials
• Next: Select a Review Type >>
• Last Updated: Nov 13, 2023 4:28 PM
• URL: https://guides.library.unt.edu/literature-review

Additional Links

UNT: Apply now UNT: Schedule a tour UNT: Get more info about the University of North Texas

UNT: Disclaimer | UNT: AA/EOE/ADA | UNT: Privacy | UNT: Electronic Accessibility | UNT: Required Links | UNT: UNT Home

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

How to write a Literature Review: Literature review process

• Literature review process
• Purpose of a literature review
• Evaluating sources
• Managing sources
• Request a literature search
• Selecting the approach to use
• Quantitative vs qualitative method
• Summary of different research methodologies
• Research design vs research methodology
• Diagram: importance of research
• Attributes of a good research scholar

Step 1: Select a topic

• Select a topic you can manage in the time frame you have to complete your project.
• Establish your research questions and organize your literature into logical categories around the subject/ topic areas of your questions.  Your research questions must be specific enou gh to guide you to the relevant literature.
• Make sure you understand the concept of ‘broader’ and ‘narrower’ terms.  The narrower your topic, the easier it will be to limit the number of sources you need to read in order to get a good survey of the literature.

Step 2: Identify the most relevant sources on your topic

Use a variety of resources - locate books , journals , and documents that contain useful information and ideas on your topic. Internet sites , theses & dissertations , conference papers , ePrints and government or industry reports can also be included. Do not rely solely on electronic full-text material which is more easily available. Reference sources such as dictionaries can assist in defining terminology, and encyclopaedias may provide useful introductions to your topic by experts in the field and will list key references.

Step 3 : Search and refine

• Unisa has a number of databases that provide full text access to articles, that allow you to refine your search to ‘peer reviewed’ journals.  These are scholarly journals which go through a rigorous process of quality assessment by several researchers or subject specialists in the academic community before they are accepted for publication. 
• Use the And, Or, Not operators, Wildcards and Logical Brackets when searching in the databases.  For instance, you can use And to narrow your search while the operator OR expands your search.  Not, on the other hand, helps to exclude irrelevant information from your search results.  Please click here for more information on searching.

Literature review process - an overview

Step 3: search and refine.

• Unisa has a number of  databases  that provide full text access to articles, that allow you to refine your search to ‘peer reviewed’ journals.  These are scholarly journals which go through a rigorous process of quality assessment by several researchers or subject specialists in the academic community before they are accepted for publication. 
• Use the  And, Or, Not  operators,  Wildcards  and  Logical Brackets  when searching in the databases.  For instance, you can use  And  to narrow your search while  the  operator  OR  expands your search.   Not,  on the other hand,   helps to exclude   irrelevant information from your search results.  Please click  here  for more information on searching.

How do I write a literature review

See the chapter below for a helpful overview of the literature review process, especially the sections on how to analyse the literature you have gathered and how to write up your literature review:

Literature Reviews and Bibliographic Searches. 2006. In V. Desai, & R. Potter (Eds.),  Doing Development Research.  (pp. 209-222). London, England: SAGE Publications, Ltd. Available at:  http://0-dx.doi.org.oasis.unisa.ac.za/10.4135/9781849208925.n22     (A student will be prompted at some stage for his/ her student number and myUnisa password. A staff member will be prompted at some stage for his/ her Unisa Network username and login password).

This book is available in the  Sage Research Methods Online  database.

Step 4: Read and analyse

Group the sources into the  themes  and  sub-themes  of your topic.  As you read widely but selectively in your topic area, consider what themes or issues connect your sources together.

• Do they present one or different solutions?
• Is there an aspect of the field that is missing?
• How well do they present the material and do they portray it according to an appropriate theory?
• Do they reveal a trend in the field?
• A raging debate?
• Pick one of these themes to focus the organization of your review.

Step 5: Write the literature review

You can organize the review in many ways; for example, you can center the review  historically  (how the topic has been dealt with over time); or center it on the  theoretical positions  surrounding your topic (those for a position vs. those against, for example); or you can focus on how each of your sources contributes to your understanding of your project.

Your literature review should include:

• an  introduction  which explains how your review is organized.
• a  body  which contains the  headings  and  subheadings  that provide a map to show the various perspectives of your argument. In other words the body contains the evaluation of the materials you want to include on your topic.
• a  summary .

Some of the information on this page is indebted to the sources below:

Caldwell College Library

Monmouth University Library

University of Cape Town Libraries

• << Previous: Home
• Next: Purpose of a literature review >>
• Last Updated: Apr 30, 2024 1:19 PM
• URL: https://libguides.unisa.ac.za/literature_review

Systematic Reviews: Study selection and appraisal

• Types of literature review, methods, & resources
• Protocol and registration
• Search strategy
• Medical Literature Databases to search
• Study selection and appraisal
• Data Extraction/Coding/Study characteristics/Results
• Reporting the quality/risk of bias
• Manage citations using RefWorks This link opens in a new window
• GW Box file storage for PDF's This link opens in a new window

Study selection: PRISMA Item 9

Inclusion/Exclusion criteria

See  http://unimelb.libguides.com/c.php?g=492361&p=3368110

First level screening - title and abstract review

At the initial screening stage read just the title and abstract of the candidate studies and make a decision to include or exclude the study from your review.  

For small reviews of a few studies (e.g. <100)

T he research team should agree on the inclusion and exclusion criteria for studies you wish to review and  put together a study screening form.   To help identify  your inclusion/exclusion criteria, r evisit the  PICOS  of interest  you came up with for your  search strategy  and gain agreement/approval from your colleagues or supervisor . The  screening form may look similar to  Table 3  of  Brown et al (2013) . You may write down your decision to include or exclude an article on an Excel spreadsheet  like this one , or if you have a small number of records you may choose to print out one copy for each record, although printing will be impractical for larger numbers of records .  S creen each potentially useful article identified in the literature search as follows:

• Read the title and abstract (where available) and apply the inclusion/exclusion criteria from the screening form.
• Make a decision on whether or not to include the study in the review.
• Record the decision and reasons for inclusion/exclusion on the study screening form or spreadsheet . You will summarise the reasons for exclusion on the PRISMA flow diagram - see Study Selection PRISMA item # 17 below.

For large reviews of many studies (e.g. >100) - in case you need to partially automate the screening process

There are three web-based software applications that can help with screening and tracking your selection decisions:

• Covidence  (GW in 2019 bought a subscription so you can use this tool now). Provides a decision dashboard and annotation tool, and the ability to screen candidate citations you locate in your literature search. Covidence is used by Cochrane review teams as their first level screening tool, the resulting study characteristics and decision data can be exported to  RevMan  (free for academic use) or Excel. 
• Abstrackr  (free, Beta, open-source). Abstrackr comprises two components; a web-based annotation tool that allows participants in a review to collaboratively screen citations for relevance, and machine learning technologies that semi-automate the screening process. The web-based annotation tool allows project leads to import the citations that are to be screened for a review from either RefMan or Pubmed. Participants can then join the project and begin screening; the tool maintains a digital paper trail of all screening decisions.  The machine learning technology permits reviewers to screen roughly half of the set of citations imported for a given review, and then let the software automatically exclude a (hopefully large) portion of the remaining citations; the reviewers will then only need to screen the articles classified as relevant by the software.  A recent article evaluating the use of Abstrackr in the systematic review process is  Rathbone, J., Hoffmann, T., & Glasziou, P. (2015). Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Systematic Reviews, 480. doi:10.1186/s13643-015-0067-6
• DistillerSR  (requires subscription). Enables you to create forms for making screening decisions, and extract data.

Second level screening - full text review

Having excluded candidate studies that did not meet your inclusion/exclusion criteria you should have a smaller number of potentially relevant studies. GW affiliates at GW and Children's National Health System can use Box to store and share the full text PDF's of copyrighted journal articles https://it.gwu.edu/backup-storage-document-management . Read and critically appraise  the full text of each study you selected at the first pass screening stage to determine whether you wish to include them in your discussion and analysis. Specifically each study must be evaluated based on the following criteria:

Does this study address a clearly focused question? Did the study use valid methods to address this question? Are the valid results of this study important? Are these valid, important results applicable to my patient or population?

If the answer to any of these questions is “no”, you may wish to read no further and exclude the study, or you may decide to include the study to inform your discussion but not include the results in your analysis. 

To help with this process you may wish to download and apply one of the following Critical Appraisal tools:

Study Quality Assessment Tools developed in 2013 by the National Heart Lung & Blood Institute: https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools   Choose an appraisal tool that matches the type of study you are reviewing from one of the following 6 study types: Controlled Intervention Studies, Systematic Reviews and Meta-Analyses, Observational Cohort and Cross-Sectional Studies, Case-Control Studies, Before-After (Pre-Post) Studies With No Control Group, & Case Series Studies.

Worksheets from the Oxford University Center for Evidence Based Medicine - choose a worksheet that matches the type of study: Systematic Review article Critical Appraisal Sheet Diagnosis study Critical Appraisal Sheet Prognosis study Critical Appraisal Sheet Therapy / Randomized Controlled Trial Critical Appraisal Sheet

Alternatively the  CASP: Critical Appraisal Skills Checklists  are eight critical appraisal tools designed to be used when reading and evaluating the quality of Systematic Reviews, Randomised Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule.

Another alternative set of Critical Appraisal checklists  are from the Joanna Briggs Institute (JBI). JBI require you use their critical appraisal checklists if you are conducing a JBI systematic review following the methods described in the JBI Manual for Evidence Synthesis .

Make a decision on whether or not to include the study in your review, and write your decision and reasons for inclusion/exclusion at this second level/full text review stage on the study screening form. You will summarize the reasons for exclusion on the PRISMA flow diagram - see Study Selection PRISMA item # 17 below.

Reporting your screening decisions

In the final report in the methods section the PRISMA checklist Item 9 study selection will be reported as:

• How studies were screened e.g. by reading title & abstract, and how they were critically appraised e.g. by applying a standardised appraisal form appropriate for that study type - see above.
• What sort of studies were excluded e.g. letters, conference abstracts, etc.
• Who reviewed/appraised the studies
• What the process was for resolving disagreements e.g. reporting the level of inter-rater agreement, how often arbitration about selection was required, & what efforts were made to resolve disagreement e.g. were original authors contacted

Study selection PRISMA Item 17

Researchers must keep the screening forms to create a summary descriptive flow diagram of study selection.

In the final report in the results section the PRISMA checklist Item 17 study selection should be reported as follows:

• Record the number of studies screened, assessed for eligibility and included in the review, with reasons for exclusions both in the text and in form of a PRISMA flow diagram of study selection e.g. similar to Fig 2 of Liberati et al. (2009). Covidence keeps track of your screening decisions and generates a PRISMA flow diagram for you, GW affiliates can register for a Covidence account here . Alternatively there is a PRISMA flow diagram generator at  http://www.prisma-statement.org/PRISMAStatement/FlowDiagram
• << Previous: Medical Literature Databases to search
• Next: Data Extraction/Coding/Study characteristics/Results >>

• Last Updated: Apr 22, 2024 9:18 AM
• URL: https://guides.himmelfarb.gwu.edu/systematic_review

• Himmelfarb Intranet
• Privacy Notice
• Terms of Use
• GW is committed to digital accessibility. If you experience a barrier that affects your ability to access content on this page, let us know via the Accessibility Feedback Form .
• Himmelfarb Health Sciences Library
• 2300 Eye St., NW, Washington, DC 20037
• Phone: (202) 994-2850
• [email protected]
• https://himmelfarb.gwu.edu

CASE REPORT article

The treatment process of a giant phyllodes tumor of the breast: a case report and review of the literature.

• 1 Department of Breast Center, Mianyang Central Hospital, School of Medicine, University of Electronic Science and Technology of China, Mianyang, China
• 2 Department of Neurology, Mianyang Central Hospital, School of Medicine, University of Electronic Science and Technology of China, Mianyang, China
• 3 Department of Pharmacy, Mianyang Central Hospital, School of Medicine, University of Electronic Science and Technology of China, Mianyang, China

Giant phyllodes tumors are rare fibroepithelial tumors that are usually larger than 10 cm in diameter, have rapid tumor growth, and are easily recurrent. They are frequently accompanied by skin necrosis and infection, particularly in malignant phyllodes tumors. This case report presents a 50-year-old woman who presented to the hospital with a huge left breast mass that was ruptured and infected. The patient received anti-infective treatment and underwent mastectomy and skin grafting, which indicated a malignant phyllodes tumor. The tumor was completely excised after a local recurrence in the chest wall 6 months post-surgery. Unfortunately, one year later, the patient pass away due to multiple organ failure. Giant phyllodes tumor management presents challenges to the surgeon. This case is being presented to enhance understanding and treatment of phyllodes tumors, specifically giant malignant phyllodes tumors, with the aim of improving patients’ quality of life.

1 Introduction

Phyllodes tumors are an uncommon occurrence, comprising less than 1% of all breast tumors ( 1 ). They are biphasic fibroepithelial tumors that encompass both stromal and epithelial components. Typically, these tumors manifest as painless nodules with a propensity for rapid growth within a short duration. The World Health Organization (WHO) has classified phyllodes tumors into three distinct histotypes based on clinicopathological characteristics: benign, borderline, and malignant. These histotypes are observed in approximately 60%-75%, 15%-20%, and 10%-20% of cases, respectively ( 2 ). Most phyllodes tumors are around 4 cm in size, but less than 10% grow larger than 10 cm and are called giant tumors ( 3 ). Giant malignant phyllodes tumors are rare and hard to treat. Surgery remains the primary treatment option, with very few surgical studies on giant phyllodes tumors. This case involved a very aggressive giant malignant phyllodes tumor that recurred, spread, and died within a short period of time despite undergoing mastectomy.

2 Case report

In June 2019, the patient identified a small lump in her left breast without associated discomfort or identifiable etiology, yet she refrained from seeking medical attention or undergoing evaluation. By September 2020, the lump exhibited rapid growth prompting her to present at a clinic where she received a misdiagnosis of mastitis, leading to an incision and drainage procedure. Subsequently, the wound failed to heal, and the tumor extended along the incision site. In November 2020, she was admitted to the Department of Breast Surgery at Mianyang Central Hospital, affiliated with the University of Electronic Science and Technology of China in Mianyang, China. We created a timeline of key points in the patient’s diagnosis and treatment ( Table 1 ). Her vital signs were as follows. Physical examination revealed a large, irregular mass in the left breast measuring 17 cm x 15 cm x 10 cm with unclear borders and rupture ( Figure 1 ). The skin over the mass became necrotic, with a foul odor and oozing fluids. This caused slight pain. The patient did not have a fever. No enlarged lymph nodes were found in the axilla or clavicle. The patient had no other medical history. Her father died from liver and pancreatic cancer. Lab results revealed an abnormal complete blood count, indicating a high white blood cell count of 10,610/μl and a high platelet count of 468,000/μl. The bacterial culture of exudate was positive for Pseudomonas aeruginosa and Enterococcus faecalith. Liver and kidney function, tumor markers, and other tests were normal. The mass had ruptured upon admission, so mammography and ultrasonography were not possible. Contrast-enhanced magnetic resonance imaging (MRI) showed a large mass in the left breast with an unclear border ( Figure 2 ). The imaging studies, including computed tomography scan of the thorax, abdominal ultrasound, and bone scan, did not show any signs of distant metastasis. The results of the core needle biopsy supported the diagnosis of a phyllodes tumor in the breast. The patient received a four-day course of intravenous antibiotics and wound care. The white blood cell count of 7880/μl indicated successful management of the infection before undergoing mastectomy.

Table 1 A timeline of key points in the patient’s diagnosis and treatment.

Figure 1 The preoperative measurement of the left breast mass was 17 cm x 15 cm x 10 cm.

Figure 2 Contrast-enhanced magnetic resonance imaging findings. The boundary between the mass and the pectoralis major muscle was indistinct, with persistent signs of infection observed.

Following standard disinfection and draping procedures, the entirety of the mass was meticulously wrapped and secured using sterile gauze to prevent potential contamination. During the surgery, surgeons noted significant vascularity of the tumor and minor adhesion to the base of the pectoralis major fascia. Subsequently, a horizontal oval incision was made to excise the entire breast, encompassing the nipple, areola, and pectoralis major fascia, while preserving the pectoralis major and pectoralis minor muscles. After excision, the wound underwent meticulous irrigation with warm sterile water to remove any debris or blood. Subsequently, hemostasis was achieved. To facilitate wound healing and chest reconstruction, skin from the abdomen was harvested and transplanted onto the chest. The patient was diagnosed with a malignant phyllodes tumor. Follow-up evaluation confirmed successful removal of breast tissue measuring 20 cm × 17 cm, with no tumor spread to the pectoralis major fascia. The histopathological examination demonstrated typical morphological features of malignant phyllodes tumors, marked stromal cellularity, marked stromal cell anisotropy with a few ductal epithelial component visible in the middle, characteristic leaf-like fronds protruding into cystically dilated fissures, mitotic rate of ≥ 10/10 high power fields, stromal overgrowth, and infiltrative tumor borders ( Figure 3 ). The patient experienced no wound-related complications during hospitalization and was discharged on the 13th day after surgery.

Figure 3 Histological analysis of the surgical specimen demonstrated heightened cellularity surrounding the glandular structures, along with cellular atypia, elevated mitotic activity, as well as hemorrhage and necrosis (HE staining; magnification 100x).

Although postoperative radiotherapy was recommended due to uncertainty about achieving sufficient negative margins for the ruptured giant tumor, the patient declined. In May 2021, the patient presented to Mianyang Central Hospital with a subcutaneous nodule anterior to the sternal body measuring approximately 3.0 cm x 2.5 cm ( Figure 4 ). Subsequently, the patient underwent a localized expended resection of the mass, resulting in a significant defect necessitating an incision in the right inframammary fold to displace the medial right breast gland to the anterior thoracic region, with negative surgical margins. The final histological diagnosis revealed a recurrent malignant phyllodes tumor. Subsequently, the patient declined postoperative radiotherapy despite medical advice. After one year of monitoring, the patient developed liver, lung, and brain metastases, leading to multiple organ failure and death. The reporting of this study follows CARE guidelines ( 4 ).

Figure 4 Computed tomography of the thorax identified a subcutaneous mass anterior to the sternum, measuring approximately 3.0 cm x 2.5 cm, characterized by smooth margins and regular morphology.

3 Discussion

According to the WHO classification, phyllodes tumors are fibroepithelial neoplasms characterized by distinct intracanalicular architectural patterns and leaf-like stromal fronds. These features are accompanied by luminal epithelial and myoepithelial layers and stromal hypercellularity. Phyllodes tumors are stratified into three subgroups (benign, borderline, and malignant) based on criteria including stromal cellularity, stromal atypia, stromal overgrowth, mitotic activity, and tumor margins (as shown in Table 2 ) ( 2 , 5 ). Immunohistochemical analysis has shown positive expression of vascular endothelial growth factor receptor (VEGFR), P53, CD117, P16, and epidermal growth factor receptor (EGFR) in malignant phyllodes tumors. Studies have demonstrated that malignant phyllodes tumors exhibit lower expression of CD34, with CD34 expression showing an inverse correlation with unfavorable histological characteristics ( 6 – 8 ). The primary objective is to differentiate malignant phyllodes tumors from spindle cell metaplastic breast carcinomas and primary breast sarcomas.

Table 2 Histopathological features of benign, borderline and malignant phyllodes tumors of the breast.

Spindle cell metaplastic breast carcinomas may contain varying amounts of malignant epithelial components, with some cases lacking epithelial components entirely or showing heterologous mesenchymal differentiation. The identification of diffuse broad-spectrum cytokeratin or p63 expression in malignant spindle cells may indicate the presence of metaplastic carcinoma, as these proteins are typically not expressed in malignant phyllodes tumors ( 9 , 10 ). Breast sarcomas are a rare subset of malignant tumors that arise from the mesenchymal tissue of the breast, characterized by a lack of distinctive histological features. In contrast to malignant phyllodes tumors, which typically exhibit a sarcomatous pattern in the stromal tissue and a benign epithelial component, breast sarcomas do not display benign epithelial components or leaf-like stromal fronds ( 11 ). Studies have suggested a potential association between genetic predispositions, such as Li-Fraumeni syndrome, and an elevated risk of developing breast sarcomas through TP53 mutations ( 12 , 13 ).

Diagnosis of breast phyllodes tumors prior to surgical intervention remains challenging. Ultrasound and mammography lack specificity in imaging phyllodes tumors. Prior studies have not established the efficacy of MRI in the detection of breast phyllodes tumors ( 14 ). Histopathological assessment remains the gold standard for diagnosing these tumors. Ultrasound-guided core needle biopsy (CNB) has emerged as the preferred method for obtaining tissue samples from suspicious breast lesions for histopathological examination due to its safety, speed, and convenience. Nonetheless, the heterogeneous nature of phyllodes tumors poses a challenge in distinguishing them from epithelial neoplasms or fibroadenomas during biopsy procedures, potentially leading to missed diagnoses. Research indicates that CNB has an approximate 50% accuracy rate in diagnosing phyllodes tumors, with larger tumor size correlating significantly with discordant biopsy results ( 15 ).

Surgical intervention is the mainstay of treatment for phyllodes tumors. Benign phyllodes tumors exhibit no heightened risk of local recurrence following any type of resection. Conversely, mastectomy for borderline and malignant phyllodes tumors has been shown to improve local recurrence rates and disease-free survival ( 16 – 18 ). The majority of studies have demonstrated a lack of statistically significant disparity in overall or cancer-specific survival rates among patients with phyllodes tumors who underwent wide excision compared to mastectomy. Margins are still a matter of debate. While systematic reviews have not found a consistent correlation between positive margins and local recurrence in benign and borderline phyllodes tumors ( 14 ), a negative margin has been consistently associated with decreased recurrence risk in malignant phyllode tumors ( 19 ). Moo et al. discovered that patients with benign phyllodes tumors who underwent re-excision due to positive or close margins did not experience lower rates of local recurrence ( 20 ). The National Comprehensive Cancer Network (NCCN) recommends a negative margin of more than 1cm for malignant phyllodes tumors. However, Thind et al. found no significant difference in local recurrence or survival rates between borderline and malignant phyllodes tumors with margins of 1 cm versus >1cm ( 21 ). Failure to accurately measure tumor margins during surgery, particularly in cases of giant and ruptured tumors with unclear borders, may contribute to early recurrence of phyllodes tumors. It is increasingly acknowledged that the appropriate surgical margin width should be tailored to the specific subtype of the phyllodes tumor. Mastectomy is primarily employed for larger and recurrent tumors, particularly those of a malignant nature. Giant phyllodes tumors typically encompass the entire breast and present challenges in achieving adequate resection margins. It is important to recognize that in cases where phyllodes tumors reach significant size, complete removal of breast tissue and the tumor-infiltrated soft tissues can decrease the likelihood of local tumor recurrence. Patients with positive surgical margins for benign phyllodes tumors could be follow-up closely, while borderline and malignant phyllodes tumors with margins of more than 1cm should be a localized expanded resection. Khosravi-Shahi, P et al. demonstrated that over 90% of malignant phyllodes tumor cases did not exhibit axillary lymph node metastasis, thus supporting the recommendation to avoid axillary surgery ( 22 ).

Certain researchers argue that breast reconstruction may pose a potential risk for phyllodes tumors, which are known for their high recurrence rate. While certain studies suggest that recurrence is not directly linked to breast reconstruction, the possibility of local recurrence could lead to reconstruction failure, necessitating multiple surgeries and imposing additional physical, psychological, and financial strain on patients ( 23 , 24 ). Therefore, timely surgical intervention is recommended for giant borderline or malignant phyllodes tumors to prevent complications such as tumor compression of the chest wall and invasion of the thoracic cavity. Preoperative assessment of tumor compression or invasion of the chest wall is essential ( 25 ). Surgical procedures should be performed cautiously to ensure complete excision of the tumor and to prevent residual disease or compromise of the tumor’s integrity, which may result in short-term recurrence ( 26 ). Careful handling of the tumor envelope during surgery is crucial, requiring blunt separation to maintain its integrity. In cases where the envelope is compromised, meticulous identification and dissection of the tumor, along with demarcation of normal tissue, are imperative. Preservation of normal skin and muscle is a priority to adequately cover the wound during complete tumor dissection and attainment of negative margins. For cases where the tumor was ulcerated prior to surgery, it is recommended to meticulously cover and isolate the ulcerated area with gauze during the procedure to prevent contamination of the operative site. Management of large skin defects may necessitate plastic restoration techniques such as skin grafts or rotation flaps.

Adjuvant radiotherapy has been shown to decrease local recurrence rates in borderline and malignant phyllodes tumors, especially following breast-conserving surgery, and is particularly beneficial for individuals under 45 years of age and those with tumors larger than 5cm ( 27 , 28 ). Nevertheless, there is currently no evidence supporting the efficacy of postoperative adjuvant radiotherapy in improving overall survival for primary phyllodes tumors, as indicated by the NCCN guidelines. The efficacy of adjuvant chemotherapy, endocrine therapy, and targeted therapy for phyllodes tumors remains unproven.

Phyllodes tumors predominantly metastasize hematogenously, with distant metastases occurring in 9% of malignant cases, most commonly in the lungs and bones. Recurrence rates of 10-17%, 14-25%, and 23-30% have been reported in the literature for benign, borderline, and malignant phyllodes tumors, respectively ( 29 ). Additionally, the pathology of recurrent tumors often presents a more severe condition. Patients afflicted with malignant phyllode tumors who experience distant metastases face a grim prognosis, with an average survival time of 4 to 17 months ( 30 ). Research has not identified a disparity in overall survival rates between mastectomy and locally extended resection following recurrence. Therefore, in accordance with NCCN guidelines, wide margin re-excision and postoperative radiotherapy may be deemed appropriate for cases of local recurrence subsequent to the resection of malignant phyllode tumors.

This particular case study serves to enhance the knowledge of diagnosing and treating malignant phyllodes tumors, as well as examining the surgical techniques used for managing large malignant phyllodes tumors. The preservation of the mammoplasty approach may be considered as a viable option when sufficient margins are achieved. Mastectomy may be considered as a treatment option for giant phyllodes tumors in cases where oncoplastic excision fails to yield satisfactory reconstructive outcomes. Limited research exists on the efficacy of combining mastectomy with breast reconstruction in such cases. The complexity of tumor localization poses challenges for reconstruction, necessitating careful preoperative planning to ensure safe and satisfactory oncoplastic outcomes.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by The studies involving humans were approved by Ethics Committee of Mianyang Central Hospital (S20230213-01). The studies were conducted in accordance with the local legislation and institutional requirements. The patient's personal information has been deidentified. Prior to the study, patient consent to treatment was obtained. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author contributions

YT: Conceptualization, Writing – original draft. SL: Conceptualization, Writing – original draft. LZ: Writing – review & editing. ZZ: Writing – review & editing. HH: Conceptualization, Writing – review & editing. YJ: Data curation, Formal analysis, Writing – review & editing. TL: Data curation, Formal analysis, Writing – review & editing.

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This study was supported by Wu Jieping Medical Foundation (Grant No.320.6750.2023-18-102).

Acknowledgments

We appreciate all the reviewers who participated in the review.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

1. Mishra SP, Tiwary SK, Mishra M, Khanna AK. Phyllodes tumor of breast: a review article. ISRN Surg . (2013) 2013:361469. doi: 10.1155/2013/361469

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Tan PH, Ellis I, Allison K, Brogi E, Fox SB, Lakhani S, et al. The 2019 World Health Organization classification of tumours of the breast. Histopathology . (2020) 77:181–5. doi: 10.1111/his.14091

3. Lissidini G, Papa G, Ashoor AA, Girardi A, Crown A, Toesca A, et al. Giant Malignant phyllodes tumor of the breast: insights into optimal management and reconstructive options for a rare and challenging entity. Breast J . (2020) 26:2400–2. doi: 10.1111/tbj.14095

4. Gagnier JJ, Kienle G, Altman DG, Moher D, Sox H, Riley D. The CARE guidelines: consensus-based clinical case reporting guideline development. Glob Adv Health Med . (2013) 2:38–43. doi: 10.7453/gahmj.2013.008

5. Tan PH, Ellis IO. Myoepithelial and epithelial-myoepithelial, mesenchymal and fibroepithelial breast lesions: updates from the WHO Classification of Tumours of the Breast 2012. J Clin Pathol . (2013) 66:465–70. doi: 10.1136/jclinpath-2012-201078

6. Vilela MH, de Almeida FM, de Paula GM, Ribeiro NB, Cirqueira MB, Silva AL, et al. Utility of Ki-67, CD10, CD34, p53, CD117, and mast cell content in the differential diagnosis of cellular fibroadenomas and in the classification of phyllodes tumors of the breast. Int J Surg Pathol . (2014) 22:485–91. doi: 10.1177/1066896914521290

7. Tse GM, Lui PC, Vong JS, Lau KM, Putti TC, Karim R, et al. Increased epidermal growth factor receptor (EGFR) expression in Malignant mammary phyllodes tumors. Breast Cancer Res Treat . (2009) 114:441–8. doi: 10.1007/s10549-008-0030-5

8. Niezabitowski A, Lackowska B, Rys J, Kruczak A, Kowalska T, Mitus J, et al. Prognostic evaluation of proliferative activity and DNA content in the phyllodes tumor of the breast: immunohistochemical and flow cytometric study of 118 cases. Breast Cancer Res Treat . (2001) 65:77–85. doi: 10.1023/a:1006457304526

9. Tse GM, Tan PH, Chaiwun B, Putti TC, Lui PC, Tsang AK, et al. p63 is useful in the diagnosis of mammary metaplastic carcinomas. Pathology . (2006) 38:16–20. doi: 10.1080/00313020500444625

10. Zhang Y, Kleer CG. Phyllodes tumor of the breast: histopathologic features, differential diagnosis, and molecular/genetic updates. Arch Pathol Lab Med . (2016) 140:665–71. doi: 10.5858/arpa.2016-0042-RA

11. Lim SZ, Ng CCY, Rajasegaran V, Guan P, Selvarajan S, Thike AA, et al. Genomic profile of breast sarcomas: a comparison with Malignant phyllodes tumours. Breast Cancer Res Treat . (2019) 174:365–373g. doi: 10.1007/s10549-018-5067-5

12. Malkin D, Li FP, Strong LC, Fraumeni JFJR, Nelson CE, Kim DH, et al. Germ line p53 mutations in a familial syndrome of breast cancer, sarcomas, and other neoplasms. Science . (1990) 250:1233–8. doi: 10.1126/science.1978757

13. Lahat G, Lazar A, Lev D. Sarcoma epidemiology and etiology: potential environmental and genetic factors. Surg Clin North Am . (2008) 250:451–81. doi: 10.1016/j.suc.2008.03.006

CrossRef Full Text | Google Scholar

14. Bogach J, Shakeel S, Wright FC, Hong NJL. Phyllodes tumors: A scoping review of the literature. Ann Surg Oncol . (2022) 29:446–59. doi: 10.1245/s10434-021-10468-2

15. Zhou ZR, Wang CC, Yang ZZ, Yu XL, Guo XM. Phyllodes tumors of the breast: diagnosis, treatment and prognostic factors related to recurrence. J Thorac Dis . (2016) 8:3361–8. doi: 10.21037/jtd.2016.11.03

16. Ben Hassouna J, Damak T, Gamoudi A, Chargui R, Khomsi F, Mahjoub S, et al. Phyllodes tumors of the breast: a case series of 106 patients. Am J Surg . (2006) 192:141–7. doi: 10.1016/j.amjsurg.2006.04.007

17. Kim S, Kim JY, Kim DH, Jung WH, Koo JS. Analysis of phyllodes tumor recurrence according to the histologic grade. Breast Cancer Res Treat . (2013) 141:353–63. doi: 10.1007/s10549-013-2684-x

18. Lim RS, Cordeiro E, Lau J, Lim A, Roberts A, Seely J. Phyllodes tumors-the predictors and detection of recurrence. Can Assoc Radiol J . (2021) 72:251–7. doi: 10.1177/0846537119899553

19. Spitaleri G, Toesca A, Botteri E, Bottiglieri L, Rotmensz N, Boselli S, et al. Breast phyllodes tumor: a review of literature and a single center retrospective series analysis. Crit Rev Oncol Hematol . (2013) 88:427–36. doi: 10.1016/j.critrevonc.2013.06.005

20. Moo TA, Alabdulkareem H, Tam A, Fontanet C, Lu Y, Landers A, et al. Association between recurrence and re-excision for close and positive margins versus observation in patients with benign phyllodes tumors. Ann Surg Oncol . (2017) 24:3088–92. doi: 10.1245/s10434-017-5955-7

21. Thind A, Patel B, Thind K, Isherwood J, Phillips B, Dhaliwal K, et al. Surgical margins for borderline and Malignant phyllodes tumours. Ann R Coll Surg Engl . (2020) 102:165–73. doi: 10.1308/rcsann.2019.0140

22. Khosravi-Shahi P. Management of non metastatic phyllodes tumors of the breast: review of the literature. Surg Oncol . (2011) 20:e143–8. doi: 10.1016/j.suronc.2011.04.007

23. Pimiento JM, Gadgil PV, Santillan AA, Lee MC, Esposito NN, Kiluk JV, et al. Phyllodes tumors: race-related differences. J Am Coll Surg . (2011) 213:537–42. doi: 10.1016/j.jamcollsurg.2011.07.012

24. Guillot E, Couturaud B, Reyal F, Curnier A, Ravinet J, Laé M, et al. Management of phyllodes breast tumors. Breast J . (2011) 17:129–37. doi: 10.1111/tbj.2011.17.issue-2

25. Kudela E, Kajo K, Kozubik E, Rokos T, Pribulova T, Danko J, et al. Recurrent giant Malignant phyllodes tumor of the breast. Case Rep Obstet Gynecol . (2021) 2021:2476691. doi: 10.1155/2021/2476691

26. Tremblay-LeMay R, Hogue JC, Provencher L, Poirier B, Poirier É, Laberge S, et al. How wide should margins be for phyllodes tumors of the breast? Breast J . (2017) 23:315–22. doi: 10.1111/tbj.12727

27. Barth RJ Jr., Wells WA, Mitchell SE, Cole BF. A prospective, multi-institutional study of adjuvant radiotherapy after resection of Malignant phyllodes tumors. Ann Surg Oncol . (2009) 16:2288–94. doi: 10.1245/s10434-009-0489-2

28. Zeng S, Zhang X, Yang D, Wang X, Ren G. Effects of adjuvant radiotherapy on borderline and Malignant phyllodes tumors: A systematic review and meta-analysis. Mol Clin Oncol . (2015) 3:663–71. doi: 10.3892/mco.2015.503

29. Tan BY, Acs G, Apple SK, Badve S, Bleiweiss IJ, Brogi E, et al. Phyllodes tumours of the breast: a consensus review. Histopathology . (2016) 68:5–21. doi: 10.1111/his.12876

30. Wei Y, Yu Y, Ji Y, Zhong Y, Min N, Hu H, et al. Surgical management in phyllodes tumors of the breast: a systematic review and meta-analysis. Gland Surg . (2022) 11:513–23. doi: 10.21037/gs-21-789

Keywords: giant phyllodes tumor, recurrence, diagnosis, treatment, surgery

Citation: Tong Y, Liu S, Zhao L, Zhang Z, Hu H, Jing Y and Liang T (2024) The treatment process of a giant phyllodes tumor of the breast: a case report and review of the literature. Front. Oncol. 14:1382985. doi: 10.3389/fonc.2024.1382985

Received: 06 February 2024; Accepted: 17 April 2024; Published: 30 April 2024.

Reviewed by:

Copyright © 2024 Tong, Liu, Zhao, Zhang, Hu, Jing and Liang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Tiantian Liang, [email protected] ; Yu Jing, [email protected]

† These authors have contributed equally to this work and share first authorship

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.