example methodology literature review

Get science-backed answers as you write with Paperpal's Research feature

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

• What is the purpose of literature review? 
• a. Habitat Loss and Species Extinction: 
• b. Range Shifts and Phenological Changes: 
• c. Ocean Acidification and Coral Reefs: 
• d. Adaptive Strategies and Conservation Efforts: 
• How to write a good literature review 
• Choose a Topic and Define the Research Question: 
• Decide on the Scope of Your Review: 
• Select Databases for Searches: 
• Conduct Searches and Keep Track: 
• Review the Literature: 
• Organize and Write Your Literature Review: 
• Frequently asked questions 

What is a literature review?

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

What is the purpose of literature review?

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

• Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 
• Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field. 
• Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 
• Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 
• Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 
• Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

a. Habitat Loss and Species Extinction:

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

b. Range Shifts and Phenological Changes:

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

c. Ocean Acidification and Coral Reefs:

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

d. Adaptive Strategies and Conservation Efforts:

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

How to write a good literature review

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

• Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
• Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
• Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
• Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
• Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

• Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
• Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
• Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
• Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
• Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

• Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
• Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
• Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
• Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
• Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

Conducting a literature review

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:

• Select a topic that is relevant to your field of study. 
• Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:

• Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
• Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
• Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:

• Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
• Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:

• Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
• Record and document your search strategy for transparency and replicability. 
• Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:

• Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
• Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
• Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
• Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
• Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:

• Literature review outline should be based on themes, chronological order, or methodological approaches. 
• Write a clear and coherent narrative that synthesizes the information gathered. 
• Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
• Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. 

Frequently asked questions

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 

References 

• Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
• Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
• Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is an AI writing assistant that help academics write better, faster with real-time suggestions for in-depth language and grammar correction. Trained on millions of research manuscripts enhanced by professional academic editors, Paperpal delivers human precision at machine speed.  

Try it for free or upgrade to  Paperpal Prime , which unlocks unlimited access to premium features like academic translation, paraphrasing, contextual synonyms, consistency checks and more. It’s like always having a professional academic editor by your side! Go beyond limitations and experience the future of academic writing.  Get Paperpal Prime now at just US$19 a month!

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• Life Sciences Papers: 9 Tips for Authors Writing in Biological Sciences
• What is an Argumentative Essay? How to Write It (With Examples)

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, 4 ways paperpal encourages responsible writing with ai, what are scholarly sources and where can you..., how to write a hypothesis types and examples , measuring academic success: definition & strategies for excellence, what is academic writing: tips for students, why traditional editorial process needs an upgrade, paperpal’s new ai research finder empowers authors to..., what is hedging in academic writing  , how to use ai to enhance your college..., ai + human expertise – a paradigm shift....

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

How To Structure Your Literature Review

3 options to help structure your chapter.

By: Amy Rommelspacher (PhD) | Reviewer: Dr Eunice Rautenbach | November 2020 (Updated May 2023)

Writing the literature review chapter can seem pretty daunting when you’re piecing together your dissertation or thesis. As  we’ve discussed before , a good literature review needs to achieve a few very important objectives – it should:

• Demonstrate your knowledge of the research topic
• Identify the gaps in the literature and show how your research links to these
• Provide the foundation for your conceptual framework (if you have one)
• Inform your own  methodology and research design

To achieve this, your literature review needs a well-thought-out structure . Get the structure of your literature review chapter wrong and you’ll struggle to achieve these objectives. Don’t worry though – in this post, we’ll look at how to structure your literature review for maximum impact (and marks!).

But wait – is this the right time?

Deciding on the structure of your literature review should come towards the end of the literature review process – after you have collected and digested the literature, but before you start writing the chapter. 

In other words, you need to first develop a rich understanding of the literature before you even attempt to map out a structure. There’s no use trying to develop a structure before you’ve fully wrapped your head around the existing research.

Equally importantly, you need to have a structure in place before you start writing , or your literature review will most likely end up a rambling, disjointed mess. 

Importantly, don’t feel that once you’ve defined a structure you can’t iterate on it. It’s perfectly natural to adjust as you engage in the writing process. As we’ve discussed before , writing is a way of developing your thinking, so it’s quite common for your thinking to change – and therefore, for your chapter structure to change – as you write. 

Need a helping hand?

Like any other chapter in your thesis or dissertation, your literature review needs to have a clear, logical structure. At a minimum, it should have three essential components – an  introduction , a  body   and a  conclusion . 

Let’s take a closer look at each of these.

1: The Introduction Section

Just like any good introduction, the introduction section of your literature review should introduce the purpose and layout (organisation) of the chapter. In other words, your introduction needs to give the reader a taste of what’s to come, and how you’re going to lay that out. Essentially, you should provide the reader with a high-level roadmap of your chapter to give them a taste of the journey that lies ahead.

Here’s an example of the layout visualised in a literature review introduction:

Your introduction should also outline your topic (including any tricky terminology or jargon) and provide an explanation of the scope of your literature review – in other words, what you  will   and  won’t   be covering (the delimitations ). This helps ringfence your review and achieve a clear focus . The clearer and narrower your focus, the deeper you can dive into the topic (which is typically where the magic lies). 

Depending on the nature of your project, you could also present your stance or point of view at this stage. In other words, after grappling with the literature you’ll have an opinion about what the trends and concerns are in the field as well as what’s lacking. The introduction section can then present these ideas so that it is clear to examiners that you’re aware of how your research connects with existing knowledge .

2: The Body Section

The body of your literature review is the centre of your work. This is where you’ll present, analyse, evaluate and synthesise the existing research. In other words, this is where you’re going to earn (or lose) the most marks. Therefore, it’s important to carefully think about how you will organise your discussion to present it in a clear way. 

The body of your literature review should do just as the description of this chapter suggests. It should “review” the literature – in other words, identify, analyse, and synthesise it. So, when thinking about structuring your literature review, you need to think about which structural approach will provide the best “review” for your specific type of research and objectives (we’ll get to this shortly).

There are (broadly speaking)  three options  for organising your literature review.

Option 1: Chronological (according to date)

Organising the literature chronologically is one of the simplest ways to structure your literature review. You start with what was published first and work your way through the literature until you reach the work published most recently. Pretty straightforward.

The benefit of this option is that it makes it easy to discuss the developments and debates in the field as they emerged over time. Organising your literature chronologically also allows you to highlight how specific articles or pieces of work might have changed the course of the field – in other words, which research has had the most impact . Therefore, this approach is very useful when your research is aimed at understanding how the topic has unfolded over time and is often used by scholars in the field of history. That said, this approach can be utilised by anyone that wants to explore change over time .

For example , if a student of politics is investigating how the understanding of democracy has evolved over time, they could use the chronological approach to provide a narrative that demonstrates how this understanding has changed through the ages.

Here are some questions you can ask yourself to help you structure your literature review chronologically.

• What is the earliest literature published relating to this topic?
• How has the field changed over time? Why?
• What are the most recent discoveries/theories?

In some ways, chronology plays a part whichever way you decide to structure your literature review, because you will always, to a certain extent, be analysing how the literature has developed. However, with the chronological approach, the emphasis is very firmly on how the discussion has evolved over time , as opposed to how all the literature links together (which we’ll discuss next ).

Option 2: Thematic (grouped by theme)

The thematic approach to structuring a literature review means organising your literature by theme or category – for example, by independent variables (i.e. factors that have an impact on a specific outcome).

As you’ve been collecting and synthesising literature , you’ll likely have started seeing some themes or patterns emerging. You can then use these themes or patterns as a structure for your body discussion. The thematic approach is the most common approach and is useful for structuring literature reviews in most fields.

For example, if you were researching which factors contributed towards people trusting an organisation, you might find themes such as consumers’ perceptions of an organisation’s competence, benevolence and integrity. Structuring your literature review thematically would mean structuring your literature review’s body section to discuss each of these themes, one section at a time.

Here are some questions to ask yourself when structuring your literature review by themes:

• Are there any patterns that have come to light in the literature?
• What are the central themes and categories used by the researchers?
• Do I have enough evidence of these themes?

PS – you can see an example of a thematically structured literature review in our literature review sample walkthrough video here.

Option 3: Methodological

The methodological option is a way of structuring your literature review by the research methodologies used . In other words, organising your discussion based on the angle from which each piece of research was approached – for example, qualitative , quantitative or mixed  methodologies.

Structuring your literature review by methodology can be useful if you are drawing research from a variety of disciplines and are critiquing different methodologies. The point of this approach is to question  how  existing research has been conducted, as opposed to  what  the conclusions and/or findings the research were.

For example, a sociologist might centre their research around critiquing specific fieldwork practices. Their literature review will then be a summary of the fieldwork methodologies used by different studies.

Here are some questions you can ask yourself when structuring your literature review according to methodology:

• Which methodologies have been utilised in this field?
• Which methodology is the most popular (and why)?
• What are the strengths and weaknesses of the various methodologies?
• How can the existing methodologies inform my own methodology?

3: The Conclusion Section

Once you’ve completed the body section of your literature review using one of the structural approaches we discussed above, you’ll need to “wrap up” your literature review and pull all the pieces together to set the direction for the rest of your dissertation or thesis.

The conclusion is where you’ll present the key findings of your literature review. In this section, you should emphasise the research that is especially important to your research questions and highlight the gaps that exist in the literature. Based on this, you need to make it clear what you will add to the literature – in other words, justify your own research by showing how it will help fill one or more of the gaps you just identified.

Last but not least, if it’s your intention to develop a conceptual framework for your dissertation or thesis, the conclusion section is a good place to present this.

Example: Thematically Structured Review

In the video below, we unpack a literature review chapter so that you can see an example of a thematically structure review in practice.

Let’s Recap

In this article, we’ve  discussed how to structure your literature review for maximum impact. Here’s a quick recap of what  you need to keep in mind when deciding on your literature review structure:

• Just like other chapters, your literature review needs a clear introduction , body and conclusion .
• The introduction section should provide an overview of what you will discuss in your literature review.
• The body section of your literature review can be organised by chronology , theme or methodology . The right structural approach depends on what you’re trying to achieve with your research.
• The conclusion section should draw together the key findings of your literature review and link them to your research questions.

If you’re ready to get started, be sure to download our free literature review template to fast-track your chapter outline.

Psst… there’s more!

This post is an extract from our bestselling short course, Literature Review Bootcamp . If you want to work smart, you don't want to miss this .

You Might Also Like:

27 Comments

Great work. This is exactly what I was looking for and helps a lot together with your previous post on literature review. One last thing is missing: a link to a great literature chapter of an journal article (maybe with comments of the different sections in this review chapter). Do you know any great literature review chapters?

I agree with you Marin… A great piece

I agree with Marin. This would be quite helpful if you annotate a nicely structured literature from previously published research articles.

Awesome article for my research.

I thank you immensely for this wonderful guide

It is indeed thought and supportive work for the futurist researcher and students

Very educative and good time to get guide. Thank you

Great work, very insightful. Thank you.

Thanks for this wonderful presentation. My question is that do I put all the variables into a single conceptual framework or each hypothesis will have it own conceptual framework?

Thank you very much, very helpful

This is very educative and precise . Thank you very much for dropping this kind of write up .

Pheeww, so damn helpful, thank you for this informative piece.

I’m doing a research project topic ; stool analysis for parasitic worm (enteric) worm, how do I structure it, thanks.

comprehensive explanation. Help us by pasting the URL of some good “literature review” for better understanding.

great piece. thanks for the awesome explanation. it is really worth sharing. I have a little question, if anyone can help me out, which of the options in the body of literature can be best fit if you are writing an architectural thesis that deals with design?

I am doing a research on nanofluids how can l structure it?

Beautifully clear.nThank you!

Lucid! Thankyou!

Brilliant work, well understood, many thanks

I like how this was so clear with simple language 😊😊 thank you so much 😊 for these information 😊

Insightful. I was struggling to come up with a sensible literature review but this has been really helpful. Thank you!

You have given thought-provoking information about the review of the literature.

Thank you. It has made my own research better and to impart your work to students I teach

I learnt a lot from this teaching. It’s a great piece.

I am doing research on EFL teacher motivation for his/her job. How Can I structure it? Is there any detailed template, additional to this?

You are so cool! I do not think I’ve read through something like this before. So nice to find somebody with some genuine thoughts on this issue. Seriously.. thank you for starting this up. This site is one thing that is required on the internet, someone with a little originality!

I’m asked to do conceptual, theoretical and empirical literature, and i just don’t know how to structure it

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What Is a Research Methodology? | Steps & Tips

What Is a Research Methodology? | Steps & Tips

Published on 25 February 2019 by Shona McCombes . Revised on 10 October 2022.

Your research methodology discusses and explains the data collection and analysis methods you used in your research. A key part of your thesis, dissertation, or research paper, the methodology chapter explains what you did and how you did it, allowing readers to evaluate the reliability and validity of your research.

It should include:

• The type of research you conducted
• How you collected and analysed your data
• Any tools or materials you used in the research
• Why you chose these methods
• Your methodology section should generally be written in the past tense .
• Academic style guides in your field may provide detailed guidelines on what to include for different types of studies.
• Your citation style might provide guidelines for your methodology section (e.g., an APA Style methods section ).

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

How to write a research methodology, why is a methods section important, step 1: explain your methodological approach, step 2: describe your data collection methods, step 3: describe your analysis method, step 4: evaluate and justify the methodological choices you made, tips for writing a strong methodology chapter, frequently asked questions about methodology.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Your methods section is your opportunity to share how you conducted your research and why you chose the methods you chose. It’s also the place to show that your research was rigorously conducted and can be replicated .

It gives your research legitimacy and situates it within your field, and also gives your readers a place to refer to if they have any questions or critiques in other sections.

You can start by introducing your overall approach to your research. You have two options here.

Option 1: Start with your “what”

What research problem or question did you investigate?

• Aim to describe the characteristics of something?
• Explore an under-researched topic?
• Establish a causal relationship?

And what type of data did you need to achieve this aim?

• Quantitative data , qualitative data , or a mix of both?
• Primary data collected yourself, or secondary data collected by someone else?
• Experimental data gathered by controlling and manipulating variables, or descriptive data gathered via observations?

Option 2: Start with your “why”

Depending on your discipline, you can also start with a discussion of the rationale and assumptions underpinning your methodology. In other words, why did you choose these methods for your study?

• Why is this the best way to answer your research question?
• Is this a standard methodology in your field, or does it require justification?
• Were there any ethical considerations involved in your choices?
• What are the criteria for validity and reliability in this type of research ?

Once you have introduced your reader to your methodological approach, you should share full details about your data collection methods .

Quantitative methods

In order to be considered generalisable, you should describe quantitative research methods in enough detail for another researcher to replicate your study.

Here, explain how you operationalised your concepts and measured your variables. Discuss your sampling method or inclusion/exclusion criteria, as well as any tools, procedures, and materials you used to gather your data.

Surveys Describe where, when, and how the survey was conducted.

• How did you design the questionnaire?
• What form did your questions take (e.g., multiple choice, Likert scale )?
• Were your surveys conducted in-person or virtually?
• What sampling method did you use to select participants?
• What was your sample size and response rate?

Experiments Share full details of the tools, techniques, and procedures you used to conduct your experiment.

• How did you design the experiment ?
• How did you recruit participants?
• How did you manipulate and measure the variables ?
• What tools did you use?

Existing data Explain how you gathered and selected the material (such as datasets or archival data) that you used in your analysis.

• Where did you source the material?
• How was the data originally produced?
• What criteria did you use to select material (e.g., date range)?

The survey consisted of 5 multiple-choice questions and 10 questions measured on a 7-point Likert scale.

The goal was to collect survey responses from 350 customers visiting the fitness apparel company’s brick-and-mortar location in Boston on 4–8 July 2022, between 11:00 and 15:00.

Here, a customer was defined as a person who had purchased a product from the company on the day they took the survey. Participants were given 5 minutes to fill in the survey anonymously. In total, 408 customers responded, but not all surveys were fully completed. Due to this, 371 survey results were included in the analysis.

Qualitative methods

In qualitative research , methods are often more flexible and subjective. For this reason, it’s crucial to robustly explain the methodology choices you made.

Be sure to discuss the criteria you used to select your data, the context in which your research was conducted, and the role you played in collecting your data (e.g., were you an active participant, or a passive observer?)

Interviews or focus groups Describe where, when, and how the interviews were conducted.

• How did you find and select participants?
• How many participants took part?
• What form did the interviews take ( structured , semi-structured , or unstructured )?
• How long were the interviews?
• How were they recorded?

Participant observation Describe where, when, and how you conducted the observation or ethnography .

• What group or community did you observe? How long did you spend there?
• How did you gain access to this group? What role did you play in the community?
• How long did you spend conducting the research? Where was it located?
• How did you record your data (e.g., audiovisual recordings, note-taking)?

Existing data Explain how you selected case study materials for your analysis.

• What type of materials did you analyse?
• How did you select them?

In order to gain better insight into possibilities for future improvement of the fitness shop’s product range, semi-structured interviews were conducted with 8 returning customers.

Here, a returning customer was defined as someone who usually bought products at least twice a week from the store.

Surveys were used to select participants. Interviews were conducted in a small office next to the cash register and lasted approximately 20 minutes each. Answers were recorded by note-taking, and seven interviews were also filmed with consent. One interviewee preferred not to be filmed.

Mixed methods

Mixed methods research combines quantitative and qualitative approaches. If a standalone quantitative or qualitative study is insufficient to answer your research question, mixed methods may be a good fit for you.

Mixed methods are less common than standalone analyses, largely because they require a great deal of effort to pull off successfully. If you choose to pursue mixed methods, it’s especially important to robustly justify your methods here.

Next, you should indicate how you processed and analysed your data. Avoid going into too much detail: you should not start introducing or discussing any of your results at this stage.

In quantitative research , your analysis will be based on numbers. In your methods section, you can include:

• How you prepared the data before analysing it (e.g., checking for missing data , removing outliers , transforming variables)
• Which software you used (e.g., SPSS, Stata or R)
• Which statistical tests you used (e.g., two-tailed t test , simple linear regression )

In qualitative research, your analysis will be based on language, images, and observations (often involving some form of textual analysis ).

Specific methods might include:

• Content analysis : Categorising and discussing the meaning of words, phrases and sentences
• Thematic analysis : Coding and closely examining the data to identify broad themes and patterns
• Discourse analysis : Studying communication and meaning in relation to their social context

Mixed methods combine the above two research methods, integrating both qualitative and quantitative approaches into one coherent analytical process.

Above all, your methodology section should clearly make the case for why you chose the methods you did. This is especially true if you did not take the most standard approach to your topic. In this case, discuss why other methods were not suitable for your objectives, and show how this approach contributes new knowledge or understanding.

In any case, it should be overwhelmingly clear to your reader that you set yourself up for success in terms of your methodology’s design. Show how your methods should lead to results that are valid and reliable, while leaving the analysis of the meaning, importance, and relevance of your results for your discussion section .

• Quantitative: Lab-based experiments cannot always accurately simulate real-life situations and behaviours, but they are effective for testing causal relationships between variables .
• Qualitative: Unstructured interviews usually produce results that cannot be generalised beyond the sample group , but they provide a more in-depth understanding of participants’ perceptions, motivations, and emotions.
• Mixed methods: Despite issues systematically comparing differing types of data, a solely quantitative study would not sufficiently incorporate the lived experience of each participant, while a solely qualitative study would be insufficiently generalisable.

Remember that your aim is not just to describe your methods, but to show how and why you applied them. Again, it’s critical to demonstrate that your research was rigorously conducted and can be replicated.

1. Focus on your objectives and research questions

The methodology section should clearly show why your methods suit your objectives  and convince the reader that you chose the best possible approach to answering your problem statement and research questions .

2. Cite relevant sources

Your methodology can be strengthened by referencing existing research in your field. This can help you to:

• Show that you followed established practice for your type of research
• Discuss how you decided on your approach by evaluating existing research
• Present a novel methodological approach to address a gap in the literature

3. Write for your audience

Consider how much information you need to give, and avoid getting too lengthy. If you are using methods that are standard for your discipline, you probably don’t need to give a lot of background or justification.

Regardless, your methodology should be a clear, well-structured text that makes an argument for your approach, not just a list of technical details and procedures.

Methodology refers to the overarching strategy and rationale of your research. Developing your methodology involves studying the research methods used in your field and the theories or principles that underpin them, in order to choose the approach that best matches your objectives.

Methods are the specific tools and procedures you use to collect and analyse data (e.g. interviews, experiments , surveys , statistical tests ).

In a dissertation or scientific paper, the methodology chapter or methods section comes after the introduction and before the results , discussion and conclusion .

Depending on the length and type of document, you might also include a literature review or theoretical framework before the methodology.

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to test a hypothesis by systematically collecting and analysing data, while qualitative methods allow you to explore ideas and experiences in depth.

A sample is a subset of individuals from a larger population. Sampling means selecting the group that you will actually collect data from in your research.

For example, if you are researching the opinions of students in your university, you could survey a sample of 100 students.

Statistical sampling allows you to test a hypothesis about the characteristics of a population. There are various sampling methods you can use to ensure that your sample is representative of the population as a whole.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, October 10). What Is a Research Methodology? | Steps & Tips. Scribbr. Retrieved 29 April 2024, from https://www.scribbr.co.uk/thesis-dissertation/methodology/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a literature review | guide, template, & examples, what is a theoretical framework | a step-by-step guide.

• Methodology
• Open access
• Published: 11 October 2016

Reviewing the research methods literature: principles and strategies illustrated by a systematic overview of sampling in qualitative research

• Stephen J. Gentles 1 , 4 ,
• Cathy Charles 1 ,
• David B. Nicholas 2 ,
• Jenny Ploeg 3 &
• K. Ann McKibbon 1  

Systematic Reviews volume  5 , Article number:  172 ( 2016 ) Cite this article

51k Accesses

26 Citations

13 Altmetric

Metrics details

Overviews of methods are potentially useful means to increase clarity and enhance collective understanding of specific methods topics that may be characterized by ambiguity, inconsistency, or a lack of comprehensiveness. This type of review represents a distinct literature synthesis method, although to date, its methodology remains relatively undeveloped despite several aspects that demand unique review procedures. The purpose of this paper is to initiate discussion about what a rigorous systematic approach to reviews of methods, referred to here as systematic methods overviews , might look like by providing tentative suggestions for approaching specific challenges likely to be encountered. The guidance offered here was derived from experience conducting a systematic methods overview on the topic of sampling in qualitative research.

The guidance is organized into several principles that highlight specific objectives for this type of review given the common challenges that must be overcome to achieve them. Optional strategies for achieving each principle are also proposed, along with discussion of how they were successfully implemented in the overview on sampling. We describe seven paired principles and strategies that address the following aspects: delimiting the initial set of publications to consider, searching beyond standard bibliographic databases, searching without the availability of relevant metadata, selecting publications on purposeful conceptual grounds, defining concepts and other information to abstract iteratively, accounting for inconsistent terminology used to describe specific methods topics, and generating rigorous verifiable analytic interpretations. Since a broad aim in systematic methods overviews is to describe and interpret the relevant literature in qualitative terms, we suggest that iterative decision making at various stages of the review process, and a rigorous qualitative approach to analysis are necessary features of this review type.

Conclusions

We believe that the principles and strategies provided here will be useful to anyone choosing to undertake a systematic methods overview. This paper represents an initial effort to promote high quality critical evaluations of the literature regarding problematic methods topics, which have the potential to promote clearer, shared understandings, and accelerate advances in research methods. Further work is warranted to develop more definitive guidance.

Peer Review reports

While reviews of methods are not new, they represent a distinct review type whose methodology remains relatively under-addressed in the literature despite the clear implications for unique review procedures. One of few examples to describe it is a chapter containing reflections of two contributing authors in a book of 21 reviews on methodological topics compiled for the British National Health Service, Health Technology Assessment Program [ 1 ]. Notable is their observation of how the differences between the methods reviews and conventional quantitative systematic reviews, specifically attributable to their varying content and purpose, have implications for defining what qualifies as systematic. While the authors describe general aspects of “systematicity” (including rigorous application of a methodical search, abstraction, and analysis), they also describe a high degree of variation within the category of methods reviews itself and so offer little in the way of concrete guidance. In this paper, we present tentative concrete guidance, in the form of a preliminary set of proposed principles and optional strategies, for a rigorous systematic approach to reviewing and evaluating the literature on quantitative or qualitative methods topics. For purposes of this article, we have used the term systematic methods overview to emphasize the notion of a systematic approach to such reviews.

The conventional focus of rigorous literature reviews (i.e., review types for which systematic methods have been codified, including the various approaches to quantitative systematic reviews [ 2 – 4 ], and the numerous forms of qualitative and mixed methods literature synthesis [ 5 – 10 ]) is to synthesize empirical research findings from multiple studies. By contrast, the focus of overviews of methods, including the systematic approach we advocate, is to synthesize guidance on methods topics. The literature consulted for such reviews may include the methods literature, methods-relevant sections of empirical research reports, or both. Thus, this paper adds to previous work published in this journal—namely, recent preliminary guidance for conducting reviews of theory [ 11 ]—that has extended the application of systematic review methods to novel review types that are concerned with subject matter other than empirical research findings.

Published examples of methods overviews illustrate the varying objectives they can have. One objective is to establish methodological standards for appraisal purposes. For example, reviews of existing quality appraisal standards have been used to propose universal standards for appraising the quality of primary qualitative research [ 12 ] or evaluating qualitative research reports [ 13 ]. A second objective is to survey the methods-relevant sections of empirical research reports to establish current practices on methods use and reporting practices, which Moher and colleagues [ 14 ] recommend as a means for establishing the needs to be addressed in reporting guidelines (see, for example [ 15 , 16 ]). A third objective for a methods review is to offer clarity and enhance collective understanding regarding a specific methods topic that may be characterized by ambiguity, inconsistency, or a lack of comprehensiveness within the available methods literature. An example of this is a overview whose objective was to review the inconsistent definitions of intention-to-treat analysis (the methodologically preferred approach to analyze randomized controlled trial data) that have been offered in the methods literature and propose a solution for improving conceptual clarity [ 17 ]. Such reviews are warranted because students and researchers who must learn or apply research methods typically lack the time to systematically search, retrieve, review, and compare the available literature to develop a thorough and critical sense of the varied approaches regarding certain controversial or ambiguous methods topics.

While systematic methods overviews , as a review type, include both reviews of the methods literature and reviews of methods-relevant sections from empirical study reports, the guidance provided here is primarily applicable to reviews of the methods literature since it was derived from the experience of conducting such a review [ 18 ], described below. To our knowledge, there are no well-developed proposals on how to rigorously conduct such reviews. Such guidance would have the potential to improve the thoroughness and credibility of critical evaluations of the methods literature, which could increase their utility as a tool for generating understandings that advance research methods, both qualitative and quantitative. Our aim in this paper is thus to initiate discussion about what might constitute a rigorous approach to systematic methods overviews. While we hope to promote rigor in the conduct of systematic methods overviews wherever possible, we do not wish to suggest that all methods overviews need be conducted to the same standard. Rather, we believe that the level of rigor may need to be tailored pragmatically to the specific review objectives, which may not always justify the resource requirements of an intensive review process.

The example systematic methods overview on sampling in qualitative research

The principles and strategies we propose in this paper are derived from experience conducting a systematic methods overview on the topic of sampling in qualitative research [ 18 ]. The main objective of that methods overview was to bring clarity and deeper understanding of the prominent concepts related to sampling in qualitative research (purposeful sampling strategies, saturation, etc.). Specifically, we interpreted the available guidance, commenting on areas lacking clarity, consistency, or comprehensiveness (without proposing any recommendations on how to do sampling). This was achieved by a comparative and critical analysis of publications representing the most influential (i.e., highly cited) guidance across several methodological traditions in qualitative research.

The specific methods and procedures for the overview on sampling [ 18 ] from which our proposals are derived were developed both after soliciting initial input from local experts in qualitative research and an expert health librarian (KAM) and through ongoing careful deliberation throughout the review process. To summarize, in that review, we employed a transparent and rigorous approach to search the methods literature, selected publications for inclusion according to a purposeful and iterative process, abstracted textual data using structured abstraction forms, and analyzed (synthesized) the data using a systematic multi-step approach featuring abstraction of text, summary of information in matrices, and analytic comparisons.

For this article, we reflected on both the problems and challenges encountered at different stages of the review and our means for selecting justifiable procedures to deal with them. Several principles were then derived by considering the generic nature of these problems, while the generalizable aspects of the procedures used to address them formed the basis of optional strategies. Further details of the specific methods and procedures used in the overview on qualitative sampling are provided below to illustrate both the types of objectives and challenges that reviewers will likely need to consider and our approach to implementing each of the principles and strategies.

Organization of the guidance into principles and strategies

For the purposes of this article, principles are general statements outlining what we propose are important aims or considerations within a particular review process, given the unique objectives or challenges to be overcome with this type of review. These statements follow the general format, “considering the objective or challenge of X, we propose Y to be an important aim or consideration.” Strategies are optional and flexible approaches for implementing the previous principle outlined. Thus, generic challenges give rise to principles, which in turn give rise to strategies.

We organize the principles and strategies below into three sections corresponding to processes characteristic of most systematic literature synthesis approaches: literature identification and selection ; data abstraction from the publications selected for inclusion; and analysis , including critical appraisal and synthesis of the abstracted data. Within each section, we also describe the specific methodological decisions and procedures used in the overview on sampling in qualitative research [ 18 ] to illustrate how the principles and strategies for each review process were applied and implemented in a specific case. We expect this guidance and accompanying illustrations will be useful for anyone considering engaging in a methods overview, particularly those who may be familiar with conventional systematic review methods but may not yet appreciate some of the challenges specific to reviewing the methods literature.

Results and discussion

Literature identification and selection.

The identification and selection process includes search and retrieval of publications and the development and application of inclusion and exclusion criteria to select the publications that will be abstracted and analyzed in the final review. Literature identification and selection for overviews of the methods literature is challenging and potentially more resource-intensive than for most reviews of empirical research. This is true for several reasons that we describe below, alongside discussion of the potential solutions. Additionally, we suggest in this section how the selection procedures can be chosen to match the specific analytic approach used in methods overviews.

Delimiting a manageable set of publications

One aspect of methods overviews that can make identification and selection challenging is the fact that the universe of literature containing potentially relevant information regarding most methods-related topics is expansive and often unmanageably so. Reviewers are faced with two large categories of literature: the methods literature , where the possible publication types include journal articles, books, and book chapters; and the methods-relevant sections of empirical study reports , where the possible publication types include journal articles, monographs, books, theses, and conference proceedings. In our systematic overview of sampling in qualitative research, exhaustively searching (including retrieval and first-pass screening) all publication types across both categories of literature for information on a single methods-related topic was too burdensome to be feasible. The following proposed principle follows from the need to delimit a manageable set of literature for the review.

Principle #1:

Considering the broad universe of potentially relevant literature, we propose that an important objective early in the identification and selection stage is to delimit a manageable set of methods-relevant publications in accordance with the objectives of the methods overview.

Strategy #1:

To limit the set of methods-relevant publications that must be managed in the selection process, reviewers have the option to initially review only the methods literature, and exclude the methods-relevant sections of empirical study reports, provided this aligns with the review’s particular objectives.

We propose that reviewers are justified in choosing to select only the methods literature when the objective is to map out the range of recognized concepts relevant to a methods topic, to summarize the most authoritative or influential definitions or meanings for methods-related concepts, or to demonstrate a problematic lack of clarity regarding a widely established methods-related concept and potentially make recommendations for a preferred approach to the methods topic in question. For example, in the case of the methods overview on sampling [ 18 ], the primary aim was to define areas lacking in clarity for multiple widely established sampling-related topics. In the review on intention-to-treat in the context of missing outcome data [ 17 ], the authors identified a lack of clarity based on multiple inconsistent definitions in the literature and went on to recommend separating the issue of how to handle missing outcome data from the issue of whether an intention-to-treat analysis can be claimed.

In contrast to strategy #1, it may be appropriate to select the methods-relevant sections of empirical study reports when the objective is to illustrate how a methods concept is operationalized in research practice or reported by authors. For example, one could review all the publications in 2 years’ worth of issues of five high-impact field-related journals to answer questions about how researchers describe implementing a particular method or approach, or to quantify how consistently they define or report using it. Such reviews are often used to highlight gaps in the reporting practices regarding specific methods, which may be used to justify items to address in reporting guidelines (for example, [ 14 – 16 ]).

It is worth recognizing that other authors have advocated broader positions regarding the scope of literature to be considered in a review, expanding on our perspective. Suri [ 10 ] (who, like us, emphasizes how different sampling strategies are suitable for different literature synthesis objectives) has, for example, described a two-stage literature sampling procedure (pp. 96–97). First, reviewers use an initial approach to conduct a broad overview of the field—for reviews of methods topics, this would entail an initial review of the research methods literature. This is followed by a second more focused stage in which practical examples are purposefully selected—for methods reviews, this would involve sampling the empirical literature to illustrate key themes and variations. While this approach is seductive in its capacity to generate more in depth and interpretive analytic findings, some reviewers may consider it too resource-intensive to include the second step no matter how selective the purposeful sampling. In the overview on sampling where we stopped after the first stage [ 18 ], we discussed our selective focus on the methods literature as a limitation that left opportunities for further analysis of the literature. We explicitly recommended, for example, that theoretical sampling was a topic for which a future review of the methods sections of empirical reports was justified to answer specific questions identified in the primary review.

Ultimately, reviewers must make pragmatic decisions that balance resource considerations, combined with informed predictions about the depth and complexity of literature available on their topic, with the stated objectives of their review. The remaining principles and strategies apply primarily to overviews that include the methods literature, although some aspects may be relevant to reviews that include empirical study reports.

Searching beyond standard bibliographic databases

An important reality affecting identification and selection in overviews of the methods literature is the increased likelihood for relevant publications to be located in sources other than journal articles (which is usually not the case for overviews of empirical research, where journal articles generally represent the primary publication type). In the overview on sampling [ 18 ], out of 41 full-text publications retrieved and reviewed, only 4 were journal articles, while 37 were books or book chapters. Since many books and book chapters did not exist electronically, their full text had to be physically retrieved in hardcopy, while 11 publications were retrievable only through interlibrary loan or purchase request. The tasks associated with such retrieval are substantially more time-consuming than electronic retrieval. Since a substantial proportion of methods-related guidance may be located in publication types that are less comprehensively indexed in standard bibliographic databases, identification and retrieval thus become complicated processes.

Principle #2:

Considering that important sources of methods guidance can be located in non-journal publication types (e.g., books, book chapters) that tend to be poorly indexed in standard bibliographic databases, it is important to consider alternative search methods for identifying relevant publications to be further screened for inclusion.

Strategy #2:

To identify books, book chapters, and other non-journal publication types not thoroughly indexed in standard bibliographic databases, reviewers may choose to consult one or more of the following less standard sources: Google Scholar, publisher web sites, or expert opinion.

In the case of the overview on sampling in qualitative research [ 18 ], Google Scholar had two advantages over other standard bibliographic databases: it indexes and returns records of books and book chapters likely to contain guidance on qualitative research methods topics; and it has been validated as providing higher citation counts than ISI Web of Science (a producer of numerous bibliographic databases accessible through institutional subscription) for several non-biomedical disciplines including the social sciences where qualitative research methods are prominently used [ 19 – 21 ]. While we identified numerous useful publications by consulting experts, the author publication lists generated through Google Scholar searches were uniquely useful to identify more recent editions of methods books identified by experts.

Searching without relevant metadata

Determining what publications to select for inclusion in the overview on sampling [ 18 ] could only rarely be accomplished by reviewing the publication’s metadata. This was because for the many books and other non-journal type publications we identified as possibly relevant, the potential content of interest would be located in only a subsection of the publication. In this common scenario for reviews of the methods literature (as opposed to methods overviews that include empirical study reports), reviewers will often be unable to employ standard title, abstract, and keyword database searching or screening as a means for selecting publications.

Principle #3:

Considering that the presence of information about the topic of interest may not be indicated in the metadata for books and similar publication types, it is important to consider other means of identifying potentially useful publications for further screening.

Strategy #3:

One approach to identifying potentially useful books and similar publication types is to consider what classes of such publications (e.g., all methods manuals for a certain research approach) are likely to contain relevant content, then identify, retrieve, and review the full text of corresponding publications to determine whether they contain information on the topic of interest.

In the example of the overview on sampling in qualitative research [ 18 ], the topic of interest (sampling) was one of numerous topics covered in the general qualitative research methods manuals. Consequently, examples from this class of publications first had to be identified for retrieval according to non-keyword-dependent criteria. Thus, all methods manuals within the three research traditions reviewed (grounded theory, phenomenology, and case study) that might contain discussion of sampling were sought through Google Scholar and expert opinion, their full text obtained, and hand-searched for relevant content to determine eligibility. We used tables of contents and index sections of books to aid this hand searching.

Purposefully selecting literature on conceptual grounds

A final consideration in methods overviews relates to the type of analysis used to generate the review findings. Unlike quantitative systematic reviews where reviewers aim for accurate or unbiased quantitative estimates—something that requires identifying and selecting the literature exhaustively to obtain all relevant data available (i.e., a complete sample)—in methods overviews, reviewers must describe and interpret the relevant literature in qualitative terms to achieve review objectives. In other words, the aim in methods overviews is to seek coverage of the qualitative concepts relevant to the methods topic at hand. For example, in the overview of sampling in qualitative research [ 18 ], achieving review objectives entailed providing conceptual coverage of eight sampling-related topics that emerged as key domains. The following principle recognizes that literature sampling should therefore support generating qualitative conceptual data as the input to analysis.

Principle #4:

Since the analytic findings of a systematic methods overview are generated through qualitative description and interpretation of the literature on a specified topic, selection of the literature should be guided by a purposeful strategy designed to achieve adequate conceptual coverage (i.e., representing an appropriate degree of variation in relevant ideas) of the topic according to objectives of the review.

Strategy #4:

One strategy for choosing the purposeful approach to use in selecting the literature according to the review objectives is to consider whether those objectives imply exploring concepts either at a broad overview level, in which case combining maximum variation selection with a strategy that limits yield (e.g., critical case, politically important, or sampling for influence—described below) may be appropriate; or in depth, in which case purposeful approaches aimed at revealing innovative cases will likely be necessary.

In the methods overview on sampling, the implied scope was broad since we set out to review publications on sampling across three divergent qualitative research traditions—grounded theory, phenomenology, and case study—to facilitate making informative conceptual comparisons. Such an approach would be analogous to maximum variation sampling.

At the same time, the purpose of that review was to critically interrogate the clarity, consistency, and comprehensiveness of literature from these traditions that was “most likely to have widely influenced students’ and researchers’ ideas about sampling” (p. 1774) [ 18 ]. In other words, we explicitly set out to review and critique the most established and influential (and therefore dominant) literature, since this represents a common basis of knowledge among students and researchers seeking understanding or practical guidance on sampling in qualitative research. To achieve this objective, we purposefully sampled publications according to the criterion of influence , which we operationalized as how often an author or publication has been referenced in print or informal discourse. This second sampling approach also limited the literature we needed to consider within our broad scope review to a manageable amount.

To operationalize this strategy of sampling for influence , we sought to identify both the most influential authors within a qualitative research tradition (all of whose citations were subsequently screened) and the most influential publications on the topic of interest by non-influential authors. This involved a flexible approach that combined multiple indicators of influence to avoid the dilemma that any single indicator might provide inadequate coverage. These indicators included bibliometric data (h-index for author influence [ 22 ]; number of cites for publication influence), expert opinion, and cross-references in the literature (i.e., snowball sampling). As a final selection criterion, a publication was included only if it made an original contribution in terms of novel guidance regarding sampling or a related concept; thus, purely secondary sources were excluded. Publish or Perish software (Anne-Wil Harzing; available at http://www.harzing.com/resources/publish-or-perish ) was used to generate bibliometric data via the Google Scholar database. Figure  1 illustrates how identification and selection in the methods overview on sampling was a multi-faceted and iterative process. The authors selected as influential, and the publications selected for inclusion or exclusion are listed in Additional file 1 (Matrices 1, 2a, 2b).

Literature identification and selection process used in the methods overview on sampling [ 18 ]

In summary, the strategies of seeking maximum variation and sampling for influence were employed in the sampling overview to meet the specific review objectives described. Reviewers will need to consider the full range of purposeful literature sampling approaches at their disposal in deciding what best matches the specific aims of their own reviews. Suri [ 10 ] has recently retooled Patton’s well-known typology of purposeful sampling strategies (originally intended for primary research) for application to literature synthesis, providing a useful resource in this respect.

Data abstraction

The purpose of data abstraction in rigorous literature reviews is to locate and record all data relevant to the topic of interest from the full text of included publications, making them available for subsequent analysis. Conventionally, a data abstraction form—consisting of numerous distinct conceptually defined fields to which corresponding information from the source publication is recorded—is developed and employed. There are several challenges, however, to the processes of developing the abstraction form and abstracting the data itself when conducting methods overviews, which we address here. Some of these problems and their solutions may be familiar to those who have conducted qualitative literature syntheses, which are similarly conceptual.

Iteratively defining conceptual information to abstract

In the overview on sampling [ 18 ], while we surveyed multiple sources beforehand to develop a list of concepts relevant for abstraction (e.g., purposeful sampling strategies, saturation, sample size), there was no way for us to anticipate some concepts prior to encountering them in the review process. Indeed, in many cases, reviewers are unable to determine the complete set of methods-related concepts that will be the focus of the final review a priori without having systematically reviewed the publications to be included. Thus, defining what information to abstract beforehand may not be feasible.

Principle #5:

Considering the potential impracticality of defining a complete set of relevant methods-related concepts from a body of literature one has not yet systematically read, selecting and defining fields for data abstraction must often be undertaken iteratively. Thus, concepts to be abstracted can be expected to grow and change as data abstraction proceeds.

Strategy #5:

Reviewers can develop an initial form or set of concepts for abstraction purposes according to standard methods (e.g., incorporating expert feedback, pilot testing) and remain attentive to the need to iteratively revise it as concepts are added or modified during the review. Reviewers should document revisions and return to re-abstract data from previously abstracted publications as the new data requirements are determined.

In the sampling overview [ 18 ], we developed and maintained the abstraction form in Microsoft Word. We derived the initial set of abstraction fields from our own knowledge of relevant sampling-related concepts, consultation with local experts, and reviewing a pilot sample of publications. Since the publications in this review included a large proportion of books, the abstraction process often began by flagging the broad sections within a publication containing topic-relevant information for detailed review to identify text to abstract. When reviewing flagged text, the reviewer occasionally encountered an unanticipated concept significant enough to warrant being added as a new field to the abstraction form. For example, a field was added to capture how authors described the timing of sampling decisions, whether before (a priori) or after (ongoing) starting data collection, or whether this was unclear. In these cases, we systematically documented the modification to the form and returned to previously abstracted publications to abstract any information that might be relevant to the new field.

The logic of this strategy is analogous to the logic used in a form of research synthesis called best fit framework synthesis (BFFS) [ 23 – 25 ]. In that method, reviewers initially code evidence using an a priori framework they have selected. When evidence cannot be accommodated by the selected framework, reviewers then develop new themes or concepts from which they construct a new expanded framework. Both the strategy proposed and the BFFS approach to research synthesis are notable for their rigorous and transparent means to adapt a final set of concepts to the content under review.

Accounting for inconsistent terminology

An important complication affecting the abstraction process in methods overviews is that the language used by authors to describe methods-related concepts can easily vary across publications. For example, authors from different qualitative research traditions often use different terms for similar methods-related concepts. Furthermore, as we found in the sampling overview [ 18 ], there may be cases where no identifiable term, phrase, or label for a methods-related concept is used at all, and a description of it is given instead. This can make searching the text for relevant concepts based on keywords unreliable.

Principle #6:

Since accepted terms may not be used consistently to refer to methods concepts, it is necessary to rely on the definitions for concepts, rather than keywords, to identify relevant information in the publication to abstract.

Strategy #6:

An effective means to systematically identify relevant information is to develop and iteratively adjust written definitions for key concepts (corresponding to abstraction fields) that are consistent with and as inclusive of as much of the literature reviewed as possible. Reviewers then seek information that matches these definitions (rather than keywords) when scanning a publication for relevant data to abstract.

In the abstraction process for the sampling overview [ 18 ], we noted the several concepts of interest to the review for which abstraction by keyword was particularly problematic due to inconsistent terminology across publications: sampling , purposeful sampling , sampling strategy , and saturation (for examples, see Additional file 1 , Matrices 3a, 3b, 4). We iteratively developed definitions for these concepts by abstracting text from publications that either provided an explicit definition or from which an implicit definition could be derived, which was recorded in fields dedicated to the concept’s definition. Using a method of constant comparison, we used text from definition fields to inform and modify a centrally maintained definition of the corresponding concept to optimize its fit and inclusiveness with the literature reviewed. Table  1 shows, as an example, the final definition constructed in this way for one of the central concepts of the review, qualitative sampling .

We applied iteratively developed definitions when making decisions about what specific text to abstract for an existing field, which allowed us to abstract concept-relevant data even if no recognized keyword was used. For example, this was the case for the sampling-related concept, saturation , where the relevant text available for abstraction in one publication [ 26 ]—“to continue to collect data until nothing new was being observed or recorded, no matter how long that takes”—was not accompanied by any term or label whatsoever.

This comparative analytic strategy (and our approach to analysis more broadly as described in strategy #7, below) is analogous to the process of reciprocal translation —a technique first introduced for meta-ethnography by Noblit and Hare [ 27 ] that has since been recognized as a common element in a variety of qualitative metasynthesis approaches [ 28 ]. Reciprocal translation, taken broadly, involves making sense of a study’s findings in terms of the findings of the other studies included in the review. In practice, it has been operationalized in different ways. Melendez-Torres and colleagues developed a typology from their review of the metasynthesis literature, describing four overlapping categories of specific operations undertaken in reciprocal translation: visual representation, key paper integration, data reduction and thematic extraction, and line-by-line coding [ 28 ]. The approaches suggested in both strategies #6 and #7, with their emphasis on constant comparison, appear to fall within the line-by-line coding category.

Generating credible and verifiable analytic interpretations

The analysis in a systematic methods overview must support its more general objective, which we suggested above is often to offer clarity and enhance collective understanding regarding a chosen methods topic. In our experience, this involves describing and interpreting the relevant literature in qualitative terms. Furthermore, any interpretative analysis required may entail reaching different levels of abstraction, depending on the more specific objectives of the review. For example, in the overview on sampling [ 18 ], we aimed to produce a comparative analysis of how multiple sampling-related topics were treated differently within and among different qualitative research traditions. To promote credibility of the review, however, not only should one seek a qualitative analytic approach that facilitates reaching varying levels of abstraction but that approach must also ensure that abstract interpretations are supported and justified by the source data and not solely the product of the analyst’s speculative thinking.

Principle #7:

Considering the qualitative nature of the analysis required in systematic methods overviews, it is important to select an analytic method whose interpretations can be verified as being consistent with the literature selected, regardless of the level of abstraction reached.

Strategy #7:

We suggest employing the constant comparative method of analysis [ 29 ] because it supports developing and verifying analytic links to the source data throughout progressively interpretive or abstract levels. In applying this approach, we advise a rigorous approach, documenting how supportive quotes or references to the original texts are carried forward in the successive steps of analysis to allow for easy verification.

The analytic approach used in the methods overview on sampling [ 18 ] comprised four explicit steps, progressing in level of abstraction—data abstraction, matrices, narrative summaries, and final analytic conclusions (Fig.  2 ). While we have positioned data abstraction as the second stage of the generic review process (prior to Analysis), above, we also considered it as an initial step of analysis in the sampling overview for several reasons. First, it involved a process of constant comparisons and iterative decision-making about the fields to add or define during development and modification of the abstraction form, through which we established the range of concepts to be addressed in the review. At the same time, abstraction involved continuous analytic decisions about what textual quotes (ranging in size from short phrases to numerous paragraphs) to record in the fields thus created. This constant comparative process was analogous to open coding in which textual data from publications was compared to conceptual fields (equivalent to codes) or to other instances of data previously abstracted when constructing definitions to optimize their fit with the overall literature as described in strategy #6. Finally, in the data abstraction step, we also recorded our first interpretive thoughts in dedicated fields, providing initial material for the more abstract analytic steps.

Summary of progressive steps of analysis used in the methods overview on sampling [ 18 ]

In the second step of the analysis, we constructed topic-specific matrices , or tables, by copying relevant quotes from abstraction forms into the appropriate cells of matrices (for the complete set of analytic matrices developed in the sampling review, see Additional file 1 (matrices 3 to 10)). Each matrix ranged from one to five pages; row headings, nested three-deep, identified the methodological tradition, author, and publication, respectively; and column headings identified the concepts, which corresponded to abstraction fields. Matrices thus allowed us to make further comparisons across methodological traditions, and between authors within a tradition. In the third step of analysis, we recorded our comparative observations as narrative summaries , in which we used illustrative quotes more sparingly. In the final step, we developed analytic conclusions based on the narrative summaries about the sampling-related concepts within each methodological tradition for which clarity, consistency, or comprehensiveness of the available guidance appeared to be lacking. Higher levels of analysis thus built logically from the lower levels, enabling us to easily verify analytic conclusions by tracing the support for claims by comparing the original text of publications reviewed.

Integrative versus interpretive methods overviews

The analytic product of systematic methods overviews is comparable to qualitative evidence syntheses, since both involve describing and interpreting the relevant literature in qualitative terms. Most qualitative synthesis approaches strive to produce new conceptual understandings that vary in level of interpretation. Dixon-Woods and colleagues [ 30 ] elaborate on a useful distinction, originating from Noblit and Hare [ 27 ], between integrative and interpretive reviews. Integrative reviews focus on summarizing available primary data and involve using largely secure and well defined concepts to do so; definitions are used from an early stage to specify categories for abstraction (or coding) of data, which in turn supports their aggregation; they do not seek as their primary focus to develop or specify new concepts, although they may achieve some theoretical or interpretive functions. For interpretive reviews, meanwhile, the main focus is to develop new concepts and theories that integrate them, with the implication that the concepts developed become fully defined towards the end of the analysis. These two forms are not completely distinct, and “every integrative synthesis will include elements of interpretation, and every interpretive synthesis will include elements of aggregation of data” [ 30 ].

The example methods overview on sampling [ 18 ] could be classified as predominantly integrative because its primary goal was to aggregate influential authors’ ideas on sampling-related concepts; there were also, however, elements of interpretive synthesis since it aimed to develop new ideas about where clarity in guidance on certain sampling-related topics is lacking, and definitions for some concepts were flexible and not fixed until late in the review. We suggest that most systematic methods overviews will be classifiable as predominantly integrative (aggregative). Nevertheless, more highly interpretive methods overviews are also quite possible—for example, when the review objective is to provide a highly critical analysis for the purpose of generating new methodological guidance. In such cases, reviewers may need to sample more deeply (see strategy #4), specifically by selecting empirical research reports (i.e., to go beyond dominant or influential ideas in the methods literature) that are likely to feature innovations or instructive lessons in employing a given method.

In this paper, we have outlined tentative guidance in the form of seven principles and strategies on how to conduct systematic methods overviews, a review type in which methods-relevant literature is systematically analyzed with the aim of offering clarity and enhancing collective understanding regarding a specific methods topic. Our proposals include strategies for delimiting the set of publications to consider, searching beyond standard bibliographic databases, searching without the availability of relevant metadata, selecting publications on purposeful conceptual grounds, defining concepts and other information to abstract iteratively, accounting for inconsistent terminology, and generating credible and verifiable analytic interpretations. We hope the suggestions proposed will be useful to others undertaking reviews on methods topics in future.

As far as we are aware, this is the first published source of concrete guidance for conducting this type of review. It is important to note that our primary objective was to initiate methodological discussion by stimulating reflection on what rigorous methods for this type of review should look like, leaving the development of more complete guidance to future work. While derived from the experience of reviewing a single qualitative methods topic, we believe the principles and strategies provided are generalizable to overviews of both qualitative and quantitative methods topics alike. However, it is expected that additional challenges and insights for conducting such reviews have yet to be defined. Thus, we propose that next steps for developing more definitive guidance should involve an attempt to collect and integrate other reviewers’ perspectives and experiences in conducting systematic methods overviews on a broad range of qualitative and quantitative methods topics. Formalized guidance and standards would improve the quality of future methods overviews, something we believe has important implications for advancing qualitative and quantitative methodology. When undertaken to a high standard, rigorous critical evaluations of the available methods guidance have significant potential to make implicit controversies explicit, and improve the clarity and precision of our understandings of problematic qualitative or quantitative methods issues.

A review process central to most types of rigorous reviews of empirical studies, which we did not explicitly address in a separate review step above, is quality appraisal . The reason we have not treated this as a separate step stems from the different objectives of the primary publications included in overviews of the methods literature (i.e., providing methodological guidance) compared to the primary publications included in the other established review types (i.e., reporting findings from single empirical studies). This is not to say that appraising quality of the methods literature is not an important concern for systematic methods overviews. Rather, appraisal is much more integral to (and difficult to separate from) the analysis step, in which we advocate appraising clarity, consistency, and comprehensiveness—the quality appraisal criteria that we suggest are appropriate for the methods literature. As a second important difference regarding appraisal, we currently advocate appraising the aforementioned aspects at the level of the literature in aggregate rather than at the level of individual publications. One reason for this is that methods guidance from individual publications generally builds on previous literature, and thus we feel that ahistorical judgments about comprehensiveness of single publications lack relevance and utility. Additionally, while different methods authors may express themselves less clearly than others, their guidance can nonetheless be highly influential and useful, and should therefore not be downgraded or ignored based on considerations of clarity—which raises questions about the alternative uses that quality appraisals of individual publications might have. Finally, legitimate variability in the perspectives that methods authors wish to emphasize, and the levels of generality at which they write about methods, makes critiquing individual publications based on the criterion of clarity a complex and potentially problematic endeavor that is beyond the scope of this paper to address. By appraising the current state of the literature at a holistic level, reviewers stand to identify important gaps in understanding that represent valuable opportunities for further methodological development.

To summarize, the principles and strategies provided here may be useful to those seeking to undertake their own systematic methods overview. Additional work is needed, however, to establish guidance that is comprehensive by comparing the experiences from conducting a variety of methods overviews on a range of methods topics. Efforts that further advance standards for systematic methods overviews have the potential to promote high-quality critical evaluations that produce conceptually clear and unified understandings of problematic methods topics, thereby accelerating the advance of research methodology.

Hutton JL, Ashcroft R. What does “systematic” mean for reviews of methods? In: Black N, Brazier J, Fitzpatrick R, Reeves B, editors. Health services research methods: a guide to best practice. London: BMJ Publishing Group; 1998. p. 249–54.

Google Scholar  

Cochrane handbook for systematic reviews of interventions. In. Edited by Higgins JPT, Green S, Version 5.1.0 edn: The Cochrane Collaboration; 2011.

Centre for Reviews and Dissemination: Systematic reviews: CRD’s guidance for undertaking reviews in health care . York: Centre for Reviews and Dissemination; 2009.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JPA, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700–0.

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9(1):59.

Article   PubMed   PubMed Central   Google Scholar  

Kastner M, Tricco AC, Soobiah C, Lillie E, Perrier L, Horsley T, Welch V, Cogo E, Antony J, Straus SE. What is the most appropriate knowledge synthesis method to conduct a review? Protocol for a scoping review. BMC Med Res Methodol. 2012;12(1):1–1.

Article   Google Scholar  

Booth A, Noyes J, Flemming K, Gerhardus A. Guidance on choosing qualitative evidence synthesis methods for use in health technology assessments of complex interventions. In: Integrate-HTA. 2016.

Booth A, Sutton A, Papaioannou D. Systematic approaches to successful literature review. 2nd ed. London: Sage; 2016.

Hannes K, Lockwood C. Synthesizing qualitative research: choosing the right approach. Chichester: Wiley-Blackwell; 2012.

Suri H. Towards methodologically inclusive research syntheses: expanding possibilities. New York: Routledge; 2014.

Campbell M, Egan M, Lorenc T, Bond L, Popham F, Fenton C, Benzeval M. Considering methodological options for reviews of theory: illustrated by a review of theories linking income and health. Syst Rev. 2014;3(1):1–11.

Cohen DJ, Crabtree BF. Evaluative criteria for qualitative research in health care: controversies and recommendations. Ann Fam Med. 2008;6(4):331–9.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reportingqualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

Article   PubMed   Google Scholar  

Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2):e1000217.

Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78.

Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005;365(9465):1159–62.

Alshurafa M, Briel M, Akl EA, Haines T, Moayyedi P, Gentles SJ, Rios L, Tran C, Bhatnagar N, Lamontagne F, et al. Inconsistent definitions for intention-to-treat in relation to missing outcome data: systematic review of the methods literature. PLoS One. 2012;7(11):e49163.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Gentles SJ, Charles C, Ploeg J, McKibbon KA. Sampling in qualitative research: insights from an overview of the methods literature. Qual Rep. 2015;20(11):1772–89.

Harzing A-W, Alakangas S. Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison. Scientometrics. 2016;106(2):787–804.

Harzing A-WK, van der Wal R. Google Scholar as a new source for citation analysis. Ethics Sci Environ Polit. 2008;8(1):61–73.

Kousha K, Thelwall M. Google Scholar citations and Google Web/URL citations: a multi‐discipline exploratory analysis. J Assoc Inf Sci Technol. 2007;58(7):1055–65.

Hirsch JE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A. 2005;102(46):16569–72.

Booth A, Carroll C. How to build up the actionable knowledge base: the role of ‘best fit’ framework synthesis for studies of improvement in healthcare. BMJ Quality Safety. 2015;24(11):700–8.

Carroll C, Booth A, Leaviss J, Rick J. “Best fit” framework synthesis: refining the method. BMC Med Res Methodol. 2013;13(1):37.

Carroll C, Booth A, Cooper K. A worked example of “best fit” framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. 2011;11(1):29.

Cohen MZ, Kahn DL, Steeves DL. Hermeneutic phenomenological research: a practical guide for nurse researchers. Thousand Oaks: Sage; 2000.

Noblit GW, Hare RD. Meta-ethnography: synthesizing qualitative studies. Newbury Park: Sage; 1988.

Book   Google Scholar  

Melendez-Torres GJ, Grant S, Bonell C. A systematic review and critical appraisal of qualitative metasynthetic practice in public health to develop a taxonomy of operations of reciprocal translation. Res Synthesis Methods. 2015;6(4):357–71.

Article   CAS   Google Scholar  

Glaser BG, Strauss A. The discovery of grounded theory. Chicago: Aldine; 1967.

Dixon-Woods M, Agarwal S, Young B, Jones D, Sutton A. Integrative approaches to qualitative and quantitative evidence. In: UK National Health Service. 2004. p. 1–44.

Download references

Acknowledgements

Not applicable.

There was no funding for this work.

Availability of data and materials

The systematic methods overview used as a worked example in this article (Gentles SJ, Charles C, Ploeg J, McKibbon KA: Sampling in qualitative research: insights from an overview of the methods literature. The Qual Rep 2015, 20(11):1772-1789) is available from http://nsuworks.nova.edu/tqr/vol20/iss11/5 .

Authors’ contributions

SJG wrote the first draft of this article, with CC contributing to drafting. All authors contributed to revising the manuscript. All authors except CC (deceased) approved the final draft. SJG, CC, KAB, and JP were involved in developing methods for the systematic methods overview on sampling.

Authors’ information

Competing interests.

The authors declare that they have no competing interests.

Consent for publication

Ethics approval and consent to participate, author information, authors and affiliations.

Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada

Stephen J. Gentles, Cathy Charles & K. Ann McKibbon

Faculty of Social Work, University of Calgary, Alberta, Canada

David B. Nicholas

School of Nursing, McMaster University, Hamilton, Ontario, Canada

Jenny Ploeg

CanChild Centre for Childhood Disability Research, McMaster University, 1400 Main Street West, IAHS 408, Hamilton, ON, L8S 1C7, Canada

Stephen J. Gentles

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Stephen J. Gentles .

Additional information

Cathy Charles is deceased

Additional file

Additional file 1:.

Submitted: Analysis_matrices. (DOC 330 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Gentles, S.J., Charles, C., Nicholas, D.B. et al. Reviewing the research methods literature: principles and strategies illustrated by a systematic overview of sampling in qualitative research. Syst Rev 5 , 172 (2016). https://doi.org/10.1186/s13643-016-0343-0

Download citation

Received : 06 June 2016

Accepted : 14 September 2016

Published : 11 October 2016

DOI : https://doi.org/10.1186/s13643-016-0343-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic review
• Literature selection
• Research methods
• Research methodology
• Overview of methods
• Systematic methods overview
• Review methods

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jan 4, 2024 10:52 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

Research Methods: Literature Reviews

• Annotated Bibliographies
• Literature Reviews
• Scoping Reviews
• Systematic Reviews
• Scholarship of Teaching and Learning
• Persuasive Arguments
• Subject Specific Methodology

A literature review involves researching, reading, analyzing, evaluating, and summarizing scholarly literature (typically journals and articles) about a specific topic. The results of a literature review may be an entire report or article OR may be part of a article, thesis, dissertation, or grant proposal. A literature review helps the author learn about the history and nature of their topic, and identify research gaps and problems.

Steps & Elements

Problem formulation

• Determine your topic and its components by asking a question
• Research: locate literature related to your topic to identify the gap(s) that can be addressed
• Read: read the articles or other sources of information
• Analyze: assess the findings for relevancy
• Evaluating: determine how the article are relevant to your research and what are the key findings
• Synthesis: write about the key findings and how it is relevant to your research

Elements of a Literature Review

• Summarize subject, issue or theory under consideration, along with objectives of the review
• Divide works under review into categories (e.g. those in support of a particular position, those against, those offering alternative theories entirely)
• Explain how each work is similar to and how it varies from the others
• Conclude which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of an area of research

Writing a Literature Review Resources

• How to Write a Literature Review From the Wesleyan University Library
• Write a Literature Review From the University of California Santa Cruz Library. A Brief overview of a literature review, includes a list of stages for writing a lit review.
• Literature Reviews From the University of North Carolina Writing Center. Detailed information about writing a literature review.
• Undertaking a literature review: a step-by-step approach Cronin, P., Ryan, F., & Coughan, M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), p.38-43

Literature Review Tutorial

• << Previous: Annotated Bibliographies
• Next: Scoping Reviews >>
• Last Updated: Feb 29, 2024 12:00 PM
• URL: https://guides.auraria.edu/researchmethods

1100 Lawrence Street Denver, CO 80204 303-315-7700 Ask Us Directions

Covidence website will be inaccessible as we upgrading our platform on Monday 23rd August at 10am AEST, / 2am CEST/1am BST (Sunday, 15th August 8pm EDT/5pm PDT) 

How to write the methods section of a systematic review

Home | Blog | How To | How to write the methods section of a systematic review

Covidence breaks down how to write a methods section

The methods section of your systematic review describes what you did, how you did it, and why. Readers need this information to interpret the results and conclusions of the review. Often, a lot of information needs to be distilled into just a few paragraphs. This can be a challenging task, but good preparation and the right tools will help you to set off in the right direction 🗺️🧭.

Systematic reviews are so-called because they are conducted in a way that is rigorous and replicable. So it’s important that these methods are reported in a way that is thorough, clear, and easy to navigate for the reader – whether that’s a patient, a healthcare worker, or a researcher. 

Like most things in a systematic review, the methods should be planned upfront and ideally described in detail in a project plan or protocol. Reviews of healthcare interventions follow the PRISMA guidelines for the minimum set of items to report in the methods section. But what else should be included? It’s a good idea to consider what readers will want to know about the review methods and whether the journal you’re planning to submit the work to has expectations on the reporting of methods. Finding out in advance will help you to plan what to include.

Describe what happened

While the research plan sets out what you intend to do, the methods section is a write-up of what actually happened. It’s not a simple case of rewriting the plan in the past tense – you will also need to discuss and justify deviations from the plan and describe the handling of issues that were unforeseen at the time the plan was written. For this reason, it is useful to make detailed notes before, during, and after the review is completed. Relying on memory alone risks losing valuable information and trawling through emails when the deadline is looming can be frustrating and time consuming! 

Keep it brief

The methods section should be succinct but include all the noteworthy information. This can be a difficult balance to achieve. A useful strategy is to aim for a brief description that signposts the reader to a separate section or sections of supporting information. This could include datasets, a flowchart to show what happened to the excluded studies, a collection of search strategies, and tables containing detailed information about the studies.This separation keeps the review short and simple while enabling the reader to drill down to the detail as needed. And if the methods follow a well-known or standard process, it might suffice to say so and give a reference, rather than describe the process at length. 

Follow a structure

A clear structure provides focus. Use of descriptive headings keeps the writing on track and helps the reader get to key information quickly. What should the structure of the methods section look like? As always, a lot depends on the type of review but it will certainly contain information relating to the following areas:

• Selection criteria ⭕
• Data collection and analysis 👩‍💻
• Study quality and risk of bias ⚖️

Let’s look at each of these in turn.

1. Selection criteria ⭕

The criteria for including and excluding studies are listed here. This includes detail about the types of studies, the types of participants, the types of interventions and the types of outcomes and how they were measured. 

2. Search 🕵🏾‍♀️

Comprehensive reporting of the search is important because this means it can be evaluated and replicated. The search strategies are included in the review, along with details of the databases searched. It’s also important to list any restrictions on the search (for example, language), describe how resources other than electronic databases were searched (for example,  non-indexed journals), and give the date that the searches were run. The PRISMA-S extension provides guidance on reporting literature searches. 

Systematic reviewer pro-tip:

 Copy and paste the search strategy to avoid introducing typos

3. Data collection and analysis 👩‍💻

This section describes:

• how studies were selected for inclusion in the review
• how study data were extracted from the study reports
• how study data were combined for analysis and synthesis

To describe how studies were selected for inclusion , review teams outline the screening process. Covidence uses reviewers’ decision data to automatically populate a PRISMA flow diagram for this purpose. Covidence can also calculate Cohen’s kappa to enable review teams to report the level of agreement among individual reviewers during screening.

To describe how study data were extracted from the study reports , reviewers outline the form that was used, any pilot-testing that was done, and the items that were extracted from the included studies. An important piece of information to include here is the process used to resolve conflict among the reviewers. Covidence’s data extraction tool saves reviewers’ comments and notes in the system as they work. This keeps the information in one place for easy retrieval ⚡.

To describe how study data were combined for analysis and synthesis, reviewers outline the type of synthesis (narrative or quantitative, for example), the methods for grouping data, the challenges that came up, and how these were dealt with. If the review includes a meta-analysis, it will detail how this was performed and how the treatment effects were measured.

4. Study quality and risk of bias ⚖️

Because the results of systematic reviews can be affected by many types of bias, reviewers make every effort to minimise it and to show the reader that the methods they used were appropriate. This section describes the methods used to assess study quality and an assessment of the risk of bias across a range of domains. 

Steps to assess the risk of bias in studies include looking at how study participants were assigned to treatment groups and whether patients and/or study assessors were blinded to the treatment given. Reviewers also report their assessment of the risk of bias due to missing outcome data, whether that is due to participant drop-out or non-reporting of the outcomes by the study authors.

Covidence’s default template for assessing study quality is Cochrane’s risk of bias tool but it is also possible to start from scratch and build a tool with a set of custom domains if you prefer.

Careful planning, clear writing, and a structured approach are key to a good methods section. A methodologist will be able to refer review teams to examples of good methods reporting in the literature. Covidence helps reviewers to screen references, extract data and complete risk of bias tables quickly and efficiently. Sign up for a free trial today!

Laura Mellor. Portsmouth, UK

Perhaps you'd also like....

Top 5 Tips for High-Quality Systematic Review Data Extraction

Data extraction can be a complex step in the systematic review process. Here are 5 top tips from our experts to help prepare and achieve high quality data extraction.

How to get through study quality assessment Systematic Review

Find out 5 tops tips to conducting quality assessment and why it’s an important step in the systematic review process.

How to extract study data for your systematic review

Learn the basic process and some tips to build data extraction forms for your systematic review with Covidence.

Better systematic review management

Head office, working for an institution or organisation.

Find out why over 350 of the world’s leading institutions are seeing a surge in publications since using Covidence!

Request a consultation with one of our team members and start empowering your researchers:

By using our site you consent to our use of cookies to measure and improve our site’s performance. Please see our Privacy Policy for more information. 

• UWF Libraries

Literature Review: Conducting & Writing

• Sample Literature Reviews
• Steps for Conducting a Lit Review
• Finding "The Literature"
• Organizing/Writing
• APA Style This link opens in a new window
• Chicago: Notes Bibliography This link opens in a new window
• MLA Style This link opens in a new window

Sample Lit Reviews from Communication Arts

Have an exemplary literature review.

• Literature Review Sample 1
• Literature Review Sample 2
• Literature Review Sample 3

Have you written a stellar literature review you care to share for teaching purposes?

Are you an instructor who has received an exemplary literature review and have permission from the student to post?

Please contact Britt McGowan at [email protected] for inclusion in this guide. All disciplines welcome and encouraged.

• << Previous: MLA Style
• Next: Get Help! >>
• Last Updated: Mar 22, 2024 9:37 AM
• URL: https://libguides.uwf.edu/litreview

How to write a literature review introduction (+ examples)

The introduction to a literature review serves as your reader’s guide through your academic work and thought process. Explore the significance of literature review introductions in review papers, academic papers, essays, theses, and dissertations. We delve into the purpose and necessity of these introductions, explore the essential components of literature review introductions, and provide step-by-step guidance on how to craft your own, along with examples.

Why you need an introduction for a literature review

When you need an introduction for a literature review, what to include in a literature review introduction, examples of literature review introductions, steps to write your own literature review introduction.

A literature review is a comprehensive examination of the international academic literature concerning a particular topic. It involves summarizing published works, theories, and concepts while also highlighting gaps and offering critical reflections.

In academic writing , the introduction for a literature review is an indispensable component. Effective academic writing requires proper paragraph structuring to guide your reader through your argumentation. This includes providing an introduction to your literature review.

It is imperative to remember that you should never start sharing your findings abruptly. Even if there isn’t a dedicated introduction section .

Instead, you should always offer some form of introduction to orient the reader and clarify what they can expect.

There are three main scenarios in which you need an introduction for a literature review:

• Academic literature review papers: When your literature review constitutes the entirety of an academic review paper, a more substantial introduction is necessary. This introduction should resemble the standard introduction found in regular academic papers.
• Literature review section in an academic paper or essay: While this section tends to be brief, it’s important to precede the detailed literature review with a few introductory sentences. This helps orient the reader before delving into the literature itself.
• Literature review chapter or section in your thesis/dissertation: Every thesis and dissertation includes a literature review component, which also requires a concise introduction to set the stage for the subsequent review.

You may also like: How to write a fantastic thesis introduction (+15 examples)

It is crucial to customize the content and depth of your literature review introduction according to the specific format of your academic work.

In practical terms, this implies, for instance, that the introduction in an academic literature review paper, especially one derived from a systematic literature review , is quite comprehensive. Particularly compared to the rather brief one or two introductory sentences that are often found at the beginning of a literature review section in a standard academic paper. The introduction to the literature review chapter in a thesis or dissertation again adheres to different standards.

Here’s a structured breakdown based on length and the necessary information:

Academic literature review paper

The introduction of an academic literature review paper, which does not rely on empirical data, often necessitates a more extensive introduction than the brief literature review introductions typically found in empirical papers. It should encompass:

• The research problem: Clearly articulate the problem or question that your literature review aims to address.
• The research gap: Highlight the existing gaps, limitations, or unresolved aspects within the current body of literature related to the research problem.
• The research relevance: Explain why the chosen research problem and its subsequent investigation through a literature review are significant and relevant in your academic field.
• The literature review method: If applicable, describe the methodology employed in your literature review, especially if it is a systematic review or follows a specific research framework.
• The main findings or insights of the literature review: Summarize the key discoveries, insights, or trends that have emerged from your comprehensive review of the literature.
• The main argument of the literature review: Conclude the introduction by outlining the primary argument or statement that your literature review will substantiate, linking it to the research problem and relevance you’ve established.
• Preview of the literature review’s structure: Offer a glimpse into the organization of the literature review paper, acting as a guide for the reader. This overview outlines the subsequent sections of the paper and provides an understanding of what to anticipate.

By addressing these elements, your introduction will provide a clear and structured overview of what readers can expect in your literature review paper.

Regular literature review section in an academic article or essay

Most academic articles or essays incorporate regular literature review sections, often placed after the introduction. These sections serve to establish a scholarly basis for the research or discussion within the paper.

In a standard 8000-word journal article, the literature review section typically spans between 750 and 1250 words. The first few sentences or the first paragraph within this section often serve as an introduction. It should encompass:

• An introduction to the topic: When delving into the academic literature on a specific topic, it’s important to provide a smooth transition that aids the reader in comprehending why certain aspects will be discussed within your literature review.
• The core argument: While literature review sections primarily synthesize the work of other scholars, they should consistently connect to your central argument. This central argument serves as the crux of your message or the key takeaway you want your readers to retain. By positioning it at the outset of the literature review section and systematically substantiating it with evidence, you not only enhance reader comprehension but also elevate overall readability. This primary argument can typically be distilled into 1-2 succinct sentences.

In some cases, you might include:

• Methodology: Details about the methodology used, but only if your literature review employed a specialized method. If your approach involved a broader overview without a systematic methodology, you can omit this section, thereby conserving word count.

By addressing these elements, your introduction will effectively integrate your literature review into the broader context of your academic paper or essay. This will, in turn, assist your reader in seamlessly following your overarching line of argumentation.

Introduction to a literature review chapter in thesis or dissertation

The literature review typically constitutes a distinct chapter within a thesis or dissertation. Often, it is Chapter 2 of a thesis or dissertation.

Some students choose to incorporate a brief introductory section at the beginning of each chapter, including the literature review chapter. Alternatively, others opt to seamlessly integrate the introduction into the initial sentences of the literature review itself. Both approaches are acceptable, provided that you incorporate the following elements:

• Purpose of the literature review and its relevance to the thesis/dissertation research: Explain the broader objectives of the literature review within the context of your research and how it contributes to your thesis or dissertation. Essentially, you’re telling the reader why this literature review is important and how it fits into the larger scope of your academic work.
• Primary argument: Succinctly communicate what you aim to prove, explain, or explore through the review of existing literature. This statement helps guide the reader’s understanding of the review’s purpose and what to expect from it.
• Preview of the literature review’s content: Provide a brief overview of the topics or themes that your literature review will cover. It’s like a roadmap for the reader, outlining the main areas of focus within the review. This preview can help the reader anticipate the structure and organization of your literature review.
• Methodology: If your literature review involved a specific research method, such as a systematic review or meta-analysis, you should briefly describe that methodology. However, this is not always necessary, especially if your literature review is more of a narrative synthesis without a distinct research method.

By addressing these elements, your introduction will empower your literature review to play a pivotal role in your thesis or dissertation research. It will accomplish this by integrating your research into the broader academic literature and providing a solid theoretical foundation for your work.

Comprehending the art of crafting your own literature review introduction becomes significantly more accessible when you have concrete examples to examine. Here, you will find several examples that meet, or in most cases, adhere to the criteria described earlier.

Example 1: An effective introduction for an academic literature review paper

To begin, let’s delve into the introduction of an academic literature review paper. We will examine the paper “How does culture influence innovation? A systematic literature review”, which was published in 2018 in the journal Management Decision.

The entire introduction spans 611 words and is divided into five paragraphs. In this introduction, the authors accomplish the following:

• In the first paragraph, the authors introduce the broader topic of the literature review, which focuses on innovation and its significance in the context of economic competition. They underscore the importance of this topic, highlighting its relevance for both researchers and policymakers.
• In the second paragraph, the authors narrow down their focus to emphasize the specific role of culture in relation to innovation.
• In the third paragraph, the authors identify research gaps, noting that existing studies are often fragmented and disconnected. They then emphasize the value of conducting a systematic literature review to enhance our understanding of the topic.
• In the fourth paragraph, the authors introduce their specific objectives and explain how their insights can benefit other researchers and business practitioners.
• In the fifth and final paragraph, the authors provide an overview of the paper’s organization and structure.

In summary, this introduction stands as a solid example. While the authors deviate from previewing their key findings (which is a common practice at least in the social sciences), they do effectively cover all the other previously mentioned points.

Example 2: An effective introduction to a literature review section in an academic paper

The second example represents a typical academic paper, encompassing not only a literature review section but also empirical data, a case study, and other elements. We will closely examine the introduction to the literature review section in the paper “The environmentalism of the subalterns: a case study of environmental activism in Eastern Kurdistan/Rojhelat”, which was published in 2021 in the journal Local Environment.

The paper begins with a general introduction and then proceeds to the literature review, designated by the authors as their conceptual framework. Of particular interest is the first paragraph of this conceptual framework, comprising 142 words across five sentences:

“ A peripheral and marginalised nationality within a multinational though-Persian dominated Iranian society, the Kurdish people of Iranian Kurdistan (a region referred by the Kurds as Rojhelat/Eastern Kurdi-stan) have since the early twentieth century been subject to multifaceted and systematic discriminatory and exclusionary state policy in Iran. This condition has left a population of 12–15 million Kurds in Iran suffering from structural inequalities, disenfranchisement and deprivation. Mismanagement of Kurdistan’s natural resources and the degradation of its natural environmental are among examples of this disenfranchisement. As asserted by Julian Agyeman (2005), structural inequalities that sustain the domination of political and economic elites often simultaneously result in environmental degradation, injustice and discrimination against subaltern communities. This study argues that the environmental struggle in Eastern Kurdistan can be asserted as a (sub)element of the Kurdish liberation movement in Iran. Conceptually this research is inspired by and has been conducted through the lens of ‘subalternity’ ” ( Hassaniyan, 2021, p. 931 ).

In this first paragraph, the author is doing the following:

• The author contextualises the research
• The author links the research focus to the international literature on structural inequalities
• The author clearly presents the argument of the research
• The author clarifies how the research is inspired by and uses the concept of ‘subalternity’.

Thus, the author successfully introduces the literature review, from which point onward it dives into the main concept (‘subalternity’) of the research, and reviews the literature on socio-economic justice and environmental degradation.

While introductions to a literature review section aren’t always required to offer the same level of study context detail as demonstrated here, this introduction serves as a commendable model for orienting the reader within the literature review. It effectively underscores the literature review’s significance within the context of the study being conducted.

Examples 3-5: Effective introductions to literature review chapters

The introduction to a literature review chapter can vary in length, depending largely on the overall length of the literature review chapter itself. For example, a master’s thesis typically features a more concise literature review, thus necessitating a shorter introduction. In contrast, a Ph.D. thesis, with its more extensive literature review, often includes a more detailed introduction.

Numerous universities offer online repositories where you can access theses and dissertations from previous years, serving as valuable sources of reference. Many of these repositories, however, may require you to log in through your university account. Nevertheless, a few open-access repositories are accessible to anyone, such as the one by the University of Manchester . It’s important to note though that copyright restrictions apply to these resources, just as they would with published papers.

Master’s thesis literature review introduction

The first example is “Benchmarking Asymmetrical Heating Models of Spider Pulsar Companions” by P. Sun, a master’s thesis completed at the University of Manchester on January 9, 2024. The author, P. Sun, introduces the literature review chapter very briefly but effectively:

PhD thesis literature review chapter introduction

The second example is Deep Learning on Semi-Structured Data and its Applications to Video-Game AI, Woof, W. (Author). 31 Dec 2020, a PhD thesis completed at the University of Manchester . In Chapter 2, the author offers a comprehensive introduction to the topic in four paragraphs, with the final paragraph serving as an overview of the chapter’s structure:

PhD thesis literature review introduction

The last example is the doctoral thesis Metacognitive strategies and beliefs: Child correlates and early experiences Chan, K. Y. M. (Author). 31 Dec 2020 . The author clearly conducted a systematic literature review, commencing the review section with a discussion of the methodology and approach employed in locating and analyzing the selected records.

Having absorbed all of this information, let’s recap the essential steps and offer a succinct guide on how to proceed with creating your literature review introduction:

• Contextualize your review : Begin by clearly identifying the academic context in which your literature review resides and determining the necessary information to include.
• Outline your structure : Develop a structured outline for your literature review, highlighting the essential information you plan to incorporate in your introduction.
• Literature review process : Conduct a rigorous literature review, reviewing and analyzing relevant sources.
• Summarize and abstract : After completing the review, synthesize the findings and abstract key insights, trends, and knowledge gaps from the literature.
• Craft the introduction : Write your literature review introduction with meticulous attention to the seamless integration of your review into the larger context of your work. Ensure that your introduction effectively elucidates your rationale for the chosen review topics and the underlying reasons guiding your selection.

Master Academia

Get new content delivered directly to your inbox.

Subscribe and receive Master Academia's quarterly newsletter.

The best answers to "What are your plans for the future?"

10 tips for engaging your audience in academic writing, related articles.

10 key skills of successful master’s students

How to write effective cover letters for a paper submission

Dealing with failure as a PhD student

Reject decisions: Sample peer review comments and examples

Guide to Thematic Analysis

• Abductive Thematic Analysis
• Collaborative Thematic Analysis
• Deductive Thematic Analysis
• How to Do Thematic Analysis
• Inductive Thematic Analysis
• Reflexive Thematic Analysis
• Advantages of Thematic Analysis
• Thematic Analysis for Case Studies
• Thematic Coding
• Disadvantages of Thematic Analysis
• Thematic Analysis in Educational Research
• Thematic Analysis Examples
• Thematic Analysis for Focus Groups
• Thematic Analysis vs. Grounded Theory
• What is Thematic Analysis?
• Increasing Rigor in Thematic Analysis
• Thematic Analysis for Interviews
• Introduction

What is a thematic literature review?

Advantages of a thematic literature review, structuring and writing a thematic literature review.

• Thematic Analysis in Mixed Methods Approach
• Thematic Analysis in Observations
• Peer Review in Thematic Analysis
• How to Present Thematic Analysis Results
• Thematic Analysis in Psychology
• Thematic Analysis of Secondary Data
• Thematic Analysis in Social Work
• Thematic Analysis Software
• Thematic Analysis in Surveys
• Thematic Analysis in UX Research
• Thematic vs. Content Analysis
• Thematic Analysis vs. Discourse Analysis
• Thematic Analysis vs. Framework Analysis
• Thematic Analysis vs. Narrative Analysis
• Thematic Analysis vs. Phenomenology

A thematic literature review serves as a critical tool for synthesizing research findings within a specific subject area. By categorizing existing literature into themes, this method offers a structured approach to identify and analyze patterns and trends across studies. The primary goal is to provide a clear and concise overview that aids scholars and practitioners in understanding the key discussions and developments within a field. Unlike traditional literature reviews , which may adopt a chronological approach or focus on individual studies, a thematic literature review emphasizes the aggregation of findings through key themes and thematic connections. This introduction sets the stage for a detailed examination of what constitutes a thematic literature review, its benefits, and guidance on effectively structuring and writing one.

A thematic literature review methodically organizes and examines a body of literature by identifying, analyzing, and reporting themes found within texts such as journal articles, conference proceedings, dissertations, and other forms of academic writing. While a particular journal article may offer some specific insight, a synthesis of knowledge through a literature review can provide a comprehensive overview of theories across relevant sources in a particular field.

Unlike other review types that might organize literature chronologically or by methodology , a thematic review focuses on recurring themes or patterns across a collection of works. This approach enables researchers to draw together previous research to synthesize findings from different research contexts and methodologies, highlighting the overarching trends and insights within a field.

At its core, a thematic approach to a literature review research project involves several key steps. Initially, it requires the comprehensive collection of relevant literature that aligns with the review's research question or objectives. Following this, the process entails a meticulous analysis of the texts to identify common themes that emerge across the studies. These themes are not pre-defined but are discovered through a careful reading and synthesis of the literature.

The thematic analysis process is iterative, often involving the refinement of themes as the review progresses. It allows for the integration of a broad range of literature, facilitating a multidimensional understanding of the research topic. By organizing literature thematically, the review illuminates how various studies contribute to each theme, providing insights into the depth and breadth of research in the area.

A thematic literature review thus serves as a foundational element in research, offering a nuanced and comprehensive perspective on a topic. It not only aids in identifying gaps in the existing literature but also guides future research directions by underscoring areas that warrant further investigation. Ultimately, a thematic literature review empowers researchers to construct a coherent narrative that weaves together disparate studies into a unified analysis.

Organize your literature search with ATLAS.ti

Collect and categorize documents to identify gaps and key findings with ATLAS.ti. Download a free trial.

Conducting a literature review thematically provides a comprehensive and nuanced synthesis of research findings, distinguishing it from other types of literature reviews. Its structured approach not only facilitates a deeper understanding of the subject area but also enhances the clarity and relevance of the review. Here are three significant advantages of employing a thematic analysis in literature reviews.

Enhanced understanding of the research field

Thematic literature reviews allow for a detailed exploration of the research landscape, presenting themes that capture the essence of the subject area. By identifying and analyzing these themes, reviewers can construct a narrative that reflects the complexity and multifaceted nature of the field.

This process aids in uncovering underlying patterns and relationships, offering a more profound and insightful examination of the literature. As a result, readers gain an enriched understanding of the key concepts, debates, and evolutionary trajectories within the research area.

Identification of research gaps and trends

One of the pivotal benefits of a thematic literature review is its ability to highlight gaps in the existing body of research. By systematically organizing the literature into themes, reviewers can pinpoint areas that are under-explored or warrant further investigation.

Additionally, this method can reveal emerging trends and shifts in research focus, guiding scholars toward promising areas for future study. The thematic structure thus serves as a roadmap, directing researchers toward uncharted territories and new research questions .

Facilitates comparative analysis and integration of findings

A thematic literature review excels in synthesizing findings from diverse studies, enabling a coherent and integrated overview. By concentrating on themes rather than individual studies, the review can draw comparisons and contrasts across different research contexts and methodologies . This comparative analysis enriches the review, offering a panoramic view of the field that acknowledges both consensus and divergence among researchers.

Moreover, the thematic framework supports the integration of findings, presenting a unified and comprehensive portrayal of the research area. Such integration is invaluable for scholars seeking to navigate the extensive body of literature and extract pertinent insights relevant to their own research questions or objectives.

The process of structuring and writing a thematic literature review is pivotal in presenting research in a clear, coherent, and impactful manner. This review type necessitates a methodical approach to not only unearth and categorize key themes but also to articulate them in a manner that is both accessible and informative to the reader. The following sections outline essential stages in the thematic analysis process for literature reviews , offering a structured pathway from initial planning to the final presentation of findings.

Identifying and categorizing themes

The initial phase in a thematic literature review is the identification of themes within the collected body of literature. This involves a detailed examination of texts to discern patterns, concepts, and ideas that recur across the research landscape. Effective identification hinges on a thorough and nuanced reading of the literature, where the reviewer actively engages with the content to extract and note significant thematic elements. Once identified, these themes must be meticulously categorized, often requiring the reviewer to discern between overarching themes and more nuanced sub-themes, ensuring a logical and hierarchical organization of the review content.

Analyzing and synthesizing themes

After categorizing the themes, the next step involves a deeper analysis and synthesis of the identified themes. This stage is critical for understanding the relationships between themes and for interpreting the broader implications of the thematic findings. Analysis may reveal how themes evolve over time, differ across methodologies or contexts, or converge to highlight predominant trends in the research area. Synthesis involves integrating insights from various studies to construct a comprehensive narrative that encapsulates the thematic essence of the literature, offering new interpretations or revealing gaps in existing research.

Presenting and discussing findings

The final stage of the thematic literature review is the discussion of the thematic findings in a research paper or presentation. This entails not only a descriptive account of identified themes but also a critical examination of their significance within the research field. Each theme should be discussed in detail, elucidating its relevance, the extent of research support, and its implications for future studies. The review should culminate in a coherent and compelling narrative that not only summarizes the key thematic findings but also situates them within the broader research context, offering valuable insights and directions for future inquiry.

Gain a comprehensive understanding of your data with ATLAS.ti

Analyze qualitative data with specific themes that offer insights. See how with a free trial.

Virtual and augmented reality to develop empathy: a systematic literature review

• Open access
• Published: 04 May 2024

Cite this article

You have full access to this open access article

• Jose Lacle-Melendez   ORCID: orcid.org/0009-0000-9573-4194 1 ,
• Sofia Silva-Medina   ORCID: orcid.org/0009-0009-4625-8510 1 &
• Jorge Bacca-Acosta   ORCID: orcid.org/0000-0003-0381-6972 2  

74 Accesses

2 Altmetric

Explore all metrics

Recent research suggests that Virtual Reality (VR) and Augmented Reality (AR) as immersive technologies are effective in developing empathy. The main reason behind this assumption is that immersive technologies allow people to experience perspective-taking. However, there is a lack of systematic literature reviews that summarize the current state of research on VR and AR to elicit empathy. This paper reports a systematic literature review of 37 academic papers published between 2007 and 2023. The following categories were analyzed in this review: field of education, data collection instruments, sample size, statistically significant results, technologies used, research design, advantages, limitations, and future research. The main findings of this review provide an overview of the current state of research on immersive technologies to elicit empathy and the future challenges in this field. Some of the main findings involve: VR/AR immersion devices are effective and appealing to participants; the Interpersonal Reactivity Index was found to be the most relevant self-report measure; and larger sample sizes (over 100 participants) are vital in VR/AR-based empathy research to provide a quantitative perspective on participants distribution.

Similar content being viewed by others

Immersive Virtual Reality and Its Use in Developing Empathy in Undergraduate Students

Using immersive virtual reality in testing empathy type for adolescents

Empathy training through virtual reality: moral enhancement with the freedom to fall?

Avoid common mistakes on your manuscript.

1 Introduction

The rise of disruptive technologies has redefined patterns of social interaction, showcasing an adaptation in the ways individuals engage with one another. Virtual reality (VR) is a collection of hardware, including computers, head-mounted displays (HMD), and sensors, designed to experience telepresence [ 1 ]. Moreover, VR is also considered a computer system that enables users to create artificial environments in which they can interact, navigate, and immerse themselves in a three-dimensional space [ 2 ]. Augmented Reality (AR) is a technology that allows a real-time combination of virtual objects and real objects so that it seems that the virtual objects are part of the real world [ 3 ]. The main difference between VR and AR is that in VR the participant is completely immersed in a computer-generated visual environment and everything that the participant sees is artificial while in AR the participant sees the real world with some virtual objects superimposed that seem to co-exist in the real world.

VR and AR have emerged as cutting-edge tools that allow users to immerse themselves in simulated environments and experience sensory sensations that simulate real life. Considering this, VR and AR often refers to “enhanced user interfaces.” This encompasses viewing and navigating a 3D environment and interacting with its components in real time. For these creators, the user’s interactive experience in the real world can be received through stimulation of the five human senses (sight, hearing, touch, taste, and smell). In the same line of thought, VR and AR can be an enhancer of prosocial behaviors through empathy, using tools intertwined with current technological developments. In other words, VR enables immersion in a simulated environment like real life, and through interaction with this sensory environment, it can strengthen communication and understanding of others’ perspectives [ 4 ]. In that regard, VR has been considered a medium for perspective-taking [ 5 ]. It allows users of this technology to directly experience feelings and perspectives in a controlled and safe environment. Thus, VR can be understood as a set of computer technologies that provide access to simulated spaces through visual devices, where a person can acquire the sensation of presence, interact within that space and be in the shoes of others.

AR and VR can be used in different contexts for creating empathy and pro-social behaviors [ 6 , 7 ]. From an etymological perspective, empathy can be understood from its Greek root Παθεûv (epathón, to feel) and the prefix εv (an inseparable preposition meaning within). The origin of the term empathy dates back to 1873 when the philosopher Robert Vischer used the German term “Einfühlung” (feeling into) as an expression in art appreciation. Later, the term was used in English in the book “Lectures on the Experimental Psychology of the Thought-processes” in 1909, with a meaning of “feeling oneself into the other, being interpenetrated” [ 8 , p.1].

Empathy can be initially defined as the emotional communication between one person and another, responding assertively within their social environment. Empathy is the ability to identify one’s own emotions and those of others and respond to them constructively [ 9 , p.40]. In other words, the development of these skills not only influences personal well-being but also has a significant impact on various areas of individual adjustment. In simpler terms, it can be understood as the process by which an individual has the ability to understand the feelings of others, allowing them to perceive reality from the other person’s perspective rather than their own. Empathy is usually divided into emotional empathy and cognitive empathy [ 10 ]. One the one hand, emotional empathy means that the person is emotionally moved by a situation. On the other hand, cognitive empathy involves understanding thoughts and the emotion of others and this has been regarded as perspective taking.

It is worth mentioning that applications of VR and AR to develop empathy, while diverse and booming in various fields of study, lack extensive systematic research that reviews data from multiple individual studies to determine a general estimation of the effects of interventions, scope, limitations, variables of interest, and to assess the consistency and variability of results among individual studies. Therefore, it is of methodological importance to conduct a systematic review of current technological trends in various areas and fields of research where VR has been used as a medium for empathy development.

In alignment with this perspective, the present systematic review seeks to comprehensively address 11 research inquiries aimed at elucidating the correlation between exposure to immersive experiences in virtual reality (VR) or augmented reality (AR) and the cultivation of empathy. These 11 research questions have been categorized into four dimensions for the sake of facilitating understanding, organization, and presentation of information. The ensuing research questions steer the course of this review:

Application-domain related research questions

The primary objective of these inquiries is to delineate how AR and VR have been employed to foster empathy across diverse professional domains and to ascertain the prevalence of significant findings in related studies. The relevance of these research questions lies in their capacity to enable researchers to pinpoint specific professional fields wherein AR/VR could potentially be applied to nurture empathy. Additionally, they aid in identifying research lacunae within various professional domains. The consequential significance of results in these studies provides valuable insights for researchers to recognize potential benefits conferred by these immersive technologies.

What are the professional fields of study where virtual reality or augmented reality have been used to promote empathy?

How many studies have reported statistically significant effectiveness percentages in research utilizing virtual reality or augmented reality to foster empathy?

Research questions about methodological aspects

the objective of this set of research questions is to elucidate the methodological intricacies employed in studies investigating the promotion of empathy through AR and VR. The significance of these research questions lies in their capacity to provide insights into the methodologies utilized within the field. This comprehension is instrumental in evaluating the robustness and reliability of findings, fostering a deeper understanding of the research landscape in this domain.

What data collection instruments have been most used in studies utilizing virtual reality or augmented reality to foster empathy?

How many participants or research sample have been used most frequently in studies where virtual reality or augmented reality has been used to foster empathy?

What research design has been predominantly employed in studies where virtual reality or augmented reality has been used to foster empathy?

What percentage of studies have been reported as scientific articles versus conference papers?

Advantages, limitations and future research directions

this set of research queries seeks to elucidate the merits, constraints, and prospective avenues for further investigation as documented in the literature pertaining to the application of AR and VR in empathy development. These inquiries hold significance as their responses encapsulate a concise overview of primary discoveries within the field, the principal constraints and challenges encountered, and the potential avenues for future research endeavors. Exploring these research questions contributes to a comprehensive understanding of the current state of knowledge, facilitating informed discussions on the advancements, challenges, and potential future directions in the domain of AR and VR’s impact on empathy.

What advantages have been described in studies where virtual reality or augmented reality has been used to foster empathy?

What limitations have been reported in studies where virtual reality or augmented reality has been used to foster empathy?

What have been the most frequent recommendations for future research in studies utilizing virtual reality or augmented reality to foster empathy?

Technology-related research questions

this set of questions seeks to ascertain the specific technological hardware and software utilized in studies focused on the utilization of AR and VR in empathy development. The intent behind these questions is to discern the capabilities of particular devices and explore potential avenues for further enhancement, aligning with the imperative for increased research in this domain emphasized by Ventura et al. [ 11 ].

What equipment or technological tools have been used for immersion in virtual or augmented reality in the analyzed studies?

Which software has been used to develop immersive environments in research studies where virtual reality or augmented reality has been used to foster empathy?

In essence, the research team has formulated this set of research questions with the aim of delineating the research landscape concerning empathy development through AR and VR applications. From our standpoint, these research questions serve the purpose of offering a comprehensive survey of studies within this domain, enabling fellow researchers to pinpoint existing gaps in the literature and identify potential avenues for further research.

The rest of this paper is organized as follows: Section 2 describes the related work; Section 3 describes the method followed to conduct the systematic literature review. Section 4 presents the results organized by each research question. Section 5 presents the risk of bias analysis. Section 6 discuss the results obtained in this review. Section 7 describes the limitations of this review. Finally, Section 8 describes the implications of this review for education and training and Section 9 presents the conclusions of this study.

2 Related work

There is a large and growing body of literature that has demonstrated that VR can be effectively utilized as a tool for the development of social and emotional skills, such as empathy. In this context, various studies have investigated the viability of virtual reality in enhancing empathy in individuals. In this section, we present a summary of similar systematic reviews and meta-analysis on VR and empathy and we show how our systematic review extend previous studies in the field and how the systematic review fills a gap in the literature. Table 1 shows a summary of previous studies in the field.

Overall, previous systematic reviews, surveys and meta-analysis on VR and empathy have shown an overview of how VR has been used to create empathy and how some associated factors such as prejudice and intergroup bias might mediate or moderate the influence of VR on empathy. However, the systematic review presented in this article extend previous reviews by updating the research up to 2023 and addressing research questions that have not been addressed in previous reviews. The systematic review presented in this paper holds significant importance in the academic domain as its primary objective is to address aspects that have not been thoroughly explored in other systematic reviews, surveys or meta-analysis such as the ones reported in [ 11 , 12 , 13 ]. Particularly, by providing detailed information, it establishes a solid foundation for future investigations, facilitating the path for researchers who seek to consult tools and technologies utilized within the study’s context, as well as identifying statistically significant impacts on empathy. Moreover, a rigorous examination of limitations encountered in previous research is proposed to establish a critical and reflective framework concerning the current frontiers of knowledge in the field. Considering the above, this systematic literature review contributes to the body of knowledge in the field of VR and AR to develop empathy and is a valuable resource to guide and suggest future research, offering clear and well-founded recommendations that will contribute to the advancement of this field.

The main contribution of this paper is that it summarizes previous research done on the use of VR and AR to develop empathy. This literature review updates previous literature reviews and surveys on VR to develop empathy and, to the best of our knowledge, is the first literature review that also summarizes research on AR to develop empathy.

Following the guidelines outlined by Botella and Zamora [ 22 ], a systematic literature review is structured as follows:

Problem Formulation

Study Search

Study Coding

Analysis and Interpretation

Publication

Figure 1 depicts the PRISMA flow diagram. This diagram was generated based on the PRISMA 2020 statement [ 23 ]. It depicts the process of study identification and selection for the current systematic review. First, a total of 738 records were found after conducting the search. By excluding those published prior to 2007, those written in a language different to English or Spanish, and those different to articles, conference papers or book chapters, the dataset resulted in 636 records. Following the screening process, 58 records were excluded as the key terms (empathy and AR/VR) were merely mentioned in the paper but did not constitute the primary focus. Instances included references to AR and VR technologies without the main topic centering on empathy, or the co-occurrence of empathy, AR, and VR terms without the primary objective of fostering empathy through these technologies. Consequently, 578 records underwent evaluation for eligibility. During this phase, the authors meticulously reviewed the abstracts of each document to confirm its relevance to the designated topic. If the abstract did not provide sufficient clarity regarding the study’s suitability, a comprehensive examination of the full text was undertaken to determine inclusion in the review. Ultimately, 37 studies met the criteria for inclusion. The authors then scrutinized the full text of these 37 articles to extract pertinent information and address the research questions.

PRISMA flowchart and selection of bibliographic material

It should be noted that the three authors of this article were in charge of collecting the articles (identification phase in Fig. 1 ) and the screening process. Then, two of the authors coded the selected articles through a matrix where all the inclusion and exclusion criteria were established. Then the third author also validated the coding process.

Taking into account the aforementioned aspects and the model proposed by Botella and Zamora [ 22 ], the impact of VR and AR to develop empathy was addressed through a systematic literature review of 37 articles retrieved from SCOPUS, using the search string ( TITLE-ABS-KEY ( “virtual reality” ) OR TITLE-ABS-KEY ( “augmented reality” ) ) AND TITLE-ABS-KEY ( “empathy” ). Scopus was selected as the primary database for retrieving studies because it is one of the largest abstracts and citation databases with a high quality of indexed publications. In this systematic review, we did not consider other databases because we had restrictions in accessing other databases such as Web of Science. Other databases such as APA PsycArticles were not included because the pilot searches did not retrieve relevant results for the scope of this systematic review.

In this review, we did not include any other term related to empathy because in the pilot searches before conducting the final search the inclusion of other terms such as prejudice, intergroup, perspective-taking among others restricted the number of results obtained and the results overlapped other existing systematic reviews. In that regard, we decided to maintain a more general search string to collect articles on the topic of VR and AR to develop empathy. In this way, we are able to provide a more general landscape of research instead of a more focused review that might overlap previous reviews.

The inclusion criteria considered in the systematic review were as follows:

Studies conducted between 2007 and 2023. The timeframe was selected and studies before 2007 were not included because the pilot searches conducted prior to the main search for this systematic review showed that the articles before 2007 provided important background and foundations on the use of VR to develop empathy but did not provide insights into the effect of VR to develop empathy due to the maturity of VR before 2007.

Research about the use of virtual reality and/or augmented reality to develop empathy.

Studies written in English or Spanish.

Journal papers, conference papers and book chapters.

Exclusion criteria was:

Studies in languages other than English and Spanish, studies prior to 2007.

Other systematic reviews (these were reported in the related work of this paper), and those that did not involve virtual reality and/or augmented reality as technologies for fostering empathy.

Book reviews, notes, erratum, editorials, letters to the editor, doctoral theses, master’s dissertations, and other non-scientific documents.

The 37 articles were thoroughly read and analyzed by two of the authors and each article was coded according to the categories defined by the researchers to consolidate and answer the research questions. The categories emerged from the research questions. Table 2 shows the 11 research questions grouped into the four dimensions (as presented in the introduction) and each research question has the category that was used to code each study.

Once the analysis categories were defined, they were coded into two groups based on their unit of measurement. On one hand, categories that could be numerically evaluated were included, organizing them into subcategories and tallying the number of identified studies and their respective percentages. This process was carried out using software such as Excel and JASP. On the other hand, categories with nominal characteristics that required interpretation were considered as the second group for coding, thus providing answers to the research questions. It is important to note that the coding matrix design and article consolidation were conducted by two researchers and validated by a third party.

This section in the systematic literature review answers the research questions previously formulated. Through a detailed analysis of the existing scientific literature, relevant data has been collected and subsequently encoded according to the categories defined to be presented in the form of tables and structured subcategories with corresponding percentages. These tools allowed for a clear and comparative visualization of the results, facilitating the understanding of the patterns and trends identified in the research. The percentage analysis provides a deeper understanding of the data distribution and enables significant conclusions to be drawn regarding the research questions. This was achieved by dividing the frequency of each subcategory by the total number of subcategories mentioned and multiplying by 100 to obtain the percentage value. The following sub-section presents the results in detail, offering a comprehensive and rigorous overview of the collected data organized for each research question.

4.1 Application-domain related research questions

4.1.1 what are the professional fields of study where virtual reality or augmented reality have been used to promote empathy.

Table 3 describes the areas where studies on virtual and/or augmented reality have been reported, along with their respective nominal quantity of studies and percentages.

The analysis examined the different areas in which virtual or augmented reality has been used to promote empathy. The results revealed that research has been conducted in fields such as Experimental Psychology, Medicine, Education, Organizational Psychology, Social Psychology, Art, Marketing, and Neuroscience. When looking at the percentages, it was observed that most of the studies focused on Education (29.73%), followed by Experimental psychology (24,32%), and Medicine (16.22%).

4.1.2 How many studies have reported statistically significant effectiveness rates in research where virtual reality or augmented reality has been used to foster empathy?

Table 4 shows the results obtained for studies that have reported statistically significant results, as well as those where, due to the nature of the research, such significance does not apply.

The percentages mentioned address the question regarding the number of studies that have reported statistically significant effectiveness in research using virtual reality or augmented reality to foster empathy. It is noteworthy that 59.46% of the reviewed studies reported statistically significant differences. On the other hand, only 2.70% reported that there were no statistically significant differences.

4.2 Research questions about methodological aspects

4.2.1 what data collection instruments have been most used in studies utilizing virtual reality or augmented reality to foster empathy.

Table 5 shows the data collection instruments used, as well as the number of studies that report them and their respective percentages.

Table 5 reveals valuable information regarding the data collection instruments used in empathy studies. The Interpersonal Reactivity Index (IRI) stands out with a significantly high percentage of 18.75%, indicating its relevance and frequent use in empathy assessment. Furthermore, the results highlight the importance of two research approaches: the Likert-type surveys and interviews, both of which obtained similar percentages around 16,67%. This suggests that these methods are considered significant in measuring and understanding empathy. On the other hand, categories such as Multidimensional empathy scale, Virtual reality quiz, Cognitive and affective empathy test, Jefferson Empathy Scale (JSE) questionnaire presented lower percentages, approximately ranging from 2,08–4,17%. Additionally, other instruments such as questionnaires accounted 14.58%.

4.2.2 How many participants or research sample have been used most frequently in studies where virtual reality or augmented reality has been used to foster empathy?

Table 6 shows the number of participants in virtual reality or augmented reality studies, along with their respective percentages.

The participants were classified into different categories based on sample size. The category with the highest percentage corresponds to studies that included more than 100 participants, representing approximately 35.14% of the total. On the other hand, there were groups with lower representation, such as those with less than 51–75 and cases of 76–100 participants both with a percentage 8,11% and 10,81%, respectively. The intermediate categories show a range where sample size was not applicable due to the type of study,13,51% and, the studies with a range of participants between 25 and 50, accounting 16,22%.

4.2.3 What research design has been predominantly employed in studies where virtual reality or augmented reality has been used to foster empathy?

Table 7 contains the information on the percentages of research design types where virtual reality or augmented reality has been used to foster empathy.

Regarding the methodological design, five different types of research were found in the total number of studies reviewed. In terms of classification by types, experimental studies accounted for a total of 19 studies, representing 51.35% of the total studies reviewed. Following that, descriptive studies accounted for 7 studies, accounting for 18,92%. Qualitative studies accounted for a total of 4 studies, with an equivalent average of 10.81%. Quasi-experimental and mixed methods came next, each comprising 3 studies, and a percentage of 8.11%. Finally, Exploratory methods studies accounted for 1 article representing 2,70%.

4.2.4 What percentage of studies have been reported both as scientific articles and conference papers?

Table 8 shows a summary of the studies reviewed classified according to the typology as journal articles, conference papers or book chapters.

Regarding the analyzed articles that used virtual or augmented reality for empathy enhancement, there is a percentage of 83.78% of studies reported as journal articles, which represents a total of 31 articles from all the reviewed documents. This is followed by 4 conference papers, equivalent to 10.81%, and finally, two book chapter with a percentage value of 5.40%.

4.3 Advantages, limitations and future research directions

4.3.1 what advantages have been described in studies where virtual reality or augmented reality has been used to foster empathy.

Regarding the advantages described in studies where virtual reality or augmented reality has been used to foster empathy, it has been found that statistically significant changes occurred in various empathy-related aspects, increasing participants’ ability to understand others’ perspectives through the alteration of their virtual bodies. This is supported in the study by Wilding et al. [ 4 ], where participants gained a greater understanding of the challenges faced by individuals with disabilities when experiencing frustration within the virtual world.

Furthermore, Fisher [ 24 ] argues that although empathy in virtual reality is not directly established between a user and the subject of a real-life experience, the medium’s capacity to place a body within a new space provides an opportunity for enhanced understanding of others through empathic realities. Additionally, statistical results and user testimonials reveal that the functionalities and elements implemented in the developed application contribute to the promotion of empathy compared to conventional methods of visualization and annotation in 360-degree videos. Findings indicate that experiencing news through a head-mounted display for 360-degree videos resulted in higher self-location and co-presence compared to interacting with the same video on a desktop or reading a textual version. Therefore, the use of virtual reality as a medium to support empathy generation holds promise due to the benefits and advantages it offers.

It is worth noting the significant contributions that VR offers to educational spaces, particularly in terms of additional pedagogical considerations regarding the use of VR in historical education, including incorporating virtual reality into constructivist approaches. According to Castaño & Gonzalez [ 25 ], university students attribute importance to AR and VR in the educational context: it improves academic performance, changes the way of teaching and learning, enables more experimental learning, increases the level of understanding, offers models of relevant experiences, and enhances possibilities for engagement and interaction in the educational context. Furthermore, by generating a highly stimulating space for understanding the reality faced by others, a sense of shared frustration and pain is incorporated within the virtual world, leading participants to gain a greater understanding of the challenges faced by individuals with disabilities, autism, among others.

Parra Vargas et al. [ 26 ] illustrate the potential of a new VR organizational environment combined with machine learning to discriminate empathy dimensions. Additionally, this multi-method approach can increase knowledge about attention and behavior patterns and decision-making processes carried out by workers with different levels of empathy in complex work situations. Furthermore, unlike most assessments that use subjective self-report measures, this method combines neuroscience with VR, attributing greater objectivity and ecological validity to the results.

In summary, the findings suggest that augmented reality (AR) and virtual reality (VR) hold significance in enhancing individuals’ capacity to comprehend others’ perspectives. A primary advantage of AR and VR lies in their capability to immerse individuals in novel environments, fostering a deeper understanding of others’ thoughts and conditions. Additionally, the heightened levels of interaction facilitated by AR and VR surpass the efficacy of 360-degree videos in empathy development. Finally, the integration of artificial intelligence methods with AR and VR technologies introduces novel possibilities for empathy cultivation, as applications can adapt to individual participants, offering more personalized scenarios to enhance empathic experiences.

4.3.2 What limitations have been reported in studies where virtual reality or augmented reality has been used to foster empathy?

Regarding the limitations evidenced in studies where virtual reality or augmented reality has been used to foster empathy, several aspects have been identified. One limitation is the sample size, as in many cases, the total number of participants was not significant enough to establish generalizability of the findings. This lack of a representative sample also reflects limited socio-demographic information, and in some instances, the absence of a control group, pretest and posttest data, and long-term follow-up of the results. Additionally, participants’ unfamiliarity with virtual reality tools was detected, which were often presented in foreign languages or with proprietary licenses that limited their use. Furthermore, difficulties related to the COVID-19 pandemic were reported in studies that started before the preventive isolation measures were implemented. These studies had to change their initially planned methodology, requiring new organization and logistics to carry out the interventions in a timely manner, as mentioned by Villalba [ 27 ].

In terms of the content and format of empathy-building interventions, problems have been identified. Previous studies have shown that people exposed to persuasive messages can experience a psychological reaction, perceiving these messages as a threat to their freedom. As a result, a “boomerang” effect can occur, where the recipient acts in the opposite direction to that advocated by the message. This limitation implies not fully utilizing all emotions, as participants’ responses may be influenced by individual differences or previous experiences with the displayed content, resulting in a poor and somewhat biased understanding of empathy and prosocial moral reasoning.

In line with the issues, inadequate methodologies have been implemented, which in turn present inconveniences. This includes qualitative data being used inappropriately, risking objectivity, and studies that are solely descriptive, making it impossible to compare theory and practice to validate theoretical assumptions. Additionally, studies solely relying on self-reported measures limit researchers’ ability to draw conclusions about how the use of virtual reality devices influenced participants’ behavior, particularly their ability to empathetically communicate with individuals experiencing auditory verbal hallucinations (AVH). Since self-reported measures are based on participants’ subjective perceptions, it cannot be certain if the perceived changes in empathetic communication would translate into empathetic behavior in real life. As mentioned by Libera et al. [ 28 ], researchers face challenges in drawing conclusions about how the use of devices influenced participants’ behavior, especially their ability to empathetically communicate with individuals experiencing AVH. Due to the subjective nature of self-reported measures, it is uncertain whether these perceived changes in empathetic communication would translate into empathetic behavior in real-life situations.

In summary, the limitations identified in the reviewed studies revolve around factors such as small sample sizes, impeding the generalizability of results and the demonstration of genuine effects. The absence of long-term follow-ups and a scarcity of studies employing longitudinal research designs further underscore limitations within the research landscape. These findings align with Ventura et al.‘s [ 11 ] observations. Additionally, the restricted familiarity with augmented reality (AR) and virtual reality (VR) equipment hampers the potential impact of these technologies on empathy, complicating researchers’ efforts to measure their effects. The influence of individual differences on participants’ responses introduces another layer of complexity, potentially leading to unexpected results in the effects of AR and VR. Lastly, the reliance on self-reported measures in some studies introduces a potential source of bias, as these measures may not fully capture participants’ attitudes.

4.3.3 What have been the most frequent recommendations for future research in studies that have used virtual reality or augmented reality to promote empathy?

Regarding the recommendations for future research, one is to conduct longitudinal studies that assess empathy [ 29 ]. In longitudinal studies, the novelty of the technology effect can be controlled and determine the real affordances of VR and AR to develop empathy. Additionally, for upcoming research on virtual tools, empathy, and prosocial moral reasoning, these studies can be conducted with larger sample sizes and implemented in other countries to determine the effect of cultural differences on empathy. It is also emphasized the importance of research that integrates VR embodiment with clinical assessments and patient experiences, as stated by Aya Briñez et al. [ 30 ]. Embodiment is relevant for perspective-taking because the participant can take the body of another person and have a better experience that might increase empathy. Furthermore, future research is needed to explore the use of virtual reality for disability advocacy. On a different note, there were some comments about technical issues, such as interruptions during the presentation due to network problems or getting lost in the virtual world, especially for those who experienced VR for the first time. Li & Kyung Kim [ 31 ] suggest that future work should further examine the distinctions between visual perspectives and perspective-taking in virtual reality.

4.4 Technology-related research questions

4.4.1 what equipment or technological tools have been used for immersion in virtual reality or augmented reality in the analyzed studies.

Table 9 shows the technological means used for immersion in virtual reality environments, along with the number of related studies for this analysis and their respective percentages.

In this research question, the sub-category with the lowest number of studies were “Mobile AR”, “Augmented Reality Cards”, “Oculus Go 360”, “Google Cardboard” and “NVIS nVisor SX111” each accounting for 2.70% of the studies. “360 Immersion Device” is the category with the highest percentage of studies after “Not specified,” representing 18,92% of the total. Moreover, the subcategory “Not specified” has the highest number of studies, accounting for 37.84% of the total. An explanation of this result might be that some of the articles reviewed are theoretical so in the research the authors did not use a particular device.

4.4.2 Which software has been used to develop immersive environments in research studies where virtual reality or augmented reality has been used to foster empathy?

Table 10 shows the software that have been used to build immersive environments in research studies where virtual reality or augmented reality has been used to promote empathy.

Regarding programming languages and software, we found that software developed by Embodiedlabs*, Autopano Video, ImercyVE, AR Foundation, Optitrack Arena Motion, Ataturk, Skybox, Unreal, and AR Core accounted for only 2.5%, with each being the focus of a single study. Unity was used in 6 articles, representing a percentage of 15%. Additionally, it is found that 22 articles did not specify the programming language, or the tool used to develop de immersive experience, accounting for 55%. Finally, the software developed by https://www.embodiedlabs.com is mentioned, which was present in two of the reviewed articles, resulting in a percentage of 5%.

5 Risk of bias analysis

A risk of bias analysis of the selected literature was carried out using the analysis tool proposed by Sterne et al. [ 32 ]. To perform this analysis, both qualitative and quantitative research designs, including experimental, non-experimental, and correlational designs, were considered. This is because these types of designs can be assessed using the tool, and a total of 32 articles were analyzed and the results are shown in Table 11 . In this table, 5 dimensions are assessed by using a group of criterions and finally the overall bias for each study is calculated. Table 11 shows the average score obtained for each dimension for the total of articles analyzed Fig. 2 .

The Table 11 with the case analysis shows the percentage values ​​of the categories or dimensions and the slight biases.

Risk of bias analysis

On one hand, it can be observed that all the dimensions analyzed by the instrument meet or exceed the 74,1% threshold for low risk of bias overall. However, it is relevant to note that the dimensions of randomization, measurement of the outcome, and selection of the reported outcome show a percentage of medium risk or are categorized as ‘some concerns,’ with a maximum of 22,2% for the first, 3,7% and 7,4% for the latter two, respectively. It is important to highlight that only one study, in a single dimension (variation in the intervention), presents a high risk of bias due to a lack of information when assessing research criteria.

In general, a predominantly low risk of bias trend was obtained in the studies analyzed, with at least 74,10% assessed by both the researchers and the instrument’s algorithm. However, the remaining 25,90% of the studies analyzed presented a moderate or high risk of bias. These studies may be more susceptible to systematic errors that could influence the results and conclusions, emphasizing the importance of caution when interpreting these findings and considering potential limitations in the available evidence.

6 Discussion

Although the different bibliographical sources consulted differ slightly in their focus of interest, such as the systematic review by Lee et al. [ 19 ], who explored the design and effectiveness of virtual patient-based medical communication skills training systems through 14 mostly quantitative studies, finding that effective virtual patient systems include well-designed educational interventions, human feedback, and reflection after the activity. Similar results to the systematic review presented in this paper are presented because it confirms that virtual reality increases and improves empathy processes, a result that is also present in the meta-analysis by Ventura et al. [ 14 ] whose main interest is focused on clarifying the existing research on virtual reality as a means to provoke empathy. The results reveal statistically significant positive changes in perspective taking after VR exposure. Likewise, Gerry et al. [ 11 ] investigated the efficacy of VR training for empathy and compassion. These components correspond to three key design characteristics of immersive VR technologies: biofeedback, perspective taking, and simulation, thus demonstrating that empathy can be trained and promoted thanks to different immersive technologies. This is something that is intended to be emphasized throughout this systematic review. Finally, Foxman et al. [ 20 ] propose that empathy is a term that journalists and researchers aspire to show the potential of immersive media for prosocial change, building on fundamental research in the field. However, it is not the only field of interest. Therefore, our systematic review seeks to delve into various areas such as the arts, education, marketing, neurosciences, and other previously mentioned areas in which VR, AR, and empathy are treated as an area of interest. The main purpose of this systematic literature review is to show an overview of the research done in the field of VR and AR to promote empathy. In this section, the results and identified trends are interpreted, the effects and relationships found will be examined, as well as the differences or similarities between subgroups and analyzed variables. Furthermore, these results will be contextualized with the existing literature, allowing for the establishment of connections and significant contributions to the field of study.

6.1 Is AR and VR effective for fostering empathy?

By analyzing current research in the field of AR and VR to develop empathy, the main conclusion is that: It is premature at this early stage to consider VR as a medium that generates empathy over other media such as film, television or photography. This finding is also in line with the findings by Sora-Domenjó [ 33 ]. This finding is also supported by previous research stating that there still a lack of empirical support for the popular claim of VR as the “ultimate empathy machine” [ 34 ]. It is clear from the research that, under certain conditions, alterations in one’s own digital representation of oneself can have a significant impact on how a person behaves in a virtual environment and that also affects their behaviors and attitudes, promoting some of the qualities of empathy. The results also indicate a lack of consensus when considering VR as a narrative medium that provokes empathy due to its immersive qualities. Empathy is a complex phenomenon where cultural and personal implications can affect VR experiences, modulating and differentiating empathy awareness depending on each person.

As shown in this article, some VR experiences designed to elicit empathy could generate negative and counterproductive effects in relation to the outgroup, depending on the subjects and the experimental design. It has been widely demonstrated that empathy in virtual reality films includes, at a minimum, social, cultural, and physical biases that can hinder empathic responses, and that different technical configurations may also be related to these affective responses. The role of interactivity and action in arousing empathy in virtual reality experiences using current technical configurations does not appear to be particularly relevant.

Furthermore, although some results suggest that VR cinematic experiences can modulate emotions and empathy in a short period of time for a specific group of people, the long-term effects of exposure with VR is still unclear, as researchers point out [ 33 ]. Based on the previous effects of mobile and web exposure, one can predict that immersive virtual reality technologies could eventually have similar or even worse results, affecting the same limbic areas involved in sympathetic resonance.

A related comment has been made about the need to consider the “conceptual position of the subject” in relation to the personal narratives developed (especially in VR social films) and the audience. Furthermore, at these stages of VR development, the reflection on the future consequences of using VR is necessary because the impact of VR on society is difficult to predict [ 33 ]. According to Sora-Domenjó [ 33 ], VR experiences could be defined as part of a collective reflection. In that regard, co-design and co-creation methodologies could be effective so that stakeholders can actively participate in the design and development process so that VR experiences can be more effective to develop empathy in certain fields.

6.2 Methodological aspects of studies about AR and VR to develop empathy

It is important to contrast the results considering the findings by Dhar et al. [ 18 ], where it is stated that virtual reality immersion devices are safe, effective, and appealing to participants despite their interdisciplinary variations. From this perspective, it can be confirmed that these findings demonstrate the wide range of areas, especially education and medicine, that have explored the potential of virtual and augmented reality as tools for fostering empathy in a transdisciplinary manner, as evidenced in the Section 4 (Table 3 ). Additionally, the importance of considering multiple data collection instruments in the study of empathy is highlighted, with the IRI report being the most relevant. This result echoes the findings by Mestre Escrivá et al. [ 35 ]: “The Interpersonal Reactivity Index (IRI) (Davis, 1980, 1983) is one of the most widely used self-report measures to assess empathy. It has been applied in different studies to evaluate gender differences in empathic disposition” (p. 255). Furthermore, adapted questionnaires on the use of virtual reality enrich the understanding of this field and have vast potential to be explored as information gathering tools for subsequent social interventions, as presented in Section 4 (Table 5 ). Moreover, this result is in line with the results of the survey of literature by Christofi [ 13 ], who found that most of the research on VR and empathy has used self-reported instruments. An implication of this result might be that future researchers in this field should validate adaptations of the questionnaires to other languages or tailor-made questionnaires to ensure the reliability and validity of the instrument.

Moving on to the population perspective, the reported findings emphasize the need to consider studies with more than 100 participants, as corroborated in Section 4 (Table 6 ). This is relevant as it underscores the importance of sample size, offering a quantitative perspective on how participants are distributed in the context of fostering empathy through the use of virtual reality, as supported by García-García et al. [ 36 ]: “Calculating the number of participants to be included in a study (…) enables researchers to know how many individuals need to be studied to estimate the desired degree of confidence or difference between study groups” (p. 218). A bigger sample size allows researchers to conduct more robust studies that are less sensible to biased, reduce error, and increase precision. According to García-García et al. [ 36 ]. “a study with an insufficient sample size will estimate a parameter with low precision or will be unable to detect differences between groups, leading to erroneous conclusions” (p. 218).

Regarding the technological means for virtual immersion, Useche Rodríguez [ 37 ] pointed out that “360 videos can be used to present audiovisual content aimed at generating empathy in viewers. The research evaluates the effectiveness of the tool developed to support empathy in 360 videos” (p. 8). These technological means encompass a wide range of equipment used in various disciplines, with the most common being 360° video immersion devices and Oculus Rift mixed reality headsets, as evidenced in the Section 4 (Table 9 ). A possible interpretation of this result is that 360° videos are easy to deploy in VR devices, are cheaper to produce and provide more realism when compared to the development of a tailor-made VR or AR experience because VR/AR experiences require the support of software developers, and their development is more time-consuming and expensive. Moreover, in 360° videos the interaction is more limited whereas in VR/AR environments the interaction is higher. Thus, we call for more research studies that involve the development of VR/AR experiences to really exploit the potential of this technologies and uncover the real affordances of the technology to foster empathy.

Another finding in this review of literature deals with the reporting of statistically significant results in research where virtual reality or augmented reality has been used to foster empathy. In total 45.95% of the studies (as shown in Table 4 ) report statistically significant results. This finding is in line with previous research that has found that VR is effective for perspective-taking but there are still some inconclusive results regarding empathy [ 14 ] so further research is needed. Connected to this idea, according to Ordoñez [ 38 ], some of the advantages of VR and AR include “multisensory learning (sight, sound, touch), cognitive improvement, effective combination of physical and virtual worlds, high-quality 3D content and animations in real space, elimination of geographical and temporal limits, content enrichment, and user-friendliness” (p.13). In this literature review, we confirmed that another advantage of VR and AR is that these technologies are effective for developing empathy and this finding contributes to the body of knowledge on the advantages of these technologies.

Experimental research designs have been used in studies of VR and AR to develop empathy (Table 7 ). Experimental research design can be useful to identify the affordances and benefits of VR and AR to foster empathy when compared to other technologies or strategies. Moreover, changes in empathy can be identified by using pre-post tests in experimental research designs. We recommend that, in future studies, researchers use experimental research designs and more robust statistical methods such as Structural Equation Modeling (SEM) to identify some predictors of empathy during VR and AR experiences that are used to foster empathy. Moreover, there is still a lack of research on the features of VR and AR that positively influence empathy and the personal traits that might moderate the factors that influence empathy. Thus, further research is needed in this aspect.

Additionally, it is highlighted that 83.78% of the research has been published in academic journals (31 out of 37 studies reviewed as depicted in Table 8 ) and only 4 studies were published in conference papers and 2 in book chapters. This result shows that most of the research conducted in the field has been peer-reviewed and this ensures the quality of the findings in each paper.

Research has demonstrated the potential of the new organizational context of VR combined with machine learning to distinguish empathy dimensions. Unlike most evaluations that use subjective self-report measures, this approach combines neuroscience with VR, providing greater objectivity and validity to the results. This, in turn, facilitates systematic reviews analyzing the role of these techniques in the importance of immersion in different contexts, as mentioned by Parra Vargas et al. [ 26 ].

6.3 Technological aspects

In this review, we identified some technologies used to create experiences in VR and AR to develop empathy. A remarkable result is that most of the studies analyzed in this review used 360° video in VR headsets. This means that participants were, in most of the cases, passive subjects in the VR experience and therefore the effect on empathy might have been diminished as a result of the lack of interaction with some elements in the experience. However, a positive aspect of using 360° videos is that the VR experience is closer to the reality. Previous research have demonstrated that social presence (which involves the sense of being there in VR) in 360° videos has a positive effect on prosocial behaviors [ 39 ]. In that regard, future studies in which the participants’ sense of presence can be maximized could contribute to a better experience increasing the levels of empathy. Moreover, another feature of VR that can be exploited for developing empathy is embodiment. Previous research has shown that empathy might increase when some features of embodiment are present [ 40 ]. The engagement and sense of presence created by VR experiences might intensify some emotional reactions such as empathy [ 41 ]. According to Ventura et al. [ 14 ], further research should determine if the sense of presence is better than the embodiment or not. The use of other VR headsets or fully immersive VR experiences is still limited so further research on the effect of highly interactive VR experiences (apart from 360° video) might provide more insights into the real effect of VR on empathy. To date, it is unclear which device would be the most effective for presenting VR or AR experiences and this is line with the call for more research stated by Ventura et al. [ 14 ].

Research on AR or Mixed Reality (MR) to develop empathy is still in its infancy. There are few studies that use AR or MR as immersive technologies to create empathy when compared to the number of studies using VR. An interpretation of this result might be that VR as a more immersive technology could be seen as a more powerful to develop empathy when compared to less immersive technologies such as AR or MR. However, further research needs to be conducted to determine the affordances of AR and MR to develop empathy. Mobile AR could be a more affordable way of creating experiences to develop empathy because smartphones are, in general, cheaper than VR headsets and most people own smartphones that can be used to deploy mobile AR apps. Mobile AR can be used to situate experiences in the user’s context to develop empathy in certain physical contexts instead of recreating the entire context in VR.

Regarding the software used to develop immersive experiences to create empathy, most of the studies do not specify the software used and some other studies use general purpose commercial software. In that regard, there are no open source frameworks for designing and developing immersive experiences to create empathy. We call for more research to fill this gap in the literature so that the software allows to configure certain parameters to effectively create the experiences and save time in the development process.

Finally, the combination of VR/AR/MR technologies and artificial intelligence (AI) for training empathy is another field that deserves more research. The possibilities offered by generative AI might provide more personalized and adaptive experiences for empathy development and current research in this aspect is still in its infancy.

6.4 Limitations and future research directions

Finally, we found some limitations in the reviewed studies. First, the sample size was not significant, which affected the generalizability of the findings. In this sense, we suggest that future research consider larger research samples. Stavroulia & Lanitis [ 42 ] conducted a study with 69 participants. On the other hand, regarding the instructions given, some were not given adequately, guaranteeing that the participants understood and complied with them, causing confusion among the participants during the execution of the test. Additionally, in terms of descriptive studies, 9 studies were used (Table 7 ), which received negative criticism since the authors suggest that they reflected deficiencies such as the qualitative use of data, which prevented a comparison between theory and theory. the practice.

However, as a future recommendation, delving into qualitative studies is suggested since this methodology also has valuable theoretical support. According to Quecedo & Castaño [ 43 ], “a qualitative study allows us to know the personal aspect, inner life, perspectives, beliefs, concepts (…) successes and failures, moral struggle, efforts,” which are close and congruent traits when fostering empathy.

7 Limitations of this review

The main limitation of this study is that some papers might have been published in other bibliographic databases such as Web of Science and those papers were not included in this review. The categories considered in this review of literature are not unique. Other categories might be considered in the systematic literature review to obtain more information about the current state of research in the field of VR and AR to develop empathy.

8 Implications for education and training

In the realm of education, it is well-established that basic empathy is a trainable trait rather than an inherent quality [ 44 ]. This implies that individuals do not possess a predetermined amount of empathy at birth but instead develop and acquire this attribute over time. Consequently, the pivotal implication drawn from the review presented in this paper is that virtual reality (VR) emerges as an effective medium for training and fostering empathy across various educational levels, including primary, secondary, and higher education. VR’s unique capabilities allow for intricate and nuanced empathy training programs that surpass the possibilities offered by other technologies [ 45 ]. Such training programs hold significant potential for students, teachers, and society at large, fostering better relationships within the educational community, cultivating prosocial behaviors among students, broadening perspectives, promoting understanding of global inequalities, and contributing to conflict resolution and mediation.

While there exists an expanding body of literature on empathy training within healthcare, medicine, and related disciplines [ 46 ], a noticeable research gap is observed in the training of empathy within other educational domains such as psychology, marketing, and art. Consequently, future research endeavors should concentrate on investigating how empathy can be effectively trained in these diverse fields, exploring the unique variables that influence this trait within specific disciplines.

Within this review, the Empathy Index (IRI) emerged as the most employed instrument for measuring empathy. However, future studies in the realm of education and training should consider validating this instrument within educational contexts or developing new instruments tailored to educational settings. In accordance with Villalba et al.‘s [ 27 ] recommendations, a periodic revision of the IRI instrument is suggested to incorporate current discussions and advancements in empathy research. Additionally, the incorporation of physiological measures and eye-tracking technologies holds promise in offering a more objective assessment of the impact of VR on empathy.

Despite the progress in research on empathy development utilizing augmented reality (AR) and VR, a notable research gap persists in understanding how to effectively train empathy across different educational levels using these technologies. Thus, an additional implication derived from this review is the imperative need for further research dedicated to elucidating mechanisms, frameworks, and methodologies for empathy training across diverse educational levels.

9 Conclusions

This systematic review underscores the potential of VR and AR as effective tools for fostering empathy in various domains. It emphasizes the importance of larger sample sizes, validated questionnaires, and rigorous research designs to advance our understanding of the VR/AR-empathy relationship and shed light on the specific factors and personal traits that influence empathetic experiences in virtual environments. It is important to note that there is a lack of research on the use of AR or MR to develop empathy and this is a gap in the literature that requires more attention. This systematic review presents a comprehensive analysis of the relationship between VR/AR and empathy, yielding significant findings:

VR/AR immersion devices are not only safe but also effective and appealing to participants, corroborating previous research. These results confirm the broad application of VR and AR in fostering empathy, particularly in education and medicine. However, more research is needed to identify the real affordances of VR/AR to develop empathy.

To study empathy, it is crucial to employ multiple data collection instruments, with the IRI report emerging as the most relevant self-report measure. However, we suggest that future research can combine self-reported instruments with more objective measures such as physiological measurements to have more insights into the effect of AR/VR technologies on empathy.

The use of adapted questionnaires tailored to VR/AR enhances the understanding of the field and holds potential as information-gathering tools for social interventions. Future researchers should validate questionnaire adaptations in different languages or develop customized instruments to ensure the reliability and validity of the assessment.

Large sample sizes (over 100 participants) are vital in VR/AR-based empathy research to provide a quantitative perspective on participant distribution. Robust studies with larger sample sizes minimize biases, reduce errors, enhance precision, and facilitates generalizability of results.

Technological means for virtual immersion, such as 360° videos and VR headsets, have gained popularity in fostering empathy across diverse disciplines. While 360° videos offer cost-effective and realistic experiences, VR environments provide higher interaction possibilities. Furthermore, the noticeable absence of research on augmented reality (AR) and mixed reality (MR) for empathy development becomes apparent in the studies scrutinized. Consequently, it is imperative for future research endeavors to delve deeper into and explore the untapped potential of AR or MR experiences in promoting and enhancing empathy.

Approximately 45.95% of the studies reported statistically significant results regarding the effectiveness of VR or AR in fostering empathy. However, the field still lacks conclusive evidence, necessitating further research to gain a comprehensive understanding of the impact of VR/AR/MR on empathy.

VR and AR offer various advantages, including multisensory learning, cognitive improvement, content enrichment, and user-friendliness. This review adds to the body of knowledge by highlighting their effectiveness in developing empathy.

Experimental research designs have commonly been employed in VR/AR empathy studies to identify the affordances and benefits of these technologies. Pre-post tests in experimental designs enable the identification of changes in empathy. Future studies should utilize experimental designs and robust statistical methods, such as Structural Equation Modeling (SEM), to identify predictors of empathy during VR/AR experiences and explore the influential features and personal traits.

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Steuer J (1992) Defining virtual reality: dimensions determining telepresence. J Commun 42:73–93. https://doi.org/10.1111/j.1460-2466.1992.tb00812.x

Article   Google Scholar  

Sousa-Ferreira R, Campanari-Xavier RA, Rodrigues-Ancioto AS (2021) La realidad virtual como herramienta para la educación básica y profesional. Rev Científica Gen José María Córdova 19:223–241. https://doi.org/10.21830/19006586.728

Azuma RT (1997) A survey of augmented reality. Presence Teleoperators Virtual Environ 6:355–385. https://doi.org/10.1162/pres.1997.6.4.355

Wilding C, Young K, Cummins C et al (2023) Virtual reality to foster empathy in disability workers: a feasibility study during COVID-19. J Appl Res Intellect Disabil 36:132–142. https://doi.org/10.1111/jar.13042

Herrera F, Bailenson J, Weisz E et al (2018) Building long-term empathy: a large-scale comparison of traditional and virtual reality perspective-taking. PLoS ONE 13:e0204494. https://doi.org/10.1371/journal.pone.0204494

Billinghurst M (2014) Using augmented reality to create empathic experiences. In: Proceedings of the 19th International Conference on Intelligent User Interfaces. Association for Computing Machinery, New York, NY, USA, pp 5–6

Loon A, Bailenson J, Zaki J et al (2018) Virtual reality perspective-taking increases cognitive empathy for specific others. PLoS ONE 13:e0202442. https://doi.org/10.1371/journal.pone.0202442

Carrillo PO, Berges BM (2009) Evolución conceptual de la Empatía. Iniciación Investig

Ruiz A, Chaux E (2005) La formación de competencias ciudadanas

Decety J, Jackson PL (2004) The functional architecture of human empathy. Behav Cogn Neurosci Rev 3:71–100. https://doi.org/10.1177/1534582304267187

Gerry LJ, Billinghurst M, Broadbent E (2022) Empathic skills training in virtual reality: a scoping review. In: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp 227–232

Tassinari M, Aulbach M, Jasinskaja-Lahti I (2022) The use of virtual reality in studying prejudice and its reduction: a systematic review. PLoS ONE 17:e0270748. https://doi.org/10.1371/journal.pone.0270748

Christofi M, Michael-Grigoriou D (2017) Virtual reality for inducing empathy and reducing prejudice towards stigmatized groups: A survey. In: 2017 23rd International Conference on Virtual System & Multimedia (VSMM). IEEE, Dublin, pp 1–8

Ventura S, Badenes-Ribera L, Herrero R et al (2020) Virtual reality as a medium to elicit empathy: a meta-analysis. Cyberpsychol Behav Soc Netw 23:667–676. https://doi.org/10.1089/cyber.2019.0681

Martingano A, Hererra F, Konrath S (2021) Virtual reality improves emotional but not cognitive empathy: a meta-analysis. Technol Mind Behav 2. https://doi.org/10.1037/tmb0000034

Seinfeld S, Hasler B, Banakou D, Levy J (2022) Editorial: virtual reality and empathy. Front Psychol 13:1089006. https://doi.org/10.3389/fpsyg.2022.1089006

Lara F, Rueda J (2021) Virtual reality not for “Being Someone” but for “Being in Someone Else’s shoes”: avoiding misconceptions in Empathy Enhancement. Front Psychol 12:741516. https://doi.org/10.3389/fpsyg.2021.741516

Dhar E, Upadhyay U, Huang Y et al (2023) A scoping review to assess the effects of virtual reality in medical education and clinical care. Digit Health 9:20552076231158024. https://doi.org/10.1177/20552076231158022

Lee J, Kim H, Kim KH et al (2020) Effective virtual patient simulators for medical communication training: a systematic review. Med Educ 54:786–795. https://doi.org/10.1111/medu.14152

Foxman M, Markowitz DM, Davis DZ (2021) Defining empathy: interconnected discourses of virtual reality’s prosocial impact. New Media Soc 23:2167–2188. https://doi.org/10.1177/1461444821993120

Villalba É, San Martín A, Jacques-García F (2021) State of the art on immersive virtual reality and its use in developing meaningful empathy. Comput Electr Eng 93:107272. https://doi.org/10.1016/j.compeleceng.2021.107272

Botella J, Zamora Á (2017) El Meta-análisis: una metodología para la investigación en educación. Educ XX1 20:17–38

Page MJ, McKenzie JE, Bossuyt PM et al (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372:n71. https://doi.org/10.1136/bmj.n71

Fisher JA (2017) Empathic actualities: toward a taxonomy of Empathy in virtual reality. In: Nunes N, Oakley I, Nisi V (eds) Interactive storytelling. Springer International Publishing, Cham, pp 233–244

Chapter   Google Scholar  

Castaño R, Gonzalez F (2019) Realidad aumentada y virtual: valoraciones, percepciones y actitudes del alumnado universitario y su aplicación en el marco educativo. In: V Conferência Ibérica de Inovação na Educação com TIC (ieTIC2019). Insituto Politécnico de Bragança, Bragança, pp 2–16

Parra E, García A, Torres S et al (2022) Virtual reality stimulation and organizational neuroscience for the assessment of empathy. Front Psychol 13:1–15. https://doi.org/10.3389/fpsyg.2022.993162

Villalba EE (2022) Realidad virtual para propiciar El Desarrollo de la empatía de una forma significativa en estudiantes universitarios del. Tecnológico de Monterrey

Libera CD, Goosse M, Larøi F, Willems S (2023) Examining the impact of experiencing auditory verbal hallucinations from a first-person perspective on the degree of empathy and stigmatization in a group of psychology students: a study using 360° immersive videos. Compr Psychiatry 123:152379. https://doi.org/10.1016/j.comppsych.2023.152379

Peng X, Wu L, Xie X et al (2020) Impact of virtual Dementia Tour on empathy level of nursing students: a quasi-experimental study. Int J Nurs Sci 7:258–261. https://doi.org/10.1016/j.ijnss.2020.06.010

Aya Y, Quiroga D, Torres L (2020) Herramientas virtuales, empatía y razonamiento prosocial en jóvenes universitarios. Universidad de Ibague. https://repositorio.unibague.edu.co/bitstream/20.500.12313/2371/2/Trabajo%20de%20grado.pdf . Accessed 5 Nov 2023

Li B, Kyung Kim H (2021) Experiencing organ failure in virtual reality: effects of self- versus other-embodied perspective taking on empathy and prosocial outcomes. New Media Soc 23:2144–2166. https://doi.org/10.1177/1461444821993122

Sterne J, Savović J, Page M et al (2019) RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ l4898. https://doi.org/10.1136/bmj.l4898

Sora-Domenjó C (2022) Disrupting the empathy machine: the power and perils of virtual reality in addressing social issues. Front Psychol 13:814565. https://doi.org/10.3389/fpsyg.2022.814565

Cummings J, Tsay-Vogel M, Cahill T, Zhang L (2022) Effects of immersive storytelling on affective, cognitive, and associative empathy: the mediating role of presence. New Media Soc 24:2003–2026. https://doi.org/10.1177/1461444820986816

Mestre Escrivá V, Frías Navarro MD, Samper P (2004) La Medida De La empatía: análisis del interpersonal reactivity index. Psicothema 16:255–260

Google Scholar  

García-García JA, Reding-Bernal A, López-Alvarenga JC (2013) Cálculo Del tamaño de la muestra en investigación en educación médica. Investig En Educ Médica 2:217–224

Useche Rodríguez JC (2022) Desarrollo humano para valientes. aplicación de realidad virtual para apoyar la generación de empatía

Luque J (2020) Realidad virtual y realidad aumentada. ACTA - Revista Digital 63:1–19

Pimentel D, Kalyanaraman S, Lee Y-H, Halan S (2021) Voices of the unsung: the role of social presence and interactivity in building empathy in 360 video. New Media Soc 23:2230–2254. https://doi.org/10.1177/1461444821993124

Bertrand P, Guegan J, Robieux L et al (2018) Learning empathy through virtual reality: multiple strategies for training empathy-related abilities using body ownership illusions in embodied virtual reality. Front Robot AI. https://doi.org/10.3389/frobt.2018.00026

Schutte N, Stilinović E (2017) Facilitating empathy through virtual reality. Motiv Emot 41:708–712. https://doi.org/10.1007/s11031-017-9641-7

Stavroulia KE, Lanitis A (2023) The role of perspective-taking on empowering the empathetic behavior of educators in VR-based training sessions: an experimental evaluation. Comput Educ 197:104739. https://doi.org/10.1016/j.compedu.2023.104739

Quecedo R, Castaño C (2003) Introducción a la metodología de investigación cualitativa. Revista de Psicodidáctica 14:5–39

Hu S, Lai B (2022) Increasing empathy for children in dental students using virtual reality. Int J Paediatr Dent 32:793–800. https://doi.org/10.1111/ipd.12957

Vázquez J (n.d.) Realidad Virtual y desarrollo de competencias de reconocimiento y empatía contra la violencia de género. Real Reflexión 114–126. https://doi.org/10.5377/ryr.v51i0.9859

Gil K, Rhim J, Ha T et al (2014) AR Petite Theater: Augmented reality storybook for supporting children’s empathy behavior. In: 2014 IEEE International Symposium on Mixed and Augmented Reality - Media, Art, Social Science, Humanities and Design (ISMAR-MASH’D), pp 13–20

del Arbol L, Nielsen M (2022) Chapter 13. Immersion and empathy:the question of the adoptionof virtual reality. In: Rativa A, Bernal S, Escobar J, Lopez A, Navarro-Newball A, Postma M, Zaanen M (eds) XR academia: research and experiences in virtual reality, augmented reality, mixed reality, and artificial intelligence in latin America and Europe. Open Press TiU. https://doi.org/10.26116/6wmr-t534

Kizhevska E, Ferreira-Brito F, Guerreiro T, Luštrek M (n.d.) Using virtual reality to elicit empathy: a narrative review. In: Workshop on Virtual Reality for Health and Wellbeing, 21st International Conference on Mobile and Ubiquitous Multimedia. Lisbon, Portugal, pp 19–22

Banchoff C, Fava L, Martin E (2021) Realidad aumentada y realidad virtual aplicadas a proyectos con fines sociales. In: XXIII Workshop de Investigadores en Ciencias de la Computación (WICC 2021). Chilecito, La Rioja, pp 618–662

Archer D, Finger K (2018) Walking in another’s virtual shoes: Do 360-degree video news stories generate empathy in viewers? Columbia Journalism Review. https://www.cjr.org/tow_center_reports/virtual-reality-news-empathy.php . Accessed 24 Jan 2023

Seinfeld S, Hortensius R, Arroyo-Palacios J et al (2023) Domestic violence from a child perspective: impact of an immersive virtual reality experience on men with a history of intimate partner violent behavior. J Interpers Violence 38:2654–2682. https://doi.org/10.1177/08862605221106130

Kim JG, Kim T, Kim S-I et al (2022) The workplace playbook VR: exploring the design space of virtual reality to foster understanding of and support for autistic people. Proc ACM Hum-Comput Interact 6:1–24. https://doi.org/10.1145/3555082

Bard J, Chung H, Shaia J et al (2023) Increased medical student understanding of dementia through virtual embodiment. Gerontol Geriatr Educ 44:211–222. https://doi.org/10.1080/02701960.2022.2067850

Elzie C, Shaia J (2021) A pilot study of the impact of virtually embodying a patient with a terminal illness. Med Sci Educ 31:665–675. https://doi.org/10.1007/s40670-021-01243-9

Jiang Z, Meltzer A, Zhang X (2023) Using virtual reality to implement disability studies’ advocacy principles: uncovering the perspectives of people with disability. Disabil Soc 1–21. https://doi.org/10.1080/09687599.2022.2150601

Patterson T, Han I, Esposito L (2022) Virtual reality for the promotion of historical empathy: a mixed-methods analysis. Theory Res Soc Educ 50:553–580. https://doi.org/10.1080/00933104.2022.2118091

Wang Y, Chen C, Nelson M, Sar S (2022) Walk in my shoes: how perspective-taking and VR enhance telepresence and empathy in a public service announcement for people experiencing homelessness. New Media Soc 14614448221108108. https://doi.org/10.1177/14614448221108108

Galvão D, Biazzi R, de Morais G, Miranda C (2022) Women in science: a Brazilian experience through immersive narratives in 360° videos. Int J Sci Educ Part B 12:309–327. https://doi.org/10.1080/21548455.2022.2076293

Hasler B, Landau D, Hasson Y et al (2021) Virtual reality-based conflict resolution: the impact of immersive 360° video on changing view points and moral judgment in the context of violent intergroup conflict. New Media Soc 23:2255–2278. https://doi.org/10.1177/1461444821993133

Bacca-Acosta J, Avila-Garzon C, Sierra-Puentes M (2023) Insights into the predictors of empathy in virtual reality environments. Information 14: 465. https://doi.org/10.3390/info14080465

Marques A, Gomes P, Araújo M et al (2022) Impact of a virtual reality-based simulation on empathy and attitudes toward Schizophrenia. Front Psychol 13:1–11. https://doi.org/10.3389/fpsyg.2022.814984

Tassinari M, Aulbach M, Jasinskaja-Lahti I (2022) Investigating the influence of intergroup contact in virtual reality on empathy: an exploratory study using AltspaceVR. Front Psychol 12:1–13. https://doi.org/10.3389/fpsyg.2021.815497

Zamin N, Khairuddin F, Rambli D et al (2018) Building empathy in young children using augmented reality: a case study in Malaysia. In: Emerging Technologies for Health and Medicine. Wiley, pp 35–41

Çakiroğlu Ü, Aydin M, Köroğlu Y, Ayvaz Kina M (2023) Looking past seeing present: teaching historical empathy skills via augmented reality. Interact Learn Environ 0:1–13. https://doi.org/10.1080/10494820.2023.2174142

Jones S, Dawkins S (2018) Walking in someone else’s shoes: creating empathy in the practice of immersive film. Media Pract Educ 19:298–312. https://doi.org/10.1080/25741136.2018.1520538

López-Faican L, Jaen J (2023) Design and evaluation of an augmented reality cyberphysical game for the development of empathic abilities. Int J Hum-Comput Stud 176:103041. https://doi.org/10.1016/j.ijhcs.2023.103041

Download references

Open Access funding provided by Colombia Consortium. This research was funded by COMISIÓN DE REGULACIÓN DE COMUNICACIONES - CRC and MINISTERIO DE CIENCIA, TECNOLOGIA E INNOVACIÓN from Colombia. This article belongs to the products of the research project entitled “Co-creación de narrativas inmersivas sobre migración en Colombia: una propuesta metodológica” presented and approved under the research call 908 from MINCIENCIAS “Nuevo conocimiento, desarrollo tecnológico e innovación para el fortalecimiento de los sectores de TIC, postal y de contenidos audiovisuales”. This study was developed by researchers from Fundación Universitaria Konrad Lorenz.

Author information

Authors and affiliations.

Faculty of Psychology, Fundación Universitaria Konrad Lorenz, Bogotá, 110231, Colombia

Jose Lacle-Melendez & Sofia Silva-Medina

Faculty of Mathematics and Engineering, Fundación Universitaria Konrad Lorenz, Bogotá, 110231, Colombia

Jorge Bacca-Acosta

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Jose Ignacio Lacle-Melendez and Sofia Vanesa Silva-Medina. The first draft of the manuscript was written by Jose Ignacio Lacle-Melendez and Sofia Vanesa Silva-Medina. Jorge Bacca-Acosta commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Jorge Bacca-Acosta .

Ethics declarations

Conflicts of interest/competing interests.

There is no conflict of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Lacle-Melendez, J., Silva-Medina, S. & Bacca-Acosta, J. Virtual and augmented reality to develop empathy: a systematic literature review. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-19191-y

Download citation

Received : 21 July 2023

Revised : 17 February 2024

Accepted : 02 April 2024

Published : 04 May 2024

DOI : https://doi.org/10.1007/s11042-024-19191-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Virtual reality
• Augmented reality
• Systematic literature review

Advertisement

• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 01 May 2024

Hospital performance evaluation indicators: a scoping review

• Shirin Alsadat Hadian   ORCID: orcid.org/0000-0002-1443-1990 1 ,
• Reza Rezayatmand   ORCID: orcid.org/0000-0002-9907-3597 2 ,
• Nasrin Shaarbafchizadeh   ORCID: orcid.org/0000-0001-7104-2214 3 ,
• Saeedeh Ketabi   ORCID: orcid.org/0000-0002-6778-5645 4 &
• Ahmad Reza Pourghaderi   ORCID: orcid.org/0000-0003-2682-2160 5  

BMC Health Services Research volume  24 , Article number:  561 ( 2024 ) Cite this article

101 Accesses

Metrics details

Hospitals are the biggest consumers of health system budgets and hence measuring hospital performance by quantitative or qualitative accessible and reliable indicators is crucial. This review aimed to categorize and present a set of indicators for evaluating overall hospital performance.

We conducted a literature search across three databases, i.e., PubMed, Scopus, and Web of Science, using possible keyword combinations. We included studies that explored hospital performance evaluation indicators from different dimensions.

We included 91 English language studies published in the past 10 years. In total, 1161 indicators were extracted from the included studies. We classified the extracted indicators into 3 categories, 14 subcategories, 21 performance dimensions, and 110 main indicators. Finally, we presented a comprehensive set of indicators with regard to different performance dimensions and classified them based on what they indicate in the production process, i.e., input, process, output, outcome and impact.

The findings provide a comprehensive set of indicators at different levels that can be used for hospital performance evaluation. Future studies can be conducted to validate and apply these indicators in different contexts. It seems that, depending on the specific conditions of each country, an appropriate set of indicators can be selected from this comprehensive list of indicators for use in the performance evaluation of hospitals in different settings.

Peer Review reports

Healthcare is complex [ 1 ] and a key sector [ 2 ] that is now globally faced with problems of rising costs, lack of service efficiency, competition, and equity as well as responsiveness to users [ 3 ]. One estimate by the WHO has shown a yearly waste of approximately 20–40% of total healthcare resources because of inefficiency [ 4 ]. European countries have spent on average 9.6% of their gross domestic product (GDP) on healthcare in 2017 and 9.92% in 2019. Germany, France, and Sweden reported the highest healthcare expenditures in Europe in 2018 (between 10.9% and 11.5% of GDP) [ 5 ]. In the U.S., healthcare spending consumes 18% of the GDP, which is likely to eclipse $6 trillion by 2027 [ 6 ].

Hospitals, as the biggest consumers of health system budgets [ 7 ], are the major part of the health system [ 8 ]. In many countries 50–80% of the health sector budget is dedicated to hospitals [ 8 , 9 ]. As a result, hospital performance analysis is becoming a routine task for every hospital manager. On the one hand, hospital managers worldwide are faced with difficult decisions regarding cost reduction, increasing service efficiency, and equity [ 10 ]. On the other hand, measuring hospital efficiency is an issue of interest among researchers because patients demand high-quality care at lower expenses [ 11 ].

To address the above mentioned need to measure hospital performance, implementing an appropriate hospital performance evaluation system is crucial in any hospital. In doing so, hospital administrators use various tools to analyse and monitor hospital activities [ 1 ], which need well-defined objectives, standards and quantitative indicators [ 12 ]. The latter are used to evaluate care provided to patients both quantitatively and qualitatively and are often related to input, output, processes, and outcomes. These indicators can be used for continuous quality improvement by monitoring, benchmarking, and prioritizing activities [ 13 ]. These parameters are developed to improve health outcomes and to provide comparative information for monitoring and managing and formulating policy objectives within and across health services [ 12 ]. Studies thus far have used their own set of indicators while evaluating hospital performance, which could be context dependent. In addition, those studies have mostly used a limited set of indicators that focus on few dimensions (2–6 dimensions) of hospital performance [ 14 , 15 , 16 , 17 , 18 ].

Therefore, comprehensive knowledge of potential indicators that can be used for hospital performance evaluation is necessary. It would help choose appropriate indicators when evaluating hospital performance in different contexts. It would also help researchers extend the range of analysis to evaluate performance from a wider perspective by considering more dimensions of performance. Although performance is a very commonly used term, it has several definitions [ 19 , 20 ], yet, it is often misunderstood [ 21 ]. Therefore, some researchers have expressed confusion about the related terms and considered them interchangeable. These terms are effectiveness, efficiency, productivity, quality, flexibility, creativity, sustainability, evaluation, and piloting [ 21 , 22 , 23 ]. Thus, this scoping review aimed to categorize and present a comprehensive set of indicators that can be used as a suitable set for hospital performance evaluation at any needed level of analysis, i.e., clinical, para-clinical, logistical, or departmental, and relate those indicators to the appropriate performance dimensions. The uniqueness of this paper is that it provides its readers with a comprehensive collection of indicators that have been used in different performance analysis studies.

Materials and methods

We conducted a scoping review of a body of literature. The scoping review can be of particular use when the topic has not yet been extensively reviewed or has a complex or heterogeneous nature. This type of review is commonly undertaken to examine the extent, range, and nature of research activity in a topic area; determine the value and potential scope and cost of undertaking a full systematic review; summarize and disseminate research findings; and identify research gaps in the existing literature. As a scoping review provides a rigorous and transparent method for mapping areas of research, it can be used as a standalone project or as a preliminary step to a systematic review [ 24 ]. While a systematic review (qualitative or quantitative) usually addresses a narrow topic/scope and is a method for integrating or comparing findings from previous studies [ 25 ].

In our study, we used the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist following the methods outlined by Arksey and O’Malley [ 26 ] and Tricco [ 27 ]. A systematic search for published and English-language literature on hospital performance evaluation models was conducted, using three databases, i.e., PubMed, Scopus, and Web of Science, from 2013 to January 2023. Initially, the identified keywords were refined and validated by a team of experts. Then, a combination of vocabularies was identified by the authors through a brainstorming process. The search strategy was formulated using Boolean operators. The title and abstract of the formulas were searched in the online databases. The search query for each database is presented in Table  1 .

In the screening process, relevant references related to hospital performance evaluation were screened and abstracted into researcher-developed Microsoft® Excel forms by dual independent reviewers and conflicting information was provided by other reviewers.

The inclusion criteria were as follows: focused only on the hospital setting, available full text and written in English. We excluded studies that focused on health organization indicators, not specifically on hospital indicators; articles without appropriate data (only focused on models and not indicators; or qualitative checklist questionnaires); and articles that focused only on clinical or disease-related indicators, not hospital performance dimensions, and provided very general items as indicators, not the domains of the indicators themselves. Then, a PRISMA-ScR Checklist was used to improve transparency in our review [ 28 ].

To extract the data, researcher-developed Microsoft® Excel forms (data tables) were designed. The following data were subsequently extracted into Microsoft®Excel for synthesis and evaluation: title, author, article year, country, indicator category, study environment (number of hospitals studied), study time frame, indicator name, number of indicators, indicator level (hospital level, department level), evaluation perspective (performance, productivity, efficiency, effectiveness, quality, cost, safety, satisfaction, etc. ) , study type (quantitative or qualitative), indicator subtype (input (structure), process, output (result), outcome and impact), and other explanations. To create a descriptive summary of the results that address the objectives of this scoping review, numerical summarization was also used.

The purpose of creating the main category and the evaluation perspective section was to develop them and create new categories, which focused on the type of indicators related to the performance term. For example, in the “Category” section, the names of the departments or wards of the hospital (such as hospital laboratories, pharmacies, clinical departments, and warehouses) and in the “Evaluation perspective” section, various terms related to the evaluation of hospital performance were extracted. These two types were used after extracting their information under the title “performance dimension”.

The indicators’ levels were collected to determine the level of performance evaluation with the relevant index. Some indicators were used to evaluate the performance of the entire hospital, some were used to evaluate the performance of hospital departments, and some were used to evaluate the performance at the level of a specific project. For example, several indicators (such as bed occupancy ratio, length of stay, and waiting time) were used to evaluate the performance of the entire hospital, and other indicators (such as laboratory department indicators, energy consumption indicators, and neonatal department indicators) were used only to measure the performance of specific departments. This sections were used under the title “category”. The “category” and “indicator’s name” sections were defined according to the results of the “subcategory” section.

The subtypes of indicators (input (structure), process, output(result), outcome and impact) were defined based on the chain model, and each of the selected indicators was linked to it (Appendix 1 ). As a result of the chain model, inputs were used to carry out activities, activities led to the delivery of services or products (outputs). The outputs started to bring about change (outcomes), and eventually, this (hopefully) contributed to the impact [ 29 ]. The classification of the set of input, process, output, outcome and impact indicators was such that readers could access these categories if necessary according to their chosen evaluation models. The term was used under the title “Indicators by types”.

The type of study was considered quantitative or qualitative for determining whether an indicator was able to perform calculations. In this way, readers can choose articles that use quantitative or qualitative indicators to evaluate hospital performance.

We included 91 full-text studies (out of 7475) in English published between 2013 and January 2023 (Fig.  1 ), approximately 40% of which were published between 2020 and 2023. More than 20% of the retrieved studies were conducted in Iran and USA.

Study selection and data abstraction

Study characteristic

As shown in Table  2 , in 85% of the reviewed studies, a number of hospitals (1 to 3828 hospitals, 13,221 hospitals in total) were evaluated. More than 90% of the studies used a quantitative approach. In more than 70% of the studies, hospital evaluation occurred at the department level, which can also be divided into three levels: administrative, clinical ward, and paramedical department. In addition, the administrative departments consist of 13 departments, including financial management [ 48 , 55 , 61 , 67 , 68 , 80 , 83 , 109 , 113 ], supply chain management and warehouse [ 15 , 43 , 84 ], value-based purchasing [ 33 , 85 ], human resource management [ 97 , 101 ], medical equipment [ 32 , 87 ], health information management department [ 90 ], information systems [ 106 ], nutritional assessment [ 93 ], energy management [ 30 , 45 , 92 ], facility management [ 52 , 53 ], building sustainability and resilience [ 35 ], research activities [ 44 ], and education [ 107 ].

The clinical wards consisted of 8 wards, namely, emergency departments (EDs) [ 16 , 39 , 56 , 57 , 69 , 70 , 89 ], surgery departments [ 58 , 62 , 63 , 91 , 102 ], intensive care units (ICUs) [ 47 , 64 , 65 ], operating rooms (ORs) [ 38 , 88 , 108 ], surgical intensive care units (SICUs) [ 111 ], obstetrics and gynecology department [ 59 ], neonatal intensive care units (NICUs) [ 74 , 103 ] and quality of care [ 18 , 31 , 40 , 50 , 72 , 92 , 95 , 112 ] indicators. The paramedical departments consisted of 3 departments, pharmacy [ 60 , 76 , 98 ], laboratory and blood bank [ 37 , 42 , 43 , 49 ], and outpatient assessment [ 86 ] indicators.

With regard to data categorization, firstly, a total of 1204 indicators in 91 studies were extracted and after detailed examination, 43 indices (such as hospital ownership, level of care, admission process, and personal discipline) were removed due to their generality and impossibility of calculation in the hospital environment. Then, 1161 performance indicators were entered in this research and were categorized based on the performance criteria (more details about the indicators can be found in Appendix 1 ). Secondly, 145 functional dimensions, including divisions based on different departments and units of the hospital, were defined according to several focus group discussions with 5 health experts. Then, re-categorization and functional summarization were performed, after which 21 performance dimensions were finalized.

As shown in Table  4 , the 21 performance dimensions were divided into three parts: category, subcategory, and related indicators. Additionally, according to the hospital levels, there were three categories: ‘organizational management’, ‘clinical management’, and ‘administrative management’. Then, according to the type of indicators, fifteen subcategories were defined for the 110 selected main indicators.

Performance dimensions

The ‘productivity’ dimension focuses on indicators reflecting the macro-performance of the hospital, considering that this index is more effective and efficient. The ‘efficiency’ dimension focuses on general performance indicators for the optimal use of resources to create optimal output in the hospital. The ‘effectiveness’ dimension is a general performance indicator with an outcome view. The ‘speed’ dimension focuses on the indicators that show attention to the service delivery time and the speed of the procedures. The ‘development’ dimension focuses on matters related to employees’ and students’ training and related training courses. In terms of ‘safety’ dimension, there were issues related to patient safety, unwanted and harmful events, and hospital infections.

The “quality of work life” dimension emphasizes matters related to personnel volume and work conditions. The ‘quality’ dimension is related to the quality of service provided in different parts of the hospital and possible complications in improving the quality of services. The ‘satisfaction’ dimension focuses on the satisfaction of patients, employees, and their complaints. The ‘innovation’ dimension relates to the research process and its output. The ‘appropriateness’ dimension involves proper service from clinical departments, pharmaceutical services, and patient treatment. The ‘evaluation’ dimension focuses on the indicators related to the assessment scores of the para-clinical departments of the hospital.

The ‘profitability’ dimension focuses on the overall output indicators for income and profitability. The ‘cost’ dimension focuses on indicators related to general expenditures and the average cost per bed and patient and budgeting. The ‘economy’ dimension is related to financial rates and their indicators. The ‘coherence’ dimension emphasizes the indicators related to the continuity of the service delivery process. The ‘patient-centeredness’ dimension focuses on the indicators related to the patient’s experience of the facility, environment, treatment processes, communications, and relevant support for the patient. The ‘equity’ dimension studies indicators related to social and financial justice and life expectancy. The ‘relationship’ dimension evaluates the process of consultations and discussions required during the patients’ care provided by the treatment team. The ‘sustainability’ dimension focuses on indicators related to energy standards. The ‘flexibility’ dimension focuses on the hospital’s response to the crisis.

According to Table  4 , most studies focused on ‘efficiency’, ‘productivity’, ‘safety’ and ‘effectiveness’ as performance dimensions in 54, 53, 38 and 37 studies, respectively (40–70% of studies). In the ‘efficiency’ subcategory, resource management, supportive unit assessment, and human resource management indicators were the first to third most common indicators used in 26, 23 and 22 studies, respectively (approximately 25% of the studies).

In addition, for the ‘efficiency’ dimension, ‘medical staff numbers’, ‘emergency department bed numbers’, and ‘nonmedical staff numbers’ were reported in 16, 13, and 11 studies, respectively (between 20 and 30% of the studies). For the ‘productivity’ subcategory, ‘bed utilization rate’ and ‘service delivery and treatment’ were reported in 50% and 20% of the studies, respectively (46 and 19 out of 91).

Additionally, for the ‘productivity’ dimension, the ‘length of stay’ indicator was used more than others and reported in approximately 80% of the studies (43 out of 53), followed by the ‘bed occupancy rate’ in approximately 40% of the studies (21 out of 53). The ‘bed turnover ratio’ and ‘hospitalization rate’ were also reported in 12 studies. Furthermore, for ‘safety’ dimensions, all indicators were in the ‘patient safety’ subcategory, which has been reported in 38 studies, and ‘complications’, ‘accidents or adverse events’, and ‘incidents or errors rates’ were the most concentrated indicators by researchers in 13, 12, and 11 studies, respectively. The performance dimension of ‘effectiveness’ was presented in 37 studies (40%), with only two indicators, ‘mortality rate’ in 29 studies and ‘readmission rate’ in 23 studies.

Performance categories

Considering the three categories shown in Table  4 , ‘organizational management’ indicators were more commonly used among the other two categories (‘clinical’ and ‘administrative’) and were present in more than 85% of the studies (78 out of 91). Two categories, ‘clinical management’ and ‘administrative management’, were reported in 62 and 51 studies, respectively.

Performance subcategories

Considering the 14 subcategories shown in Table  4 , both the ‘bed utilization rate’ and ‘patient safety’ indicators were mentioned in 46 studies and were more common among the other subcategories. The second most common indicator of the ‘financial management’ subcategory was reported in 38 studies. At the third level, both the ‘human resource management’ and ‘time management’ indicators were presented in 31 studies. The ‘paramedical’ subcategory indicators were presented in less than 10% of the studies [ 60 , 96 , 97 , 98 , 106 , 113 ].

Performance indicators

According to the indicator columns in Table  3 , the most used indicators in reviewed studies were the length of stay, mortality rate, and readmission rate in 47%, 32%, and 25% of studies, respectively. Bed occupancy rate and non-personnel costs were reported in 23% of studies. Additionally, among the 110 indicators, 16 indicators, namely, the lab cancellation rate, exam-physician ratios, number of coded diagnoses, number of medical records, laboratory sample/report intervals, medical information request time, safety standards in the archives, nutritional risk screening, imaging quality control failures, errors in medical reports, average impact factor, nutritional measures, laboratory scoring, imaging inspection, discharge process and emergency response rate, were reported in less than 1% of the studies.

The classification of the indicators in Table  4 was performed based on the chain model, which included the input, process, output, outcome and impact. The assignment of the indicators to each category was performed according to the experts’ opinions. For instance, the number of publications by academic member of an academic hospital and the average impact factor of those publications were considered outcome indicators. As depicted in the Table  4 , most studies (80%) focused more on output indicators. Additionally, fifteen studies focused on introducing and extracting some of the input, process, output, outcome and impact indicators; among those, only one study [ 96 ] has examined the input, process, output and impact indicators simultaneously.

Additionally, in approximately 42% (36 out of 91) of the studies, the indicators’ definitions, formulas, or descriptions have been illustrated, while less than 10% of the studies have defined measuring units, standard or benchmark units for all studied indicators [ 15 , 43 , 45 , 51 , 52 , 57 , 67 ].

Overall, nine studies related to hospital performance evaluation were conducted using systematic review methodologies (five systematic reviews [ 16 , 29 , 30 , 56 , 113 ], two literature reviews [ 79 , 80 ], one narrative review [ 98 ] and one brief review [ 92 ]). Most of these studies focused on extracting performance indicators from one or more hospital departments (e.g., the emergency department) [ 16 , 56 ], hospital laboratory and radiology information systems [ 106 ], supply chain performance [ 29 ], resources and financial results and activity [ 113 ], hospital water consumption [ 30 ], and the pharmaceutical sector [ 98 ]. Other reviews included a three-step process to review, evaluate and rank these hospital indicators in a systematic approach [ 16 ], or to evaluate performance indicator models to create an interactive network and visualize the causal relationships between performance indicators [ 79 ]; moreover, some have focused on the importance of indicators to ensure adequate coverage of the relevant areas of health care services to be evaluated [ 92 ].

Only one scoping review aimed to identify current assessments of hospital performance and compared quality measures from each method in the context of the six qualitative domains of STEEEP (safety, timeliness, effectiveness, efficiency, equity, and patient-centeredness) of the Institute of Medicine (IOM) in accordance with Donabedian’s framework and formulating policy recommendations [ 115 ].

In addition, 21 studies divided performance indicators into 2 to 6 dimensions of performance. Also, the reviewed studies included 2–40 indicators in zero [ 29 , 30 , 98 ] to 6 domains [ 34 ]. Moreover, none of the studies have tried to comprehensively summarize and categorize the performance indicators in several categories, focusing on all the indicators reflecting the performance of the entire hospital organization, or the indicators of administrative units or clinical departments.

In this scoping review, a unique set of hospital performance evaluation indicators related to the various performance dimensions was categorized from 91 studies over the past ten years.

Similarly, in a study, 19 performance dimensions, 32 sub-dimensions, and 138 indicators were extracted from only six studies. Those dimensions were described by all studies included in the review, but only three studies specified the relevant indicators, and the list provided for all possible indicators was not comprehensive. Also, despite current review, there was no classification of indicators based on the hospital levels: managerial, clinical, or organizational levels [ 116 ]. Another study has similarly investigated the performance evaluation indicators of the hospital in such a way that among 42 studies, 111 indicators were presented in the four categories: input, output, outcome, and impact. But, there was no classification of indicators based on performance dimensions and hospital levels [ 117 ].

In this study, the importance of categorized indicators, for the first time to our knowledge, was determined based on their frequency of use in the published literature (Appendix 2 ). The ‘Organizational management’ indicators were the most common compared with the other two categories (‘clinical’ and ‘administrative’). It could be because of the fact that the indicators such as ‘bed occupancy rate’, ‘average length of stay’, ‘mortality rate’, ‘hospital infection rate’, and ‘patient safety’ are easier to be registered in hospital software compared to other indicators, and also they better reflect the overall performance of hospital. Thus, researchers are more interested in using these indicators.

Considering 14 subcategories, indicators related to three subcategories i.e. bed utilization, patient safety and financial management are the most frequent used indicators for hospital performance evaluation. It reflects the need of hospital managers to increase the profitability of hospital in one hand, and to control cost on the other hand. As a results, researchers have paid special attention to ‘cost income’, ‘profitability’, ‘economic’, etc., as indicators for evaluating hospital performance.

When considering indicators by type, more studies have focused on output indicators, while input indicators were the least common used. This might be because of the fact that at hospital level, it is difficult for managers to change those inputs such as ‘beds’, ‘human resources’, ‘equipment and facilities’. In addition, due to the complexity of interdepartmental relationships in hospitals, process indicators seemed to provide more variety for analysis than input indicators, so they were more often used. As mentioned above, output indicators were the most used indicators for hospital performance evaluation due to their ease of calculation and interpretation.

The main purpose of this paper was to identify a comprehensive set of indicators that can be used to evaluate hospital performance in various hospital settings by being distilled into a smaller and more related set of indicators for every hospital or department setting. future studies could be designed to validate each set of indicators in any specific context. In addition, they could investigate the relationship between the indicators and their outcomes of interest and the performance dimension each could address. This will enable hospital managers to build their own set of indicators for performance evaluation both at organization or at department level. Also it should be mentioned that.

Although some previous studies have provided definitions for each indicator and determined the standard criteria for them, this was not done in this study because the focus of this study was to provide a collection of all the indicators used in hospital performance evaluation, which resulted in the identification of more than a thousand indicators without limiting to specific country or context. So while preparing a smaller set of indicators, specific conditions of each country, such as the type of health system and its policy, the type of financing system, and the structure of services, should be taken into account to select appropriate indicators.

In addition, although it is important to examine the scope of each article to compare the list of indicators and the relationships between the dimensions of the hospital in terms of size and type and between the number and type of selected indicators, this was considered beyond the scope of this review due to the high number of indicators, which made the abovementioned investigations impossible. Future studies could do that while working with a smaller set of indicators.

This review aimed to categorize and present a comprehensive set of indicators for evaluating overall hospital performance in a systematic way. 1161 hospital performance indicators were drawn from 91 studies over the past ten years. They then were summarized into 110 main indicators, and categorized into three categories: 14 subcategories, and 21 performance dimensions This scoping review also highlighted the most frequent used indicators in performance evaluation studies which could reflect their importance for that purpose. The results of this review help hospital managers to build their own set of indicators for performance evaluation both at organization or at department level with regard to various performance dimensions.

As the results of this review was not limited to any specific country or context, specific conditions of each country, such as the type of health system and its policy, the type of financing system, and the structure of services, should be taken into account while selecting appropriate indicators as a smaller set of indicators for hospital performance evaluation in specific context.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Gross domestic product

Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews

Emergency departments

Intensive care unit

Operating room

Surgical intensive care unit

Neonatal intensive care unit

Readmission rate

Quality Control

Medication use evaluation

safety, timeliness, effectiveness, efficiency, equity, and patient-centeredness

Institute of Medicine

Abdullah A, Ahmad S, Athar MA, Rajpoot N, Talib F. Healthcare performance management using integrated FUCOM-MARCOS approach: the case of India. Int J Health Plann Manage. 2022;37(5):2635–68.

Article   PubMed   Google Scholar  

Pestana M, Pereira R, Moro S. Improving health care management in hospitals through a productivity dashboard. J Med Syst. 2020;44(4):87.

Amos D. A practical framework for performance measurement of facilities management services in developing countries’ public hospitals. J Facil Manag. 2022;20(5):713–31.

Article   Google Scholar  

Ahmed S, Hasan MZ, MacLennan M, Dorin F, Ahmed MW, Hasan MM, et al. Measuring the efficiency of health systems in Asia: a data envelopment analysis. BMJ Open. 2019;9(3):e022155.

Article   PubMed   PubMed Central   Google Scholar  

Mitkova Z, Doneva M, Gerasimov N, Tachkov K, Dimitrova M, Kamusheva M, et al. Analysis of healthcare expenditures in Bulgaria. Healthc. 2022;10(2):274.

Patrinely JR, Walker SH, Glassman GE, Davis MJ, Abu-Ghname A, Khan U, et al. The importance of financial metrics in physician funding and performance evaluation. Plast Reconstr Surg. 2021;147:1213–8.

Article   CAS   PubMed   Google Scholar  

Buathong S, Bangchokdee S. The use of the performance measures in Thai public hospitals. ASIAN Rev Acc. 2017;25(4):472–85.

Google Scholar  

Imani A, Alibabayee R, Golestani M, Dalal K. Key indicators affecting hospital efficiency: a systematic review. Front Public Heal. 2022;10:830102.

Mahdiyan S, Dehghani A, Tafti AD, Pakdaman M, Askari R. Hospitals’ efficiency in Iran: a systematic review and meta-analysis. J Educ Health Promot. 2019;8(1):126.

PubMed   PubMed Central   Google Scholar  

Amos D, Musa ZN, Au-Yong CP. Performance measurement of facilities management services in Ghana’s public hospitals. Build Res Inf. 2020;48(2):218–38.

Feibert DC, Andersen B, Jacobsen P. Benchmarking healthcare logistics processes–a comparative case study of Danish and US hospitals. Total Qual Manag Bus Excell. 2019;30(1–2):108–34.

Gün I, Yilmaz F, Şenel IK. Efficiency analysis of health systems in world bank countries. Arch Heal Sci Res. 2021;8(2):147–52.

Breyer JZ, Giacomazzi J, Kuhmmer R, Lima KM, Hammes LS, Ribeiro RA, et al. Hospital quality indicators: a systematic review. Int J Health Care Qual Assur. 2019;32(2):474–87.

Regragui H, Sefiani N, Azzouzi H. Improving performance through measurement: the application of BSC and AHP in healthcare organization. In: Equipe De Recherche, Ingénierie, Innovation Et Management Des Systèmes Industriels, Université Abdelmalek Saadi. Tanger, Morocco: Institute of Electrical and Electronics Engineers Inc; 2018. p. 51–6.

Ghozali MT, Latifah DN, Darayani A. Analysis of Drug Supply Management of the Pharmacy Warehouse of Prof. Dr. Soerojo Mental Health Hospital, Magelang, Indonesia. Clin Schizophr Relat Psychoses. 2021;15:1–6.

Etu EE, Monplaisir L, Aguwa C, Arslanturk S, Masoud S, Markevych I, et al. Identifying indicators influencing emergency department performance during a medical surge: a consensus-based modified fuzzy Delphi approach. PLoS ONE. 2022;17(4 April):e0265101.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Lin C-Y, Shih F-C, Ho Y-H. Applying the balanced scorecard to build service performance measurements of medical institutions: An AHP-DEMATEL approach. Int J Environ Res Public Health. 2023;20(2):1022.

Backman C, Vanderloo S, Forster AJ. Measuring and improving quality in university hospitals in Canada: the collaborative for excellence in healthcare quality. Health Policy (New York). 2016;120(9):982–6.

Ghalem Â, Okar C, Chroqui R, Semma E. Performance: A concept to define. In: Performance: A concept to define. LOGISTIQUA 2016; 2016. p. 1–13.

Sonnentag S, Frese M. Performance Concepts and Performance Theory. In 2005. p. 1–25.

Tangen S. Demystifying productivity and performance. Int J Prod Perform Manag. 2005;54:34–46.

Elena-Iuliana I, Maria C. Organizational Performance – A Concept That Self-Seeks To Find Itself. Ann - Econ Ser Constantin Brancusi Univ Fac Econ. 2016;4(4):179–83.

Riratanaphong C, Van der Voordt T, Sarasoja A. Performance Measurement in the context of CREM and FM. In: Per Anker Jensen, Theo Van der Voordt CC, editor. The added value of facilities management: concepts, findings and perspectives. Lyngby Denmark: Polyteknisk Forlag; 2012. p. 1–21.

Pham M, Rajić A, Greig J, Sargeant J, Papadopoulos A, Mcewen S. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods. 2014;5:371–85.

Chaney M. So you want to write a narrative review article? J Cardiothorac Vasc Anesth. 2021;35:3045–9.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Tricco A, Lillie E, Zarin W, O’Brien K, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Dolatabad AH, Mahdiraji HA, Babgohari AZ, Garza-Reyes JA, Ai A. Analyzing the key performance indicators of circular supply chains by hybrid fuzzy cognitive mapping and Fuzzy DEMATEL: evidence from healthcare sector. Environ Dev Sustain. 2022;1–27.

Batista KJM, da Silva SR, Rabbani ERK, Zlatar T. Systematic review of indicators for the assessment of water consumption rates at hospitals. Water Supply. 2020;20(2):373–82.

Beta G, Role D, Berloviene D, Balkena Z. PATIENTS’ SATISFICATION AS THE QUALITY INDICATOR OF NURSING. In: Lubkina V, Kaupuzs A, Znotina D, editors. SOCIETY INTEGRATION EDUCATION, VOL VI: PUBLIC HEALTH AND SPORT, RESEARCHES IN ECONOMICS AND MANAGEMENT FOR SUSTAINABLE EDUCATION. 2020. p. 79–88.

Bhardwaj P, Joshi NK, Singh P, Suthar P, Joshi V, Jain YK, et al. Competence-based assessment of biomedical equipment management and maintenance system (e-Upkaran) using benefit evaluation framework. CUREUS J Med Sci. 2022;14(10):e30579.

Cheon O, Song M, Mccrea AM, Meier KJ. Health care in America: the relationship between subjective and objective assessments of hospitals. Int PUBLIC Manag J. 2021;24(5):596–622.

Craig KJT, McKillop MM, Huang HT, George J, Punwani ES, Rhee KB. US hospital performance methodologies: a scoping review to identify opportunities for crossing the quality chasm. BMC Health Serv Res. 2020;20(1):640.

Cristiano S, Ulgiati S, Gonella F. Systemic sustainability and resilience assessment of health systems, addressing global societal priorities: Learnings from a top nonprofit hospital in a bioclimatic building in Africa. Renew Sustain ENERGY Rev. 2021;141:110765.

Dadi D, Introna V, Santolamazza A, Salvio M, Martini C, Pastura T, et al. Private Hospital Energy Performance Benchmarking Using Energy Audit Data: An Italian Case Study. Energies. 2022;15(3):1–16.

Dawande PP, Wankhade RS, Akhtar FI, Noman O. Turnaround time: an efficacy measure for medical laboratories. CUREUS J Med Sci. 2022;14(9):e28824.

De Sousa LR, Mazzo A, De Almeida ACF, Tonello C, Lourençone LFM. Evaluation of quality indicators in the management of an operating room at a tertiary-level hospital. Med. 2022;55(1):1–8.

Drynda S, Schindler W, Slagman A, Pollmanns J, Horenkamp-Sonntag D, Schirrmeister W, et al. Evaluation of outcome relevance of quality indicators in the emergency department (ENQuIRE): study protocol for a prospective multicentre cohort study. BMJ Open. 2020;10(9):e038776.

Fekri O, Manukyan E, Klazinga N. Appropriateness, effectiveness and safety of care delivered in Canadian hospitals: a longitudinal assessment on the utility of publicly reported performance trend data between 2012–2013 and 2016–2017. BMJ Open. 2020;10(6):e035447.

Galloa AJO, Ramírez CA. Evaluating Colombian public hospitals productivity during 2004–2015. A luenberger-indicator approach. Rev Gerenc Y Polit Salud. 2020;19:1–23.

Gebreyes M, Sisay A, Tegen D, Asnake A, Wolde M. Evaluation of laboratory performance, associated factors and staff awareness towards achieving turnaround time in tertiary hospitals, Ethiopia. Ethiop J Health Sci. 2020;30(5):767–76.

Gnanaraj J, Kulkarni RG, Sahoo D, Abhishekh B. Assessment of the Key Performance Indicator Proposed by NABH in the Blood Centre of a Tertiary Health Care Hospital in Southern India. Indian J Hematol Blood Transfus. 2022;39:308–16.

Horenberg F, Lungu DA, Nuti S. Measuring research in the big data era: the evolution of performance measurement systems in the Italian teaching hospitals. Health Policy (New York). 2020;124(12):1387–94.

Hwang DK, Cho J, Moon J. Feasibility study on energy audit and data driven analysis procedure for building energy efficiency: bench-marking in Korean hospital buildings. Energies. 2019;14(15):3006.

Jaskova D. Efficiency of management, processes in a private hospital. Entrep Sustain Issues. 2021;9(1):436–46.

Jebraeily M, Valizadeh MA, Rahimi B, Saeidi S. The development of a clinical dashboard for monitoring of key performance indicators in ICU. J Iran Med Counc. 2022;5(2):308–17.

Kang Y, Kim M, Jung K. The equity of health care spending in South Korea: testing the impact of publicness. Int J Environ Res Public Health. 2020;17(5):1775.

Abou Tarieh RR, Zayyat R, Naoufal RN, Samaha HR. A case study exploring the impact of JCI standards implementation on staff productivity and motivation at the laboratory and blood bank. Heal Sci Rep. 2022;5(1):e497.

Kadoic N, Simic D, Mesaric J, Redep NB. Measuring quality of public hospitals in croatia using a multi-criteria Approach. Int J Environ Res Public Health. 2021;18:19.

Khalilabad T, Amir N, Asl P, Raeissi Shali M, Niknam N. Assessment of clinical and paraclinical departments of military hospitals based on the Pabon Lasso Model. J Educ Health Promot. 2020;9:1–6.

Lai JHK, Hou H, Edwards DJ, Yuen PL. An analytic network process model for hospital facilities management performance evaluation. Facilities. 2022;40(5–6):333–52.

Lai J, Yuen PL. Identification, classification and shortlisting of performance indicators for hospital facilities management. Facilities. 2021;39(1–2):4–18.

Lin CS, Chiu CM, Huang YC, Lang HC, Chen MS. Evaluating the operational efficiency and quality of Tertiary hospitals in Taiwan: the application of the EBITDA Indicator to the DEA Method and TOBIT Regression. Healthcare. 2022;10(1):58.

Matos R, Ferreira D, Pedro MI. Economic analysis of portuguese public hospitals through the construction of quality, efficiency, access, and financial related composite indicators. Soc Indic Res. 2021;157(1):361–92.

Morisod K, Luta X, Marti J, Spycher J, Malebranche M, Bodenmann P. Measuring health equity in emergency care using routinely collected data: a systematic review. Heal Equity. 2021;5(1):801–17.

Nik Hisamuddin R, Tuan Hairulnizam TK. Developing key performance indicators for emergency department of teaching hospitals: a mixed fuzzy Delphi and nominal group technique approach. Malays J Med Sci. 2022;29(2):114–25.

Ramírez Calazans A, Paredes Esteban RM, Grijalva Estrada OB, Ibarra Rodríguez MR. Assessment of quality indicators in pediatric major outpatient surgery. Influence of the COVID-19 pandemic. Cir Pediatr. 2023;36(1):17–21.

PubMed   Google Scholar  

Shaqura II, Gholami M, Akbari Sari A. Assessment of public hospitals performance in Gaza governorates using the Pabón Lasso Model. Int J Health Plann Manage. 2021;36(4):1223–35.

Al-Jazairi AS, Alnakhli AO. Quantifying clinical pharmacist activities in a tertiary care hospital using key performance indicators. Hosp Pharm. 2021;56(4):321–7.

Aloh HE, Onwujekwe OE, Aloh OG, Nweke CJ. Is bed turnover rate a good metric for hospital scale efficiency? A measure of resource utilization rate for hospitals in Southeast Nigeria. Cost Eff Resour Alloc. 2020;18(1):1–8.

Bari S, Incorvia J, Ahearn O, Dara L, Sharma S, Varallo J, et al. Building safe surgery knowledge and capacity in Cambodia: a mixed-methods evaluation of an innovative training and mentorship intervention. Glob Health Action. 2021;14(1):1998996.

Bari S, Incorvia J, Iverson KR, Bekele A, Garringer K, Ahearn O, et al. Surgical data strengthening in Ethiopia: results of a Kirkpatrick framework evaluation of a data quality intervention. Glob Health Action. 2021;14(1):1–11.

Bastos LSL, Hamacher S, Zampieri FG, Cavalcanti AB, Salluh JIF, Bozza FA. Structure and process associated with the efficiency of intensive care units in low-resource settings: an analysis of the CHECKLIST-ICU trial database. J Crit Care. 2020;59:118–23.

Bastos LSL, Wortel SA, de Keizer NF, Bakhshi-Raiez F, Salluh JIF, Dongelmans DA, et al. Comparing continuous versus categorical measures to assess and benchmark intensive care unit performance. J Crit Care. 2022;70:154063.

Kocisova K, Hass-Symotiuk M, Kludacz-Alessandri M. Use of the dea method to verify the performance model for hospitals. E M Ekon A Manag. 2018;21(4):125–40.

Lee D, Yu S, Yoon SN. Analysis of hospital management based on the characteristics of hospitals: focusing on financial indicators. Glob Bus Financ Rev. 2019;24(3):1–13.

Mirzaei A, Tabibi SJ, Nasiripour AA, Riahi L. Evaluating the feasibility of financial variables of health: A hospital administrator’s viewpoint. Galen Med J. 2016;5(1):25–30.

Middleton S, Gardner G, Gardner A, Considine J, FitzGerald G, Christofis L, et al. Are service and patient indicators different in the presence or absence of nurse practitioners? The EDPRAC cohort study of Australian emergency departments. BMJ Open. 2019;9(7):e024529.

Nobakht S, Jahangiri K, Hajinabi K. Correlation of performance indicators and productivity: A cross sectional study of emergency departments in Tehran, Iran during year 2016. Trauma Mon. 2018;23(5):1–6.

Nuti S, Grillo Ruggieri T, Podetti S. Do university hospitals perform better than general hospitals? A comparative analysis among Italian regions. BMJ Open. 2016;6(8):e011426.

Petrovic GM, Vukovic M, Vranes AJ. The impact of accreditation on health care quality in hospitals. Vojnosanit Pregl. 2018;75(8):803–8.

Pirani N, Zahiri M, Engali KA, Torabipour A. Hospital efficiency measurement before and after health sector evolution plan in Southwest of Iran: a DEA-panel data study. Acta Inf Med. 2018;26(2):106–10.

Profit J, Gould JB, Bennett M, Goldstein BA, Draper D, Phibbs CS, et al. The association of level of care with NICU quality. Pediatrics. 2016;137(3):44–51.

Rahimi H, Bahmaei J, Shojaei P, Kavosi Z, Khavasi M. Developing a strategy map to improve public hospitals performance with balanced scorecard and dematel approach. Shiraz E Med J. 2018;19(7):1–12.

Ahmed S, Hasan MZ, Laokri S, Jannat Z, Ahmed MW, Dorin F, et al. Technical efficiency of public district hospitals in Bangladesh: a data envelopment analysis. COST Eff Resour Alloc. 2019;17:17.

Rahman MH, Tumpa TJ, Ali SM, Paul SK. A grey approach to predicting healthcare performance. Meas J Int Meas Confed. 2019;134:307–25.

Sajadi HS, Sajadi ZS, Sajadi FA, Hadi M, Zahmatkesh M. The comparison of hospitals’ performance indicators before and after the Iran’s hospital care transformations plan. J Educ Health Promot. 2017;6:89.

Si S-L, You X-Y, Liu H-C, Huang J. Identifying key performance indicators for holistic hospital management with a modified DEMATEL approach. Int J Environ Res Public Health. 2017;14(8): 934.

Váchová L, Hajdíková T. Evaluation of Czech hospitals performance using MCDM methods. In: A SI, G WS, C D, editors. Department of exact methods, faculty of management, university of economics, Prague, Jarošovská 1117, Jindřichuv Hradec, vol. 37701. Czech Republic: Newswood Limited; 2017. p. 732–5.

Xenos P, Yfantopoulos J, Nektarios M, Polyzos N, Tinios P, Constantopoulos A. Efficiency and productivity assessment of public hospitals in Greece during the crisis period 2009–2012. Cost Eff Resour Alloc. 2017;15(1):6.

Zhang L, Liu R, Jiang S, Luo G, Liu H-C. Identification of key performance indicators for hospital management using an extended hesitant linguistic DEMATEL Approach. Healthc (Basel Switzerland). 2019;8(1):7.

Aksezer CS. A nonparametric approach for optimal reliability allocation in health services. Int J Qual Reliab Manag. 2016;33(2):284–94.

Cagliano AC, Grimaldi S, Rafele C. Assessing warehouse centralization and outsourcing in the healthcare sector: an Italian case study. In: Department of Management and Production Engineering, Politecnico Di Torino, Corso Duca Degli Abruzzi 24, Torino, 10129. Italy: AIDI - Italian Association of Industrial Operations Professors; 2017. p. 244–50.

Cefalu MS, Elliott MN, Setodji CM, Cleary PD, Hays RD. Hospital quality indicators are not unidimensional: a reanalysis of Lieberthal and Comer. Health Serv Res. 2019;54(2):502–8.

Gao H, Chen H, Feng J, Qin X, Wang X, Liang S, et al. Balanced scorecard-based performance evaluation of Chinese county hospitals in underdeveloped areas. J Int Med Res. 2018;46(5):1947–62.

Gonnelli V, Satta F, Frosini F, Iadanza E. Evidence-based approach to medical equipment maintenance monitoring. In: V HEO, V J, editors. University of Florence, Dept. of Information Engineering. Florence, Italy: Springer; 2017. p. 258–61.

Helkio P, Aantaa R, Virolainen P, Tuominen R. Productivity benchmarks for operative service units. ACTA Anaesthesiol Scand. 2016;60(4):450–6.

Khalifa M, Zabani I. Developing emergency room key performance indicators: What to measure and why should we measure it? J. M, A. H, P. G, A. K, M.S. H, editors. Vol. 226. King Faisal Specialist Hospital and Research Center, Jeddah, Saudi Arabia: IOS Press BV; 2016. p. 179–182.

Ajami S, Ebadsichani A, Tofighi S, Tavakoli N. Medical records department and balanced scorecard approach. J Educ Health Promot. 2013;2:7.

Bosse G, Mtatifikolo F, Abels W, Strosing C, Breuer J-P, Spies C. Immediate outcome indicators in perioperative care: a controlled intervention study on quality improvement in hospitals in Tanzania. PLoS One. 2013;8(6):e65428.

Hung K-Y, Jerng J-S. Time to have a paradigm shift in health care quality measurement. J Formos Med Assoc. 2014;113(10):673–9.

Jeejeebhoy KN, Keller H, Gramlich L, Allard JP, Laporte M, Duerksen DR, et al. Nutritional assessment: comparison of clinical assessment and objective variables for the prediction of length of hospital stay and readmission. Am J Clin Nutr. 2015;101(5):956–65.

Kittelsen SAC, Anthun KS, Goude F, Huitfeldt IMS, Häkkinen U, Kruse M, et al. Costs and quality at the hospital level in the nordic countries. Heal Econ (United Kingdom). 2015;24:140–63.

Koné Péfoyo AJ, Wodchis WP. Organizational performance impacting patient satisfaction in Ontario hospitals: a multilevel analysis. BMC Res Notes. 2013;6: 509.

Li CH, Yu CH. Performance evaluation of public non-profit hospitals using a BP Artificial neural network: the case of Hubei Province in China. Int J Environ Res Public Health. 2013;10(8):3619–33.

Liu K, Jain S, Shi J. Physician performance assessment using a composite quality index. Stat Med. 2013;32(15):2661–80.

Lloyd GF, Bajorek B, Barclay P, Goh S. Narrative review: Status of key performance indicators in contemporary hospital pharmacy practice. J Pharm Pract Res. 2015;45(4):396–403.

Mehrtak M, Yusefzadeh H, Jaafaripooyan E. Pabon Lasso and data envelopment analysis: a complementary approach to hospital performance measurement. Glob J Health Sci. 2014;6(4):107–16.

Mohammadi M, Ziapoor A, Mahboubi M, Faroukhi A, Amani N, Pour FH, et al. Performance evaluation of hospitals under supervision of Kermanshah medical sciences using pabonlasoty diagram of a five-year period (2008–2012). Life Sci J. 2014;11:77–81 ( 1 SPECL. ISSUE) ).

Niaksu O, Zaptorius J. Applying operational research and data mining to performance based medical personnel motivation system. In: Vilnius University, Institute of Mathematics and Informatics. Lithuania: IOS; 2014. p. 63–70.

Córdoba S, Caballero I, Navalón R, Martínez-Sánchez D, Martínez-Morán C, Borbujo J. Analysis of the surgical activity in the dermatology department of Fuenlabrada University Hospital, Madrid, Spain, between 2005 and 2010: determination of the standard operating times. Actas Dermosifiliogr. 2013;104(2):141–7.

Profit J, Kowalkowski MA, Zupancic JAF, Pietz K, Richardson P, Draper D, et al. Baby-MONITOR: a composite indicator of NICU Quality. Pediatrics. 2014;134(1):74–82.

Rabar D, Pap N. Evaluation of crotia’s regional hospital effiency: an application of data envelopment analysis . Bacher U, Barkovic D, Dernoscheg KH, LamzaMaronic M, Matic B, Runzheimer B, editors. Interdisciplinary Management Research IX. 2013;9:649–59.

Ramos MCA, da Cruz LP, Kishima VC, Pollara WM, de Lira ACO, Couttolenc BF. Performance evaluation of hospitals that provide care in the public health system, Brazil. Rev Saude Publica. 2015;49:1–9.

Schuers M, Joulakian MB, Griffon N, Pachéco J, Périgard C, Lepage E, et al. In: S IN, de PM AM, editors. Quality indicators from laboratory and radiology information systems. A. G. Volume 216. France: IOS; 2015. pp. 212–6. Department of Biomedical Informatics, Rouen University Hospital, Rouen Cedex, 76031,.

Tabrizi JS, Saadati M, Sadeghi-Bazargani H, Ebadi A, Golzari SEJ. Developing indicators to improve educational governance in hospitals. Clin Gov. 2014;19(2):117–25.

Costa A Jr, aS., Leão LE, Novais MA, Zucchi P. An assessment of the quality indicators of operative and non-operative times in a public university hospital. Einstein (Sao Paulo). 2015;13(4):594–9.

Coyne JS, Helton J. How prepared are US hospitals for the affordable care act? A financial condition analysis of US hospitals in 2011. J Health Care Finance. 2015;41(3).

Davis P, Milne B, Parker K, Hider P, Lay-Yee R, Cumming J, et al. Efficiency, effectiveness, equity (E-3). Evaluating hospital performance in three dimensions. Health Policy (New York). 2013;112(1–2):19–27.

Flatow VH, Ibragimova N, Divino CM, Eshak DSA, Twohig BC, Bassily-Marcus AM, et al. Quality outcomes in the surgical intensive care unit after electronic health record implementation. Appl Clin Inf. 2015;6(4):611–8.

Article   CAS   Google Scholar  

Fonseca JRS, Ramos RMP, Santos AMP, Fonseca APSS. Policy effects on the quality of public health care: evaluating Portuguese public hospitals’ quality through customers’ views. Cent Eur J Public Policy. 2015;9(2):122–40.

Hadji B, Meyer R, Melikeche S, Escalon S, Degoulet P. Assessing the Relationships Between Hospital Resources and Activities: A Systematic Review. J Med Syst. 2014;38(10):1–21.

Hajduová Z, Herbrik G, Beslerová S. Application of DEA in the environment of Slovak hospitals. Invest Manag Financ Innov. 2015;12(4):148–53.

Thomas Craig KJ, McKillop MM, Huang HT, George J, Punwani ES, Rhee KB. U.S. hospital performance methodologies: a scoping review to identify opportunities for crossing the quality chasm. BMC Health Serv Res. 2020;20(1):640.

Carini E, Gabutti I, Frisicale EM, Di Pilla A, Pezzullo AM, de Waure C, et al. Assessing hospital performance indicators. What dimensions? Evidence from an umbrella review. BMC Health Serv Res. 2020;20(1):1038.

Rasi V, Delgoshaee B, Maleki M. Identification of common indicators of hospital performance evaluation models: a scoping review. J Educ Health Promot. 2020;9(1):63.

Xenos P, Yfantopoulos J, Nektarios M, Polyzos N, Tinios P, Constantopoulos A. Efficiency and productivity assessment of public hospitals in Greece during the crisis period 2009–2012. COST Eff Resour Alloc. 2017;15:15.

Shaqura II, Gholami M, Sari AA. Evaluation of performance at Palestinian public hospitals using Pabon Lasso model. Int J Health Plann Manage. 2021;36(3):896–910.

Li J, Seale H, Ray P, Wang Q, Yang P, Li S, et al. E-Health preparedness assessment in the context of an influenza pandemic: a qualitative study in China. BMJ Open. 2013;3(3):e002293.

Huang C-Y, Lai C-H. Effects of internal branding management in a hospital context. Serv Ind J. 2021;41(15–16):985–1006.

Download references

Acknowledgements

The authors are grateful for the support of the Vice Chancellor for Research of Isfahan University of Medical Sciences.

The present article is part of the result of a doctoral thesis approved by Isfahan University of Medical Sciences with code 55657 (IR.MUI.NUREMA.REC.1401.005), without financial source.

Author information

Authors and affiliations.

Student Research Committee, School of Management and Medical Information Sciences, Isfahan University of Medical Sciences, Isfahan, Iran

Shirin Alsadat Hadian

Health Management and Economics Research Center, Isfahan University of Medical Sciences, Isfahan, Iran

Reza Rezayatmand

Hospital Management Research Center, Health Management Research Institute, Iran University of Medical Sciences, Tehran, Iran

Nasrin Shaarbafchizadeh

Department of Management, Faculty of Administrative Sciences and Economics, University of Isfahan, Isfahan, Iran

Saeedeh Ketabi

School of Public Health and Preventive Medicine, Monash University, Victoria, Australia

Ahmad Reza Pourghaderi

You can also search for this author in PubMed   Google Scholar

Contributions

Shirin Alsadat Hadian and Reza Rezayatmans and Saeedeh Ketabi: Study conceptualization and design. Acquisition of data: Shirin Alsadat Hadian, Reza Rezayatmand. Analysis and interpretation of data: Shirin Alsadat Hadian, Reza Rezayatmand, Nasrin Shaarbafchizadeh, Saeedeh Ketabi. Drafting of the manuscript: Shirin Alsadat Hadian, Reza Rezayatmand. Critical revision of the manuscript for important intellectual content: Reza Rezayatmand, Nasrin Shaarbafchizadeh, Saeedeh Ketabi, Ahmad Reza Pourghaderi.

Corresponding author

Correspondence to Reza Rezayatmand .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Hadian, S.A., Rezayatmand, R., Shaarbafchizadeh, N. et al. Hospital performance evaluation indicators: a scoping review. BMC Health Serv Res 24 , 561 (2024). https://doi.org/10.1186/s12913-024-10940-1

Download citation

Received : 03 January 2024

Accepted : 02 April 2024

Published : 01 May 2024

DOI : https://doi.org/10.1186/s12913-024-10940-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Performance evaluation

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Open access
• Published: 29 April 2024

Untangling the mess of CGRP levels as a migraine biomarker: an in-depth literature review and analysis of our experimental experience

• Gabriel Gárate 1 ,
• Julio Pascual 1 ,
• Marta Pascual-Mato 1 ,
• Jorge Madera 1 ,
• María Muñoz-San Martín 1 &
• Vicente González-Quintanilla 1  

The Journal of Headache and Pain volume  25 , Article number:  69 ( 2024 ) Cite this article

459 Accesses

9 Altmetric

Metrics details

Calcitonin gene-related peptide (CGRP) is the most promising candidate to become the first migraine biomarker. However, literature shows clashing results and suggests a methodological source for such discrepancies. We aimed to investigate some of these methodological factors to evaluate the actual role of CGRP as biomarker.

Previous to the experimental part, we performed a literature review of articles measuring CGRP in migraine patients. Using our 399 bio-bank sera samples, we performed a series of experiments to test the validity of different ELISA kits employed, time of sample processing, long-term storage, sampling in rest or after moderate exercise. Analysis of in-house data was performed to analyse average levels of the peptide and the effect of sex and age.

Literature review shows the high variability in terms of study design, determination methods, results and conclusions obtained by studies including CGRP determinations in migraine patients. CGRP measurements depends on the method and specific kit employed, also on the isoform detected, showing completely different ranges of concentrations. Alpha-CGRP and beta-CGRP had median with IQR levels of 37.5 (28.2–54.4) and 4.6 (2.4–6.4)pg/mL, respectively. CGRP content is preserved in serum within the 24 first hours when samples are stored at 4°C after clotting and immediate centrifugation. Storages at -80°C of more than 6 months result in a decrease in CGRP levels. Moderate exercise prior to blood extraction does not modulate the concentration of the peptide. Age positively correlates with beta-CGRP content and men have higher alpha-CGRP levels than women.

Conclusions

We present valuable information for CGRP measurements in serum. ELISA kit suitability should be tested prior to the experiments. Alpha and beta-CGRP levels should be analysed separately as they can show different behaviours even within the same condition. Samples can be processed in a 24-h window if they have been kept in 4°C and should not be stored for more than 6 months at -80°C before assayed. Patients do not need to rest before the blood extraction unless they have performed a high-endurance exercise. For comparative studies, sex and age should be accounted for as these parameters can impact CGRP concentrations.

Graphical Abstract

Peer Review reports

Introduction

Migraine and its subtypes are diagnosed based on clinical criteria [ 1 ]. Thus, multiple phenotypes sharing the same diagnosis are treated the same way with clashing outcomes. However, as many real-world data studies have shown [ 2 ], the different phenotypes have been proved ineffective to create profiles prone to respond to the different treatment options. Historical therapies for migraine, which is worth to mention that none of these were initially developed to treat this condition, apart from the triptans, and are not specific for it [ 3 ], have not met the challenge of effectively aborting and/or preventing the symptoms, in some cases with limited efficacy, tolerability and patient adherence [ 4 ].

Since the 1990s decade our understanding of migraine has expanded markedly and new therapeutic agents have been brought to the market in an effort to alleviate the personal and economic burden that migraineurs suffer. These are the calcitonin gene-related peptide (CGRP)-targeted therapies which have revolutionized the management of migraine [ 5 ], including monoclonal antibodies against the CGRP ligand or its receptor [ 6 ], and small molecules antagonists to the CGRP receptor, the gepants [ 7 ]. Nonetheless, there is still a portion of patients who do not respond to the treatments, highlighting the importance that a biomarker would have in migraine, allowing to create objective diagnostic criteria besides the clinical ones, which may be subject to errors [ 8 ], and to monitor objectively the response to treatments.

CGRP is a multifunctional neuropeptide which was first discovered in 1982, described as the result of the alternative splicing of the calcitonin gene (CALCA in humans) transcript, hence its name [ 9 ]. Later on, this first form of CGRP will be named alpha-CGRP, as opposed to the beta-CGRP, encoded in a different gene (CALCB in humans), with a different regulation and expression pattern to the alpha-CGRP [ 10 ]. These two peptides differ in 3 out of the 37 amino acids of their sequence but share a common structure and are part of the CGRP peptide family, also comprised by calcitonin, adrenomedullin 1 and adrenomedullin 2 [ 11 ]. Although their distribution in the human body tends to overlap [ 12 ], alpha-CGRP has been described to be the predominant form in the central and peripheral nervous system while beta-CGRP is more relatively abundant in the enteric nervous system [ 13 ].

The relevance of the peptide goes beyond its use as a therapeutical target, having been proposed as a biomarker in migraine. Several studies have reported elevation of the peptide in ictal and/or interictal phases in medication-free periods of migraine patients [ 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 ], the reduction of the CGRP levels after abortive and prophylactic treatment [ 26 , 28 , 38 , 39 , 40 , 41 , 42 , 43 , 44 ] and the induction of migraine-like headaches when infused in humans [ 45 ]. Despite these results, there are other works contradicting the findings [ 35 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 ] and which emphasize the way until its eventual validation and clinical use is still far way to become a reality. The source of such discrepancies, although still unknown, is most probably multifactorial. There is a methodological component [ 55 , 56 ] and the influence of other individual parameters such as comorbidities [ 36 ], concomitant treatments [ 57 ] or menstrual cycle [ 58 , 59 ], which have not been taken into account or which have not been sufficiently described to be considered properly.

In this work we have analysed in detail the existing literature about CGRP measurements in migraine patients, discussing their methodological differences and their effect on the reported concentrations of the peptide. In addition, we have conducted a series of experiments aimed to elucidate the potential effects on serum content of total CGRP, alpha-CGRP, and beta-CGRP, of a number of variables, including different enzyme-linked immunosorbent assay (ELISA) kits, sample processing time, long-term storage or immediate practise of exercise before sampling. Finally, we have analysed our in-house database of CGRP measurements to investigate the effect that sex and age might have on the molecules.

Review of previously published works including CGRP measurements in migraine patients

A systematic search was conducted in the databases PubMed, Scopus and Science Direct until February 2024 using the following terms: (a) CGRP; (b) migraine; and one of the following terms: (c) levels; (d) concentration; (e) measurements. We included original articles with CGRP measurements in humans with migraine. We only included and analysed works written in English language.

Methodological experiments

Kit analysis.

We tested 4 different ELISA kit references with serum samples, 2 based on competitive ELISA (Biorbyt, UK, ref: orb438605; BMA Biomedicals, Switzerland, ref: S-1198), specifically designed for the detection of total-CGRP, and 2 based on ELISA sandwich (Abbexa, UK, ref: abx257902; CUSABIO, China, ref: CSB-E08210h), designed for the detection of alpha and beta-CGRP, respectively. All 4 of these products were assayed multiple times (at least 4 for each kit reference) to analyse the optimal dilutions of the samples, their reproducibility and their reported concentrations. All the procedures were carried out strictly following the manufacturer’s instructions of use of their products, they were performed by the same researcher, using the same equipment, and in the same facilities. Regarding the last step of the ELISA processes, in which manufacturers give a window of time, specifying that the user must determine the optimum, we incubated the substrate for 15 min for alpha-CGRP and for 20 min for beta-CGRP. All the samples were measured in duplicate, obtained from morning blood extractions, 9–12 am, from patients in a fast of at least 12h. These samples were let to clot for 10–15 min, centrifuged at 3500 rpm for 10 min and then immediately stored at -80°C until assayed. A standard curve was generated for every single batch, and they were calculated using a 4-parameter logistic (4-PL) regression with r 2  > 0.999.

Influence of sample processing time

We recruited 6 individuals without history of migraine and subjective absence of headache at the day of the sampling (50% male; age range: 24–65 years). These individuals had a blood extraction in the early morning, between 9 am and 9:30 am, performed in rest at our laboratory facilities. The blood was then let to clot for 10 min at room temperature, then centrifuged at 3500 rpm for 10 min to obtain serum. Serum was divided into 4 tubes. First one was immediately stored at -80°C, the other three were kept in the refrigerator at 4°C for, 2, 4 and 24 h respectively before frozen. None of the samples were added peptidase inhibitor. These samples were measured by triplicate.

Effect of exercise

Additionally, after the first blood extraction, these same 6 subjects were asked to perform a 20 min run at moderate pace before a second blood extraction. Blood obtained was then processed following the same procedure as the resting samples but in this case all the serum was immediately stored at -80°C. These samples were measured by triplicate.

Long-term storage

We assayed 11 consecutive samples from previous works (36.4% male; age range: 26–65) that had been stored at -80°C for more than 6 months and which had been assayed altogether before being stored for a month and before reaching this time point.

Analysis of our CGRP database

Samples coming from our bio-bank were grouped together, reaching 399 individuals (29.3% male; age range: 18–96 years), and then analysed to see the average levels of the peptide and possible effects of sex and age in the circulating concentrations of the molecules.

Statistical analysis

Data are displayed as average with standard deviation (SD) unless stated differently. Comparisons between samples immediately processed and stored at -80°C obtained in resting subjects and right after exercising, and samples analysed before and after they had been stored for 6 months were made using the Wilcoxon matched-pairs signed rank test. Comparisons between samples from same individuals that were frozen at different timepoints have been made using Friedman test followed by Dunn’s test. Correlation relationships of the meta-analysis were evaluated by Spearman correlation test and summarized by Spearman’s rho coefficient and related p-values. Comparisons between sub-groups in the meta-analysis were performed using the Mann Whitney U test.

Article review

Applying the criteria specified in the method section we included 52 articles from the initial search that have been sorted by sample source and detection methodology and are displayed in Table  1 .

Out of these 52 articles, the main source of sample were blood extractions, with 44 (84.6%) works performing them. Twenty-eight (53.8%) used plasma samples, 6 (11.5%) from the jugular vein and the remaining 22 (42.3%) from the cubital vein. Serum was employed in 16 (30.8%) of the studies. Continue by order of use, saliva was the third sample source with 7 (13.5%), followed by cerebrospinal fluid (CSF) with 3 (5.8%) and by tear fluid with 2 (3.8%), and last, gingival crevicular fluid (GCF) with 1 (1.9%). According to the determination method, 21 (40.4%) of the studies measured CGRP by radioimmunoassay (RIA), 2 (3.8%) of them together with Bradford protein assay (Bradford), and 29 (55.8%) by ELISA, 1 (1.9%) performed along with bicinchoninic acid protein assay (BCA), and 2 (3.8%) used undefined enzyme immune assay (EIA).

Seventeen (32.7%) studies did not include healthy controls while the remaining 35 (67.3%) did. Sampling of the migraine patients were performed only in the ictal phase for 5 (9.6%) studies, only in the interictal phase for 20 (38.5%), in both phases for 22 (42.3%) works, and in 5 (9.6%) of them the phase was not specified.

Data was presented in different ways including mean ± standard deviation, ± standard error of mean (SEM), ± 2*SEM; median with range, interquartile range (IQR), 95% confidence intervals (CI), and in multiple units, pmol/L, fmol/mL, pmol/mg of total protein, pg/mL, pg/µg of total protein.

Therefore, these methodologies, sampling differences and variable data displays did not allow for a meta-analysis, and the absolute CGRP range among all the studies could be inferred, showing a wide range of concentrations (2.45–219,700 pg/mL) [ 28 , 54 ].

Experimental results

The kit from Biorbyt showed an elevated content of CGRP (range: 150-980pg/mL) compared to what has been reported in the bibliography [ 24 , 25 , 26 , 27 , 29 , 38 , 49 , 54 ] when undiluted serum samples were used. Moreover, the reproducibility of the kit was not satisfactory as the assayed samples did not meet the intra and inter-assay coefficient of variance criteria set by the manufacturer (> 10% and > 12%, respectively). This kit also showed a total lack of linearity for the dilutions of 1:2, 1:4, 1:8, 1:16 and 1:32 with each dilution showing higher CGRP concentrations than the one before (data not shown).

For BMA Biomedical kit we were unable to obtain a single measurement within the detection range. Since we decided to strictly follow the manufacturer’s instructions, we could not modify the standard curve points. All the readout absorbance measurements from the tested samples exceed the absorbance range obtained from the readout of the standard curve, and because this is a competitive ELISA technique, no dilution could be tested and neither we could assayed the reproducibility of the test.

Alpha-CGRP specific kit, from Abbexa, showed similar CGRP concentrations (range: 25-105pg/mL) to what has been described previously in most studies using serum from our group [ 38 , 69 , 70 ] and others [ 25 , 26 , 27 , 49 ]. Most of the samples fall within mid-range of the standard curve but the kit showed a good linearity of the measurements when samples were diluted 1:2, 1:3, 1:4 and 1:8 (data not shown). Across the different plates results fulfilled the reproducibility criteria by having an intra and inter-assay coefficient of variance below the maximum set by the manufacturer (< 8% and < 10%, respectively).

The last kit, from CUSABIO, showed similar beta-CGRP concentrations than reported in the literature (range: 1.6–10.5pg/mL) [ 31 , 35 , 36 , 38 , 70 , 71 ]. Because the samples fall within the lower part of the standard curve dilution of 1:2, 1:3 and 1:4 resulted in a lack of signal and the impossibility to determine the concentration of the peptide in all the samples but those with the higher beta-CGRP content. In this latter group the linearity found was between the ranges supplied by the manufacturer. Across the different plates results fulfilled the reproducibility criteria by having an intra and inter-assay coefficient of variance below the maximum set by the manufacturer (< 8% and < 10%, respectively).

Because the 2 kits based on competitive ELISA did not meet the quality requirements and did not adjust to the reported units in the literature the following experiments were carried out using the kits from Abbexa and CUSABIO which have been used by our group in previous studies [ 38 , 69 , 70 , 71 ].

We did not find changes in alpha nor beta-CGRP across samples which remained for 2 h (alpha: 29.9 ± 18.6pg/mL; beta: 4.9 ± 1.7pg/mL), 4 h (alpha: 30.4 ± 18.2pg/mL; beta: 4.7 ± 1.5pg/mL) and 24 h (alpha: 30.2 ± 19.6pg/mL; beta: 4.4 ± 1.8pg/mL) at 4°C compared to those which got deep frozen right away (alpha: 29.2 ± 20.6pg/mL; beta: 4.6 ± 1.6pg/mL; p  = 0.99; p  = 0.84; p  = 0.99; respectively) (Fig.  1 ).

Sample processing: evolution of individual A alpha-CGRP and B beta-CGRP values for each subject throughout the time samples remained stored at 4°C before froze at -80°C

No differences were found in none of the molecules when comparing serum samples obtained in rest an immediately stored at -80°C and those obtained after exercise and with the same processing protocol (alpha: 31.1 ± 19.0pg/mL; beta: 4.8 ± 1.7pg/mL; p  = 0.44) (Fig.  2 ).

Effect of exercise: evolution of individual A alpha-CGRP and B beta-CGRP values for each subject when sampling was performed in rest of after 20 minutes of moderate exercise

The first significant differences between samples which were measured before they remained stored at -80°C for a month (alpha: 42.3 ± 15.1pg/mL; beta: 4.9 ± 2.0pg/mL) and assayed after this date appeared from the sixth month of storage for both alpha-CGRP and beta-CGRP (alpha: 28.6 ± 11.3pg/mL, p  < 0.01; beta: 3.0 ± 1.3pg/mL, p  < 0.01) (Fig.  3 ).

Effect of storage: changes of individual A alpha-CGRP and B beta-CGRP values when samples were immediately analysed or analysed when they surpassed 6 months storage. Data is shown as average ± SD. Comparisons were made using Wilcoxon matched-pairs signed rank test. ** p  < 0.01

Analysis of our database

Alpha and beta-CGRP did follow a normal distribution and averaged (median with IQR) 37.5 (28.2–54.4)pg/mL and 4.6 (2.4–6.4)pg/mL, respectively. Spearman correlation between alpha-CGRP and age was non-significant ( p  = 0.300; r  = -0.05), while it was significant for beta-CGRP and age ( p  < 0.0001; r  = 0.24). When these correlations were analysed with females and males alone it kept being non-significant for alpha-CGRP (male: p  = 0.151, r  = -0.14; female: p  = 0.514, r  = -0.04) and significant for beta-CGRP (male: p  = 0.028, r  = 0.21; female: p  < 0.0001, r  = 0.26). Alpha and beta-CGRP levels did not correlate significantly ( p  = 0.056; r  = 0.11). When sorted by sex, groups had no significant differences in their age distribution (male: 55.6 ± 17.7 years; female: 54.1 ± 16.9 years; p  = 0.222), and showed significant differences in their alpha-CGRP content (median [IQR]; males: 54.4 [38.1–77.6] pg/mL; females: 45.2 [32.5–65.3] pg/mL; p  < 0.01) and unaltered beta-CGRP levels (median [IQR]; males: 4.0 [2.3–6.2] pg/mL; females: 3.9 [2.1–6.1] pg/mL; p  = 0.728) (Fig.  4 ).

In-house data analysis: A distribution of alpha-CGRP levels vs. age, green line represents a linear regression and red dotted line represents the CI; B distribution of beta-CGRP levels vs. age, green line represents a linear regression and red dotted line represents the CI; C distribution of beta-CGRP vs. alpha-CGRP levels, green line represents a linear regression and red dotted line represents the CI; D comparison of alpha-CGRP concentrations in subjects sorted by sex; E comparison of alpha-CGRP concentrations in subjects sorted by sex. Data is shown as average ± SD. Comparisons were made using Mann–Whitney U test, ns: non-significant; ** p  < 0.01

Our literature analysis (Table  1 ) shows that studies based on CGRP determinations are highly variable in terms of measuring method and study design, including sample source, sample processing, inclusion/exclusion criteria for patients and controls and aim of the study [ 14 , 15 , 19 , 31 , 39 , 42 , 60 , 66 , 68 ]. Data analysis and presentation of laboratory determinations is also changeful, which hinder the comparison of the data. Despite all the difficulties, it results obvious that the overall outcomes and the conclusions drawn from them are inconsistent across works. Some authors have hypothesized that methodological differences might be the reason for such discrepancies [ 55 , 56 ], and, although this is likely to be the case, there is not to date a consensus of how CGRP determinations should be carried out.

If we analyse the methods used to measure CGRP in migraine patients we can see there have been mainly based on two different techniques, RIA and ELISA. RIA was the first, and until the late 2000s, the only one employed. RIA is based on the competitive incubation for specific antibody sites to form antigen–antibody complexes of radio-labelled and native unlabelled antigen. At equilibrium, the complexes formed are separated from the unbound antigen with a resulting ratio between these two. The bound/free antigen ratio is dependent on the amount of native antigen present in the sample as the radio-labelled is always added at a stable known concentration [ 72 ]. Therefore, this technique relies on the antiserum used, which has to provide an appropriate specificity in order to detect the antigen but no other analogues, and a proper affinity to do so in the range of interest.

The use of different antisera across all the CGRP-measuring studies based on RIA is a main source of variability among articles (Table  1 ). Works employing the same protocol, antiserum, and sample source usually have similar peptide concentrations [ 14 , 39 , 47 ], with some exceptions [ 48 ], while the use of different brands containing different antiseras and protocols show differing concentration ranges even when performed with same sample source [ 15 , 39 , 63 , 64 ], and even if they were done by the same specialist technician with the same samples [ 48 ]. Another problem is that even though studies with the exact same quantification method obtain similar concentration ranges they arrive to clashing conclusions, such as the presence of differences in CGRP concentrations between interictal migraine patients and healthy controls [ 17 , 65 ].

ELISA technique first appears to be used to determine CGRP concentration in migraine patients in 2007 [ 21 ]. ELISA is an immunological assay based on the interaction between the antigen and a primary antibody against the antigen of interest. These will interact, forming a complex that is later confirmed through the enzyme-linked antibody catalysis of an added substrate, which can be quantitatively measured using readouts from either a luminometer or a spectrophotometer. ELISA techniques are broadly classified into direct, indirect, sandwich, and competitive ELISA. For CGRP determinations only competitive and sandwich ELISA have been employed. Competitive ELISA involves a competition between the sample antigen and the plate-coated antigen for the primary antibody, followed by the binding of enzyme-linked secondary antibodies (Fig.  5 ). Sandwich ELISA technique includes a sample antigen introduced to the antibody-precoated plate, followed by sequential binding of detection and enzyme-linked secondary antibodies to the recognition sites on the antigen (Fig.  6 ) [ 73 ]. In both cases, and similarly to what has been pointed out for RIA, the techniques rely on the specificity and sensitivity of the antibodies included in the kit. This is the reason why ELISA-based studies are also subjected to the exact same issues associated with RIA-based works. As it has been described, investigations using the same brand also reports similar peptide concentration ranges [ 25 , 26 , 30 , 44 , 49 , 67 ], even though this is not always the case [ 32 ], but, most importantly, those using different kits clash in the range of concentrations [ 23 , 61 , 62 ] on top of the conclusions drawn [ 33 , 61 ]. For this point we need to explain that kits from USCN Life Sciences and Cloud Clone Corp., and from Peninsula Laboratories and BMA Biomedicals have been considered as only two brands since these companies have merged or have been acquired by the other at some point in their history. Moreover, and this last point serves as an example, there is a lack of information by part of the researchers regarding the kits used, because sometimes the brand cited offers more than one kit or two different brands over the history have been in charge of its production, and with the given information it cannot be inferred which one it was [ 27 , 34 ]. This could be the reason why across studies using kits from the same brand they obtained different concentrations. Also, this lack often comes from the manufacturers, which most of the times do not report essential information to the user such as the specific epitope recognised by the antibodies or their cross-reactivity for analogues of CGRP. This has caused some controversies such as works employing kits specifically designed, according to the manufacturer, for the detection of beta-CGRP reporting results as total-CGRP [ 35 , 36 , 59 ] without proving in their papers whether the technique recognises alpha, beta, or total-CGRP.

Schematic representation of a competitive ELISA protocol

Schematic representation of a sandwich ELISA protocol

CGRP has been analysed in a broad number of samples sources including plasma and serum from the peripheral circulation and jugular vein, CSF, saliva, tear fluid and GCF. Due to the enormous variability of concentrations found within the sources (Table  1 ) and the fact that results are not homogenous even when the same technique and sample source were used, we thought the comparison between sample sources did not make sense.

Nonetheless, and because our group has focused on the determinations in serum with ELISA, we have done a specific analysis of the studies matching these two criteria. There seems to be a consensus range achieved by most of the studies, independently of the brand employed, and which approximately goes from 15 to 150 pg/mL for total and alpha-CGRP, because the data from the literature exhibits that most of the measured CGRP is the alpha isoform, and from 2 to 10 pg/mL for beta-CGRP.

Because there are examples of different works employing the same method, specific technique, sample source and similar inclusion/exclusion criteria whose results are contradictory [ 14 , 33 , 47 , 59 ], we cannot conclude that all the problematic with CGRP measurements is related to the quantifying method and/or the sample chose by the authors. There has to be other factors playing a role in the discrepancies, such as fluctuations with the circadian [ 74 ] or with the menstrual cycles [ 58 , 59 , 75 ], effect of resting/exercise [ 76 , 77 ], fast degradation of the peptide due to its short half-life [ 78 ], long-term storage stability [ 55 ], migraine and other comorbidities [ 69 , 71 , 79 , 80 , 81 ], and the effects of pharmacological treatments [ 26 , 28 , 38 , 39 , 40 , 41 , 42 , 43 , 44 ]. From our review we could not analyse these parameters, because they were not displayed with enough accuracy in most articles.

Experimental studies

Here, in an effort to provide more detailed information about the suitability of serum from peripheral blood for CGRP determinations, we carried out a series of experiments in order to shed light on some of the main questions regarding the lack of consistency with CGRP quantifications beyond the data already discussed from our review.

We have found that the specific ELISA kit employed has a crucial effect on the CGRP measurements, showing completely different concentration ranges depending on the reference.

Besides the differences in the range we have obtained some alarming results. One of the kits assayed, from Biorbyt, did not meet the reproducibility criteria, which automatically should make this kit unsuitable for any kind of research. On top of that it did not conserve the linearity when diluting the samples which adds more doubts to its reliability. The one from BMA Biomedicals, even though a kit from this brand has been used for a published work when the company had the name Peninsula Laboratories [ 24 ], showed for 4 different times results below the detection limit (20pg/mL), contradicting the data of the cited article. Once again, these data call for a more exhaustive description of the methodology, not only by the researchers but also the companies.

The other two kits assayed fulfilled all the quality requirements and presented a range of measurements which fit the range observed in studies using the same sample source. Because the kit from CUSABIO is specific for beta-CGRP we have considered that the objective range for this kind of determination is different to the range for the Abbexa kit, which detects alpha-CGRP. This comes with no surprise because notwithstanding we have not displayed it, in our previous works the exact internal validations were performed and our researches already shown that these kits were reliable and were in accordance with the results published in the past by other groups [ 25 , 26 , 27 , 31 , 35 , 36 , 49 ].

Overall, the analysis of the kits performed here acts as a probe that the determination methodology needs to be carefully assayed and critically analysed as this is the ultimate guarantee of the validity of the data. Because we have already done so with the 4 kit references listed in this study, we would like to encourage researchers to share their internal validation data with other kits they might have been using, as well as to invite the companies to share more details about their products, as we believe it has been a huge limitation in the field and this would produce a significant advance, saving a lot of time and money to the research.

Throughout the literature many different studies have acknowledged the reported short half-life of CGRP [ 38 , 52 , 55 , 56 , 82 ] as a main limitation for their works. Still, many fail to describe precisely enough their methodology for sample processing so readers can infer how this limitation took place. This problem has been pointed out before and the latest works have included a more accurate description of the sample processing [ 35 , 36 , 38 , 55 ]. To avoid this rapid degradation of the peptide Messlinger et al. [ 55 ] proposed buffering the sample with peptidase inhibitor, but they concluded that immediate freezing was the most effective way to preserve CGRP content.

We did not add peptidase inhibitors as we were using serum as sample and the addition of a peptidase inhibitor needed to be done right after centrifugation but we opt to freeze the samples immediately. Our results show that the degradation of the peptide did not happen, at least in the first 24 h, when samples were stored at 4°C. This complies with the instructions of most of the ELISA kits our group has assayed and which provide a window of time for sample storage depending on the temperature, specifying that samples can be stored at 4°C for up to 24 h before being analysed. These data appear to be contradicting the results of Kraenzlin et al. [ 78 ] regarding the half-life of CGRP. One could argue that the content of serum and plasma is different and the differences found in these studies could be accounted for the binding of CGRP to cellular compartments or to fibrinogen, effectively modifying its degradation. However, the cited article, performed in 1985, is not exempt from limitations and should be reconsidered when analysing the stability of the peptide, at least in isolated biological fluids. First, this pharmacokinetic (PK) study fails to achieve some critical points that are currently required for this kind of works [ 83 ]. CGRP concentration should achieve a steady state in order to be able to extrapolate the half-life as it at this point when the phenomena of absorption, distribution, metabolism and excretion have reached and equilibrium and therefore stopping the infusion will give the information about the actual elimination half-life. Moreover, results from human in-vivo PK studies are not necessarily equivalent to those obtained from in-vitro or animal models in-vivo [ 84 ]. Our findings show that serum freezing does not need to be immediate as long as it is kept in the fridge after instant centrifugation following the clotting. This discovery has the potential to ease the methodology of sample processing for CGRP determinations. Although this is a disruptive finding, data should come with no surprise as other neuropeptides with similar and even shorter half-life than CGRP, such as vasoactive intestinal peptide (VIP) [ 85 ], amylin [ 86 ], and pituitary adenylate cyclase activating peptide-38 (PACAP-38) [ 87 ] are being measured without controversy over the sample processing time [ 33 , 51 , 88 ].

Another point recurrently mentioned in the literature is the long-term stability of the molecules when frozen. Available data indicates that storages of 8 months [ 55 ] significantly decrease the concentration of CGRP. Our results show that storages over 6 months have a decreasing effect on the serum levels of both isoforms of the peptide. With all the evidences collected future research should specify the time samples remained stored prior to being assayed as this could be a main limitation of the study and to date this data is not usually displayed. This opens up the question about whether controls should be matched not only by age and sex but also by the time their samples remained stored until measured, meaning that both groups, patients and controls, should be enrolled simultaneously to ensure the comparability of their CGRP measurements. This point has already been discussed in studies employing CGRP measurements with controversy results where cases and controls were recruited in two different time frames [ 89 ].

The first potential association between physical exercise and CGRP was described by Wyon et al. [ 90 ] with an animal model showing that rats had higher concentrations of CGRP in urine, CSF and serum after 1-h of running. Subsequent studies with more animal models have confirmed this relationship [ 91 , 92 ]. To date the evidence derived from studies with humans is scarce, with only two works [ 76 , 77 ]. The first one [ 76 ] showed that CGRP increased its concentration in samples collected by microdialysis in 8 individuals who had been subjected to eccentric exercise. In the second, completion of a half marathon produced an immediate CGRP increase dependent on the running intensity in 48 individuals [ 77 ].

The relevance for these discoveries in clinical practise is limited because subjects do not usually perform that kind of exercises right before a blood sampling. This is why we analysed the effect of exercise in a way that would reflect more accurately what might be happening at the actual sampling. The results showed that this kind of practise does not have an effect on alpha nor beta-CGRP levels and consequently the patients do not need to be on a strict rest prior to the blood extraction. However, data need to be managed carefully because the exact amount of exercise that has an effect has not been described yet and because the window between the no effect of a 20 min run and a half marathon is wide.

All the results obtained from the experimental analysis would need to be further explored with a bigger number of participants and to be tested in other samples sources that are being considered for CGRP determinations. Nonetheless, it is important to highlight that when considering the future use of CGRP as a biomarker it is necessary to select a sample source that is easy to obtain, which does not have irregular fluctuations associated with unknown factors and which offers reproducible and robust results. Jugular blood, tear fluid, CSF, GCF are not easy to obtain and saliva sampling has to follow very strict protocols to be reliable [ 93 ], so our opinion is that future research should perhaps be focusing in plasma and serum from peripheral blood.

In-house meta-analysis

Our results show that, with a huge number of participants, the levels obtained with Abbexa and CUSABIO kits fit the consensus range seen in the literature review for alpha and beta-CGRP, respectively, and contribute to set a more standardized range of concentration for the peptides.

The effects of sex and age on the circulating levels of CGRP is a point which has not been explored deeply enough. While some studies affirm that CGRP can correlate with age [ 38 ], others do not find such correlation [ 35 , 70 ]. For the data obtained from our in-house samples we found that beta-CGRP correlated positively with age, contradicting previous results obtained with the same kit in plasma and saliva [ 35 ]. Besides, the sub-group comprised by males had different alpha-CGRP content that the female, a finding which had not been described. Taking all these data together, the results call for a stricter control of the group design, which would need to be carefully matched in terms of sex and age, to avoid the effect that these two parameters could have on the comparisons.

Also, as the discrimination between alpha and beta-CGRP in research papers has recently begun [ 38 , 69 , 70 , 71 ], we have shown that these two peptides do not correlate their circulating levels and therefore the results obtained from measuring one or the other are not interchangeable and could lead to opposite conclusions because these molecules can have different behaviours even within the same disorder [ 38 ].

Strengths and limitations

Our work has several strengths. Our literature review summarizes in an easy to understand way all the mess regarding CGRP measurements, showing all the differences not only in terms of results, but also in their aims, design, measuring methodologies and conclusions, allowing for a critical analysis and which will serve as a basis for future comparisons.

Due recent literature has begun to differentiate between alpha and beta-CGRP, we have performed all our experiments to continue doing so, in an effort to expand the knowledge about the different traits of the two molecules.

All the enrolled individuals of the analysis of exercising and duration of the sample processing had their blood extraction performed at the same day and time, limiting the variability that the circadian cycle might have on the levels of the peptide, and all of them were carried out at our laboratory facilities, ensuring an immediate processing and freeze of the serum. Samples for the long-term storage analysis were also obtained at the same time of the day and all of them were collected within a week and assayed for the first and the second time altogether, limiting the effects of different storage time until the determinations and intra-assay variations.

Nonetheless, it has also some limitations that need to be listed. Although we had a bigger list of ELISA kits that have been employed by other researchers, we could not test them all and we decided to probe only 4, including both competitive and sandwich ELISA targeting total, alpha and beta-CGRP. The validity of other kits apart from the ones included in this study would need to be evaluated separately. Also, the results derived from our methodological experiments should be tested in other samples sources as we only included serum because this is, in our opinion, the best sample source for CGRP determinations. For the analysis of our data base, we acknowledge that we did not account for some of the comorbidities of the patients when analysing the effects of sex and age, but because these samples were from our bio-bank their clinical information was limited to the original aim why they were obtained and therefore did not allow such kind of correction.

We have reviewed the different results obtained throughout the years measuring CGRP making an effort to highlight their differences in terms of aim, inclusion/exclusion criteria, methodology, data display and conclusions. We have also analysed the way these differences might have affect the CGRP levels reported and we have come to the conclusion that is not only the sample or the method (RIA or ELISA) but even the brand employed which ultimately determine the concentration range.

Finally, we have illustrated some new features of CGRP determinations in serum which are very valuable for the planning of future studies. Concentrations of alpha-CGRP and beta-CGRP seems to be about (median with IQR) 37–5 (28.2–54.4) pg/mL and 4.6 (2.4–6.4) pg/mL, respectively, according to our in-house analysis, which agrees with what can be seen from the literature review. The facts that serum kept refrigerated conserves the CGRP content up to 24 h and that moderate exercise does not exert a modulation effect on the concentrations will ease the design of sample extraction and processing protocols. Also, we point out that storage time should be controlled as a new way to ensure the validity of results, probably by the simultaneous enrolling of all the subjects included in the study and/or by assaying their samples within similar time-ranges from the extraction. Ultimately, we have shown that alpha and beta-CGRP should be analysed separately as the isoforms do not correlate their concentrations and it has been illustrated in the literature that these can have different behaviours within the same disorder.

Overall, this work has brought new methodological data to progress in our way to evaluate the actual role of CGRP as a migraine biomarker at the same time it has evaluated the previous advances with a critical point of view, trying to produce a constructive criticism that will help to progress in this challenging topic.

Availability of data and materials

No datasets were generated or analysed during the current study.

Abbreviations

Bicinchoninic acid protein assay

Chronic daily headache

Calcitonin gene-related peptide

Confidence interval

Chronic migraine

Combined contraception

Cerebrospinal fluid

Cubital vein

Enzyme immune assay

Enzyme-linked immunosorbent assay

Episodic migraine

Gingival crevicular fluid

Healthy controls

International classification of headache disorders 3rd edition

Interquartile range

Jugular vein

Migraine with aura

Medication overuse

Migraine without aura

Pituitary adenylate cyclase activating peptide-38

Pharmacokinetics

Post menopause

Radioimmuno assay

Regular menstrual cycle

Standard deviation

Standard error of mean

Vasoactive intestinal peptide

Without aura

Olesen J (2018) Headache Classification Committee of the International Headache Society (IHS) The International Classification of Headache Disorders, 3rd edition. Cephalalgia 38:1–211

Hong J Bin, Lange KS, Overeem LH, et al (2023) A Scoping Review and Meta-Analysis of Anti-CGRP Monoclonal Antibodies: Predicting Response. Pharmaceuticals 16:934. https://doi.org/10.3390/ph16070934

Zobdeh F, ben Kraiem A, Attwood MM, et al (2021) Pharmacological treatment of migraine: Drug classes, mechanisms of action, clinical trials and new treatments. Br J Pharmacol 178:4588–4607

Article   CAS   PubMed   Google Scholar  

Hepp Z, Dodick DW, Varon SF et al (2015) Adherence to oral migraine-preventive medications among patients with chronic migraine. Cephalalgia 35:478–488. https://doi.org/10.1177/0333102414547138

Article   PubMed   Google Scholar  

Edvinsson L, Haanes KA, Warfvinge K, DiN K (2018) CGRP as the target of new migraine therapies - Successful translation from bench to clinic. Nat Rev Neurol 14:338–350. https://doi.org/10.1038/s41582-018-0003-1

Gago-Veiga J-M, MGEAPGPMMÁG-G, Castañeda S, (2022) Treatment of migraine with monoclonal antibodies. Expert Opin Biol Ther 22:707–716. https://doi.org/10.1080/14712598.2022.2072207

Negro A, Martelletti P (2019) Gepants for the treatment of migraine. Expert Opin Investig Drugs 28:555–567. https://doi.org/10.1080/13543784.2019.1618830

Angus-Leppan H (2013) Migraine: mimics, borderlands and chameleons. Pract Neurol 13:308–318. https://doi.org/10.1136/practneurol-2012-000502

Amara SG, Jonas V, Rosenfeld MG, et al (1982) Alternative RNA processing in calcitonin gene expression generates mRN As encoding different polypeptide products. Nature 298:240–244. https://doi.org/10.1038/298240a0

Brain SD, Macintyre lAIN, Williams TJ, et al (1986) A second form of human calcitonin gene-related peptide which is a potent vasodilator. Eur J Phamacol 124:349–352. https://doi.org/10.1016/0014-2999(86)90238-4

Russo AF, Hay DL (2023) CGRP physiology, pharmacology, and therapeutic targets: migraine and beyond. Physiol Rev 103:1565–1644

Schütz B, Mauer D, Salmon AM et al (2004) Analysis of the cellular expression pattern of β-CGRP in α-CGRP-deficient mice. J Comp Neurol 476:32–43. https://doi.org/10.1002/cne.20211

Mulderry PK, Ghatei MA, Spokes RA et al (1988) Differential expression of α-CGRP and β-CGRP by primary sensory neurons and enteric autonomic neurons of the rat. Neuroscience 25:195–205. https://doi.org/10.1016/0306-4522(88)90018-8

Goadsby PJ, Edvinsson L, Ekman R (1990) Vasoactive peptide release in the extracerebral circulation of humans during migraine headache. Ann Neurol 28:183–187. https://doi.org/10.1002/ana.410280213

Gallai V, Sarchielli P, Floridi A, et al (1995) Vasoactive peptide levels in the plasma of young migraine patients with and without aura assessed both interictally and ictally. Cephalalgia 15:384–390. https://doi.org/10.1046/j.1468-2982.1995.150538

Sarchielli P, Alberti A, Codini M et al (2000) Nitric oxide metabolites, prostaglandins and trigeminal vasoactive peptides in internal jugular vein blood during spontaneous migraine attacks. Cephalalgia 20:907–918

Ashina M, Bendtsen L, Jensen R, et al (2000) Evidence for increased plasma levels of calcitonin gene-related peptide in migraine outside of attacks. Pain 83:133–138. https://doi.org/10.1016/s0304-3959(00)00232-3

Sarchielli P, Alberti A, Floridi A, Gallai V (2001) Levels of nerve growth factor in cerebrospinal fluid of chronic daily headache patients. Neurology 57:132–134. https://doi.org/10.1212/WNL.57.1.132

Gallai V, Alberti A, Gallai B, et al (2003) Glutamate and nitric oxide pathway in chronic daily headache: evidence from cerebrospinal fluid. Cephalalgia 23:166–174. https://doi.org/10.1046/j.1468-2982.2003.00552.x

Bellamy JL, Cady RK, Durham PL (2006) Salivary levels of CGRP and VIP in rhinosinusitis and migraine patients. Headache 46:24–33. https://doi.org/10.1111/j.1526-4610.2006.00294.x

Fusayasu E, Kowa H, Takeshima T et al (2007) Increased plasma substance P and CGRP levels, and high ACE activity in migraineurs during headache-free periods. Pain 128:209–214. https://doi.org/10.1016/j.pain.2006.09.017

Sarchielli P, Pini LA, Coppola F et al (2007) Endocannabinoids in chronic migraine: CSF findings suggest a system failure. Neuropsychopharmacology 32:1384–1390. https://doi.org/10.1038/sj.npp.1301246

Jang M-U, Park J-W, Kho H-S et al (2011) Plasma and saliva levels of nerve growth factor and neuropeptides in chronic migraine patients. Oral Dis 17:187–193. https://doi.org/10.1111/j.1601-0825.2010.01717.x

Rodríguez-Osorio X, Sobrino T, Brea D et al (2012) Endothelial progenitor cells: A new key for endothelial dysfunction in migraine. Neurology 79:474–479. https://doi.org/10.1212/WNL.0b013e31826170ce

Cernuda-Morollón E, Larrosa D, Ramón C et al (2013) Interictal increase of CGRP levels in peripheral blood as a biomarker for chronic migraine. Neurology 81:1191–1196. https://doi.org/10.1212/WNL.0b013e3182a6cb72

Cernuda-Morollõn E, Martínez-Camblor P, Ramõn C et al (2014) CGRP and VIP levels as predictors of efficacy of onabotulinumtoxin type A in chronic migraine. Headache 54:987–995. https://doi.org/10.1111/head.12372

Fekrazad R, Sardarian A, Azma K et al (2018) Interictal levels of calcitonin gene related peptide in gingival crevicular fluid of chronic migraine patients. Neurol Sci 39:1217–1223. https://doi.org/10.1007/s10072-018-3340-3

Domínguez C, Vieites-Prado A, Pérez-Mato M et al (2018) CGRP and PTX3 as Predictors of Efficacy of Onabotulinumtoxin Type A in Chronic Migraine: An Observational Study. Headache 58:78–87. https://doi.org/10.1111/head.13211

Leira Y, Ameijeira P, Domínguez C et al (2019) Periodontal inflammation is related to increased serum calcitonin gene-related peptide levels in patients with chronic migraine. J Periodontol 90:1088–1095. https://doi.org/10.1002/JPER.19-0051

Han D (2019) Association of serum levels of calcitonin gene-related peptide and cytokines during migraine attacks. Ann Indian Acad Neurol 22:277–281. https://doi.org/10.4103/aian.AIAN_371_18

Article   PubMed   PubMed Central   Google Scholar  

Kamm K, Straube A, Ruscheweyh R (2019) Calcitonin gene-related peptide levels in tear fluid are elevated in migraine patients compared to healthy controls. Cephalalgia 39:1535–1543. https://doi.org/10.1177/0333102419856640

Pérez-Pereda S, Toriello-Suárez M, Ocejo-Vinyals G et al (2020) Serum CGRP, VIP, and PACAP usefulness in migraine: a case–control study in chronic migraine patients in real clinical practice. Mol Biol Rep 47:7125–7138. https://doi.org/10.1007/s11033-020-05781-0

Irimia P, Martínez-Valbuena I, Mínguez-Olaondo A et al (2021) Interictal amylin levels in chronic migraine patients: A case-control study. Cephalalgia 41:604–612. https://doi.org/10.1177/0333102420977106

Vural S, Albayrak L (2022) Can calcitonin gene-related peptide (CGRP) and pentraxin-3 (PTX-3) be useful in diagnosing acute migraine attack? J Recept Signal Transduction 42:562–566. https://doi.org/10.1080/10799893.2022.2097264

Article   CAS   Google Scholar  

Alpuente A, Gallardo VJ, Asskour L et al (2022) Salivary CGRP can monitor the different migraine phases: CGRP (in)dependent attacks. Cephalalgia 42:186–196

Alpuente A, Gallardo VJ, Asskour L et al (2022) Salivary CGRP and erenumab treatment response: towards precision medicine in migraine. Ann Neurol 92:846–859. https://doi.org/10.1002/ana.26472

Liu J, Wang G, Dan Y, Liu X (2022) CGRP and PACAP-38 play an important role in diagnosing pediatric migraine. J Headache Pain 23:1–13. https://doi.org/10.1186/s10194-022-01435-7

Gárate G, González-Quintanilla V, González A et al (2023) Serum Alpha and Beta-CGRP Levels in chronic migraine patients before and after monoclonal antibodies against CGRP or its receptor. Ann Neurol 94:285–294. https://doi.org/10.1002/ana.26658

Goadsby PJ, Edvinsson L (1993) The Trigreminovascular System and Migraine: Studies Characterizing Cerebrovascular and Neuropeptide Changes Seen in Humans and Cats. Annals of Neurology 33:48–56. https://doi.org/10.1002/ana.410330109

Juhasz G, Zsombok T, Jakab B, et al (2005) Sumatriptan causes parallel decrease in plasma calcitonin gene-related peptide (CGRP) concentration and migraine headache during nitroglycerin induced migraine attack. Cephalalgia 25:179–183. https://doi.org/10.1111/j.1468-2982.2005.00836.x

Sarchielli P, Pini LA, Zanchin G et al (2006) Clinical-biochemical correlates of migraine attacks in rizatriptan responders and non-responders. Cephalalgia 26:257–265. https://doi.org/10.1111/j.1468-2982.2005.01016.x

Cady RK, Vause CV, Ho TW et al (2009) Elevated saliva calcitonin gene-related peptide levels during acute migraine predict therapeutic response to rizatriptan. Headache 49:1258–1266. https://doi.org/10.1111/j.1526-4610.2009.01523.x

Cady R, Turner I, Dexter K et al (2014) An exploratory study of salivary calcitonin gene-related peptide levels relative to acute interventions and preventative treatment with onabotulinumtoxinA in chronic migraine. Headache 54:269–277. https://doi.org/10.1111/head.12250

Cernuda-Morollón E, Ramón C, Martínez-Camblor P et al (2015) OnabotulinumtoxinA decreases interictal CGRP plasma levels in patients with chronic migraine. Pain 156:820–824. https://doi.org/10.1097/j.pain.0000000000000119

Lassen LH, Haderslev PA, Jacobsen VB et al (2002) CGRP may play a causative role in migraine. Cephalalgia 22:54–61. https://doi.org/10.1046/j.1468-2982.2002.00310.x

Nicolodi M, Bianco E Del (1990) Sensory neuropeptides (substance P, calcitonin gene-related peptide) and vasoactive intestinal polypeptide in human saliva: their pattern in migraine and cluster headache. Cephalalgia 10:39–50. https://doi.org/10.1046/j.1468-2982.1990.1001039.x

Friberg L, Olesen J, Skyhøi Olsen T et al (1994) Absence of vasoactive peptide release from brain to cerebral circulation during onset of migraine with aura. Cephalalgia 14:47–54

Tvedskov JF, Lipka K, Ashina M et al (2005) No increase of calcitonin gene-related peptide in jugular blood during migraine. Ann Neurol 58:561–568. https://doi.org/10.1002/ana.20605

Lee MJ, Lee SY, Cho S, et al (2018) Feasibility of serum CGRP measurement as a biomarker of chronic migraine: a critical reappraisal. Journal of Headache and Pain 19:. https://doi.org/10.1186/s10194-018-0883-x

Latif R, Rafique N, Al AL et al (2021) Diagnostic accuracy of serum calcitonin gene-related peptide and apolipoprotein e in migraine: A preliminary study. Int J Gen Med 14:851–856. https://doi.org/10.2147/IJGM.S303350

Hanci F, Kilinc YB, Kilinc E et al (2021) Plasma levels of vasoactive neuropeptides in pediatric patients with migraine during attack and attack-free periods. Cephalalgia 41:166–175. https://doi.org/10.1177/0333102420957588

de Vries Lentsch S, Garrelds IM, Danser AHJ, et al (2022) Serum CGRP in migraine patients using erenumab as preventive treatment. J Headache Pain 23. https://doi.org/10.1186/s10194-022-01483-z

Goldstein ED, Gopal N, Badi MK et al (2023) CGRP, Migraine, and Brain MRI in CADASIL: A Pilot Study. Neurologist 28:231–236. https://doi.org/10.1097/NRL.0000000000000478

Neyal A, Ekmekyapar Fırat Y, Çekmen MB et al (2023) Calcitonin gene-related peptide and adrenomedullin levels during ictal and interictal periods in patients with migraine. Cureus. https://doi.org/10.7759/cureus.37843

Messlinger K, Vogler B, Kuhn A et al (2021) CGRP measurements in human plasma – a methodological study. Cephalalgia 41:1359–1373. https://doi.org/10.1177/03331024211024161

Kamm K (2022) CGRP and Migraine: What Have We Learned From Measuring CGRP in Migraine Patients So Far? Front Neurol 13:. https://doi.org/10.3389/fneur.2022.930383

Greco R, De Icco R, Demartini C, et al (2020) Plasma levels of CGRP and expression of specific microRNAs in blood cells of episodic and chronic migraine subjects: towards the identification of a panel of peripheral biomarkers of migraine? J Headache Pain 21. https://doi.org/10.1186/s10194-020-01189-0

Raffaelli B, Overeem LH, Mecklenburg J et al (2021) Plasma calcitonin gene-related peptide (CGRP) in migraine and endometriosis during the menstrual cycle. Ann Clin Transl Neurol 8:1251–1259. https://doi.org/10.1002/acn3.51360

Article   CAS   PubMed   PubMed Central   Google Scholar  

Raffaelli B, Storch E, Overeem LH et al (2023) Sex hormones and calcitonin gene-related peptide in women with migraine: a cross-sectional, matched cohort study. Neurology 100:E1825–E1835. https://doi.org/10.1212/WNL.0000000000207114

Frank F, Kaltseis K, Messlinger K, Broessner G (2022) Short Report of Longitudinal CGRP-Measurements in Migraineurs During a Hypoxic Challenge. Front Neurol 13. https://doi.org/10.3389/fneur.2022.925748

Raffaelli B, Terhart M, Fitzek MP et al (2023) Change of CGRP plasma concentrations in migraine after discontinuation of CGRP-(Receptor) monoclonal antibodies. Pharmaceutics 15:1–9. https://doi.org/10.3390/pharmaceutics15010293

Etefagh HH, Shahmiri SS, Melali H et al (2022) Bariatric surgery in migraine patients: CGRP level and weight loss. Obes Surg 32:3635–3640. https://doi.org/10.1007/s11695-022-06218-2

Juhasz G, Zsombok T, Modos EA et al (2003) NO-induced migraine attack: Strong increase in plasma calcitonin gene-related peptide (CGRP) concentration and negative correlation with platelet serotonin release. Pain 106:461–470. https://doi.org/10.1016/j.pain.2003.09.008

Pellesi L, Al-Karagholi MAM, De Icco R, et al (2022) Plasma Levels of CGRP During a 2-h Infusion of VIP in Healthy Volunteers and Patients With Migraine: An Exploratory Study. Front Neurol 13. https://doi.org/10.3389/fneur.2022.871176

Guo S, Vollesen ALH, Hansen YBL et al (2017) Part II: Biochemical changes after pituitary adenylate cyclase-activating polypeptide-38 infusion in migraine patients. Cephalalgia 37:136–147. https://doi.org/10.1177/0333102416639517

Abbas A, Moustafa R, Shalash A et al (2022) Serum CGRP changes following ultrasound-guided bilateral greater-occipital-nerve block. Neurol Int 14:199–206. https://doi.org/10.3390/neurolint14010016

Riesco N, Cernuda-Morollón E, Martínez-Camblor P et al (2017) Relationship between serum levels of VIP, but not of CGRP, and cranial autonomic parasympathetic symptoms: A study in chronic migraine patients. Cephalalgia 37:823–827. https://doi.org/10.1177/0333102416653232

Babapour M, Khorvash F, Rouhani MH, et al (2022) Effect of soy isoflavones supplementation on migraine characteristics , mental status and calcitonin gene - related peptide ( CGRP ) levels in women with migraine : results of randomised controlled trial. Nutr J 1–11. https://doi.org/10.1186/s12937-022-00802-z

Gárate G, Toriello M, González-Quintanilla V, et al (2023) Serum alpha-CGRP levels are increased in COVID-19 patients with headache indicating an activation of the trigeminal system. BMC Neurol 23. https://doi.org/10.1186/s12883-023-03156-z

Gárate G, Pascual M, Rivero M, et al (2023) Serum Calcitonin Gene-Related Peptide α and β Levels are Increased in COVID-19 Inpatients. Arch Med Res 54. https://doi.org/10.1016/j.arcmed.2022.12.002

Gárate G, Pascual M, Olmos JM, et al (2022) Increase in Serum Calcitonin Gene-Related Peptide β (CGRPβ) Levels in COVID-19 Patients with Diarrhea: An Underlying Mechanism? Dig Dis Sci 67. https://doi.org/10.1007/s10620-022-07473-0

Alhabbab RY (2018) Radioimmunoassay (RIA). Basic Serological Testing. Springer International Publishing, Cham, pp 77–81

Chapter   Google Scholar  

Sadat TM, Ahmed M (2022) Enzyme-Linked Immunosorbent Assay (ELISA). In: Christian SL (ed) Cancer Cell Biology: Methods and Protocols. Springer, US, New York, NY, pp 115–134

Google Scholar  

Wimalawansa SJ (1991) Circadian variation of plasma calcitonin gene-related peptide in man. J Neuroendocrinol 3:319–322. https://doi.org/10.1111/j.1365-2826.1991.tb00281.x

de Vries LS, Rubio-Beltrán E, MaassenVanDenBrink A (2021) Changing levels of sex hormones and calcitonin gene-related peptide (CGRP) during a woman’s life: Implications for the efficacy and safety of novel antimigraine medications. Maturitas 145:73–77. https://doi.org/10.1016/j.maturitas.2020.12.012

Jonhagen S, Ackermann P, Saartok T, Renstrom PA (2006) Calcitonin gene related peptide and neuropeptide Y in skeletal muscle after eccentric exercise: A microdialysis study. Br J Sports Med 40:264–267. https://doi.org/10.1136/bjsm.2005.022731

Tarperi C, Sanchis-Gomar F, Montagnana M et al (2020) Effects of endurance exercise on serum concentration of calcitonin gene-related peptide (CGRP): A potential link between exercise intensity and headache. Clin Chem Lab Med 58:1707–1712. https://doi.org/10.1515/cclm-2019-1337

Kraenzlin ME, Ch’ng JLC, Mulderry PK, et al (1985) Infusion of a novel peptide, calcitonin gene-related peptide (CGRP) in man. Pharmacokinetics and effects on gastric acid secretion and on gastrointestinal hormones. Regul Pept 10:189–197. https://doi.org/10.1016/0167-0115(85)90013-8

Reich A, Orda A, Wiśnicka B, Szepietowski JC (2007) Plasma concentration of selected neuropeptides in patients suffering from psoriasis. Exp Dermatol 16:421–428. https://doi.org/10.1111/j.1600-0625.2007.00544.x

Smillie SJ, Brain SD (2011) Calcitonin gene-related peptide (CGRP) and its role in hypertension. Neuropeptides 45:93–104

Li FJ, Zou YY, Cui Y et al (2013) Calcitonin gene-related peptide is a promising marker in ulcerative colitis. Dig Dis Sci 58:686–693. https://doi.org/10.1007/s10620-012-2406-y

Edvinsson L, Ekman R, Goadsby PJ (2010) Measurement of vasoactive neuropeptides in biological materials: Problems and pitfalls from 30 years of experience and novel future approaches. Cephalalgia 30:761–766

Krause A, Lott D, Dingemanse J (2021) Estimation of attainment of steady-state conditions for compounds with a long half-life. J Clin Pharmacol 61:82–89. https://doi.org/10.1002/jcph.1701

Cho HY, Choi GW, Lee YB (2019) Interpretation of non-clinical data for prediction of human pharmacokinetic parameters: In vitro-in vivo extrapolation and allometric scaling. Pharmaceutics 11. https://doi.org/10.3390/pharmaceutics11040168

Domschke S, Domschke W, Bloom2 SR, et al (1978) Vasoactive intestinal peptide in man: pharmacokinetics, metabolic and circulatory effects. Gut 19:1049–1053. https://doi.org/10.1136/gut.19.11.1049

Mathiesen DS, Lund A, Holst JJ et al (2022) Amylin and calcitonin – physiology and pharmacology. Eur J Endocrinol 186:R93–R111

Birk S, Sitarz JT, Petersen KA et al (2007) The effect of intravenous PACAP38 on cerebral hemodynamics in healthy volunteers. Regul Pept 140:185–191. https://doi.org/10.1016/j.regpep.2006.12.010

Al-Keilani MS, Almomani BA, Al-Sawalha NA et al (2022) Significance of serum VIP and PACAP in multiple sclerosis: an exploratory case–control study. Neurol Sci 43:2621–2630. https://doi.org/10.1007/s10072-021-05682-5

Ochoa-Callejero L, García-Sanmartín J, Villoslada-Blanco P, et al (2021) circulating levels of calcitonin gene-related peptide are lower in COVID-19 patients. J Endocr Soc 5. https://doi.org/10.1210/jendso/bvaa199

Wyon Y, Hammar M, Theodorsson E, Lundeberg T (1998) Effects of physical activity and acupuncture on calcitonin gene-related peptide immunoreactivity in different parts of the rat brain and in cerebrospinal fluid, serum and urine. Acta Physiol Scand 162:517–522. https://doi.org/10.1046/j.1365-201X.1998.0317e.x

Parnow A, Gharakhanlou R, Gorginkaraji Z, et al (2012) Effects of endurance and resistance training on calcitonin gene-related peptide and acetylcholine receptor at slow and fast twitch skeletal muscles and sciatic nerve in male wistar rats. Int J Pept 2012. https://doi.org/10.1155/2012/962651

Kooshki R, Abbasnejad M, Shamsizadeh A, et al (2020) Physical exercise enhances vulnerability to migraine headache associated with CGRP up-expression in trigeminal nucleus caudalis of stressed rats. Neurol Res 42:952–958. https://doi.org/10.1080/01616412.2020.1794243

Jasim H, Carlsson A, Hedenberg-Magnusson B, et al (2018) Saliva as a medium to detect and measure biomarkers related to pain. Sci Rep 8. https://doi.org/10.1038/s41598-018-21131-4

Download references

Acknowledgements

Not applicable.

This study has been founded by Instituto de Salud Carlos III (ISCII) through the project PI20/01358, co-funded by Fondos Europeos de Desarrollo Regional (FEDER), “Una manera de hacer Europa”, and through the project PMP22/00183, co-founded by the Recovery and Resilience Plan by The European Union NextGenerationUE.

Author information

Authors and affiliations.

Instituto de Investigación Marqués de Valdecilla (IDIVAL), Hospital Universitario Marqués de Valdecilla & Universidad de Cantabria, Santander, Spain

Gabriel Gárate, Julio Pascual, Marta Pascual-Mato, Jorge Madera, María Muñoz-San Martín & Vicente González-Quintanilla

You can also search for this author in PubMed   Google Scholar

Contributions

GG, VGQ, JP designed the study, collected and analysed the data and wrote the manuscript. VGQ, JM, MPM and JP recruited participants for the study. All authors reviewed, contributed, and edited the final draft. All authors approved the final version.

Corresponding author

Correspondence to Gabriel Gárate .

Ethics declarations

Ethics approval and consensus to participate.

The study was approved by the Ethics Committee of Cantabria and its approval has been published in the record 28/2020 of December 11, 2020. All participants gave written informed consent for their inclusion in the study.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Gárate, G., Pascual, J., Pascual-Mato, M. et al. Untangling the mess of CGRP levels as a migraine biomarker: an in-depth literature review and analysis of our experimental experience. J Headache Pain 25 , 69 (2024). https://doi.org/10.1186/s10194-024-01769-4

Download citation

Received : 18 March 2024

Accepted : 09 April 2024

Published : 29 April 2024

DOI : https://doi.org/10.1186/s10194-024-01769-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

The Journal of Headache and Pain

ISSN: 1129-2377

• Submission enquiries: [email protected]

SYSTEMATIC REVIEW article

Evidence quality assessment of acupuncture intervention for stroke hemiplegia: an overview of systematic reviews and meta-analyses.

• 1 First Clinical Medical College, Shandong University of Traditional Chinese Medicine, Jinan, China
• 2 Dongying People’s Hospital (Dongying Hospital of Shandong Provincial Hospital Group), Dongying, China
• 3 Department of Geriatric Medicine, Affiliated Hospital of Shandong University of Traditional Chinese Medicine, Jinan, China

Objective: Summarize the conclusions of the systematic review/meta-analysis of the clinical efficacy of acupuncture for stroke hemiplegia, and evaluate its methodological quality and the quality of evidence.

Methods: Two researchers searched and extracted 8 databases for systematic reviews (SRs)/meta-analyses (MAs), and independently assessed the methodological quality, risk of bias, reporting quality, and quality of evidence of SRs/MAs included in randomized controlled trials (RCTs). Tools used included the Assessment of Multiple Systematic Reviews 2 (AMSTAR-2), the Risk of Bias in Systematic (ROBIS) scale, the list of Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA), and the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system. The search time is from database building to July 2023.

Results: A total of 11 SRs/MAs were included, including 2 English literature and 9 Chinese literature, with all study sites in China. AMSTAR-2 evaluation results showed that the methodological quality of 11 articles was rated as very low quality; Based on the ROBIS evaluation results, the SRs/MAs was assessed as a high risk of bias; According to the results of the PRISMA checklist evaluation, most of the SRs/MAs reports are relatively complete; according to GRADE system, 42 outcomes were extracted from the included SRs/MAs for evaluation, of which 1 was rated as high-quality evidence, 14 as moderate-quality evidence, 14 as low-quality evidence, and 13 as very low-quality evidence.

Conclusion: The available evidence indicates that acupuncture has certain clinical efficacy in the treatment of stroke hemiplegia. However, there are still some limitations to this study, such as the lower quality of SRs/MAs methodologies and evidence included, and more high-quality studies are needed to verify them.

1 Introduction

Stroke is a common cerebrovascular disease, also often known as “stroke” ( 1 ), belongs to a common clinical disease. It is characterized by high morbidity, high mortality rate, high disability rate and many complications ( 2 , 3 ), and the prevalence rate is increasing year by year ( 4 ). With the improvement of medical technology level, the mortality of stroke patients decreased significantly, but still about 80% of patients in the disease recovery will leave a variety of different degrees of neurological and limb dysfunction ( 5 ), including hemiplegia is one of the common sequelae of stroke ( 6 ), not only affect the quality of life, also bring heavy burden to the family and society. Therefore, taking timely and effective rehabilitation treatment to promote the recovery of limb movement function of stroke patients and improve the self-care ability has become an important problem to be solved in clinical practice.

Acupuncture therapy for stroke and its sequelae has received much attention due to its long history and unique treatment experience ( 7 ). Related studies have found that acupuncture can improve the clinical performance of patients with by promoting cerebral blood circulation, affecting astrocytes, blood rheology, and neuronal plasticity ( 8 , 9 ). In recent years, a large number of clinical randomized controlled trials showed that acupuncture is effective in the treatment of stroke hemiplegia ( 10 , 11 ). At present, several systematic reviews (SRs)/meta-analyses (MAs) have comprehensively analyzed the relevant clinical data, which provides a theoretical basis for the research of acupuncture for stroke hemiplegia. However, the quality of the evidence obtained by the secondary institute is influenced by the quality of the original data and the subjective factors of the researchers, and whether the method evaluation is standardized, objective and fair, and can fully and effectively evaluate the clinical efficacy of acupuncture in stroke hemiplegia. To provide higher evidence support, this study will re-evaluate the existing SRs/MAs of acupuncture for the treatment of stroke hemiplegia at home and abroad. Using tools including the Assessment of Multiple Systematic Reviews 2 (AMSTAR-2) ( 12 , 13 ), the Risk of Bias in Systematic (ROBIS) ( 14 ) scale, the list of Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) ( 15 , 16 ), and the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) ( 17 – 19 ) system for methodology, risk of bias, reporting quality and evidence quality rating. The existing research results are systematically summarized in order to provide a more accurate evidence-based basis for subsequent clinical decisions ( 20 ).

2 Materials and methods

The methodology of this overview follows the Cochrane Handbook, and the report of this overview is in line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 checklist. This overview has been registered with the PROSPERO website (Registration number: PROSPERO CRD42024495984).

2.1 Inclusion and exclusion criteria

Contributions to acupuncture treatment of post-stroke paralysis based on existing RCTs. The criteria for inclusion of SRs/MAs in this overview are as follows: (1) Based on the SRs/MAs of randomized controlled trials (RCTs), the published language was in Chinese or English. (2) Patients were diagnosed with paralysis after stroke, regardless of the type of stroke (hemorrhagic stroke or ischemic stroke), the diseased brain area, the affected limb, gender, age, ethnicity, and nationality. (3) The intervention measures in the treatment group were acupuncture or acupuncture combined with other treatments, while the control group received non-acupuncture therapy. (4) The main outcome measures are: ① fugl-meyer as-sessment of low limb (FMA-L); ② Barthel Index (BI); ③ Clinic Neurological Function Deficit Scale (NDS); ④ The Modified Ashworth Spasticity Rating Scale score (MAS); ⑤ Clinical effectiveness; ⑥ Brunnstorm stage score; ⑦ The Clinical Spasticity Index score (CSI); ⑧ The Berg Balance Scale score; ⑨ functional comprehensive assessment score (FCA).

Exclusion criteria: Repeated published literature, animal studies, dissertations, conference articles, systematic review protocols and literature where raw data cannot be extracted.

2.2 Search strategy

A computer search was conducted across China National Knowledge Network (CNKI), Wanfang database, VIP database, Chinese biomedical literature database (CBM) and PubMed, EMbase, Cochrane Library and other English databases, the search time from the establishment of the database to December 31, 2023. In addition, we manually supplemented the references and grey literature of the included studies. The retrieval method adopts the combination of subject words and free words. Key words include acupuncture, electric acupuncture, needling, head needle, body needle, flying needle, fire needle, acupuncture with warmed needle, acusector, prod, Post-stroke hemiplegia, stroke hemiplegia, cerebral infarction hemiplegia, spastic hemiplegia, flaccid hemiplegia, systematic review, meta analysis. Taking CNKI as an example, the specific search strategy is as follows: SU = (“acupuncture” + “electric acupuncture” + “needling” + “head needle” + “acupuncture with warmed needle” + “acusector” + “prod”) and SU = (“Post-stroke hemiplegia” + “stroke hemiplegia” + “cerebral infarction hemiplegia” + “spastic hemiplegia” + “flaccid hemiplegia”) and SU = (“systematic review” + “meta analysis”); Search strategy for the PubMed database see Table 1 .

Table 1 . Search strategy for the PubMed database.

2.3 Literature screening and data extraction

Two researchers (MX-F and RM-L) independently screened and extracted the literatures, and cross-checked them. Any disagreement during this process were subject to discussion and negotiation or the decision of a third expert (WL-G). The extracted literature information included: Authors, publication year, nationality, sample size, intervention measures, quality assessment tools and main conclusions.

2.4 Quality assessment

Two researchers (MX-F and RM-L) independently assessed the methodological and evidence quality of the included SRs/MAs, Any discrepancies were resolved by consensus or adjudication by a third author (WL-G).

2.4.1 Methodological quality assessment

The methodological quality of the included SRs/MAs was evaluated using the Assessment System for Evaluating Methodological Quality 2 (AMSTAR-2). The AMSTAR-2 scale contains 16 items that can be answered with a “yes,” “partially yes” or “no.” Item 1: Whether the research question and inclusion criteria include the elements of PICO; 2: Whether the study method is determined before its implementation, Whether the report is consistent with the plan; 3: Whether to explain the reasons for the study type; 4: Whether the search literature is comprehensive; 5: Whether the literature is independently screened by two people; 6: Whether the data is extracted by two people; 7: Whether to provide the list of documents and reasons for exclusion; 8: Whether to describe the basic characteristics of the included study in detail; 9: Whether the risk of included study bias is reasonably assessed; 10: Whether to report the source of research funding; 11: whether appropriate statistical methods are used to combine and analyze the results during the meta-analysis; 12: whether to consider the risk of bias in the meta-analysis or other evidence integration; 13: Whether the inclusion risk of study bias was considered when interpreting the results; 14: Whether the heterogeneous results are interpreted or discussed; 15: whether the publication bias was investigated during the quantitative synthesis, Whether to discuss its impact on the results; 16: Whether to report conflicts of interest and sources of funding.

According to the evaluation criteria, it can be rated “high,” “moderate,” “low” and “very low,” and 7 out of 16 items in the tool (2, 4, 7, 9, 11, 13 and 15) are critical areas.

2.4.2 Risk of bias assessment

The Risk of Bias in Systematic Review (ROBIS), which aims at the bias risk of system evaluation, is not only used to evaluate the bias risk in the process of making and interpreting the results of multiple SRs/MAs such as intervention, diagnosis, etiology, and prognosis, etc., but also used to evaluate the correlation between the SRs/MAs problems and the practical problems to be solved by users. ROBIS scale was used in this overview to evaluate the risk of bias in the inclusion of SRs/MAs and the evaluation was carried out in three stages. ROBIS is useful for assessing the extent of bias in four domains: (1) Eligibility criteria for each study; (2) The identification and selection of studies; (3) Data collection and study appraisal; and (4) Overall synthesis and major findings. Within each domain, specific questions were used to determine the risk of bias, which was rated as “low,” “high,” or “unclear.”

2.4.3 Report quality assessment

SR/MA is an important evidence to guide clinical practice. The clarity of its report affects the realization of its clinical value. Standard reports can reduce the bias between actual research results and published results, and increase the transparency of articles. The PRISMA Report Guide is designed to help authors improve the quality of their reports, obtain key information, and improve readability and credibility.

The quality of each SR/MA report for the included SRs/MAs was assessed by the PRISMA 2020 checklist, and each of the 27 items included in PRISMA 2020 was scored as “yes,” “partially yes” or “no.”

2.4.4 Evidence quality assessment

In order to be useful to decision-makers, clinicians and patients, the SRs/MAs must provide not only the effect estimates of each result, but also the information needed to judge the correctness of these effect estimates. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) ( 19 ) provides a structure for SRs/MAs and clinical practice guidelines to ensure that it addresses all key issues of outcome evidence quality evaluation related to a specific issue in a consistent and systematic manner.

The quality of evidence for each SR/MA outcome was evaluated by the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system. Since the initial quality of evidence for RCTs is high, the quality of evidence for the outcomes of the study was evaluated based on downgrading factors such as limitations, inconsistencies, indirectness, imprecision, and publication bias of the study. According to the level of downgrade, they are rated as “high,” “moderate,” “low” and “very low.”

3.1 Literature search and screening results

A total of 82 related literatures were retrieved, after 32 duplicate literatures were deleted, 28 unrelated literatures were deleted by reading titles and abstracts, then 5 literatures with inconsistent intervention measures were removed, and finally 6 literatures of systematic review protocols and literatures with missing data were deleted, 11 literatures ( 21 – 31 ) were finally included after layer-by-layer screening. The specific screening process is shown in Figure 1 .

Figure 1 . Flow diagram of literature screening process.

3.2 The basic characteristics of the included literature

Of the 11 included SRs/MAs, 2 of which were published in English and 9 in Chinese, all studies were conducted in China. The number of RCTs in the included SRs/MAs ranged from 5 to 27, with a sample size of between 750 and 1963. For quality evaluation, 6 ( 21 , 22 , 24 , 28 , 30 , 31 ) SRs/MAs used Cochrane risk of bias tool, and 5 ( 23 , 25 – 27 , 29 ) SRs/MAs used the Jadad scale. The specific information is shown in Table 2 .

Table 2 . Basic information of the included SRs/MAs.

3.3 Results on SRs/MAs quality assessment

3.3.1 results of the methodological quality.

The included SRs/MAs was assessed for its methodological quality by the AMSTAR-2. The results showed that the quality of the 9 included SRs/MAs were very low, since none of the SRs/MAs included in the quality assessment met the requirement of key Item 7 (none of the SRs/MAs provided an exclusion list). A summary of the results is shown in Table 3 .

Table 3 . Result of the AMSTAR-2 assessments.

3.3.2 Results of the risk of bias assessment

For ROBIS, Phase 1 assesses the relevance of the research topic, and all SRs/MAs were rated as low risk of bias. In the Phase 2: Domain 1 assessed the study eligibility criteria, and all SRs/MAs were rated as low risk of bias; Domain 2 assessed the identification and selection studies, and 3 SRs/MAs were rated as low risk of bias; Domain 3 assessed the collection and study appraisal, 3 SRs/MAs were rated as high risk of bias; and Domain 4 assessed the synthesis and findings, and 8 SRs/MAs were rated as high risk of bias. Phase 3 considered the overall risk of bias in the reviews, and all SRs/MAs were rated as high risk of bias. The ROBIS scale evaluation results are shown in Table 4 .

Table 4 . Results of the ROBIS assessments.

3.3.3 Reporting quality of the included SRs/MAs

Reporting quality of the included SRs/MAs was evaluated as per the PRISMA checklist.6 of these items did not report at 100%. Mainly concentrated in the reporting rate of item 5 (protocol and registration) was 18%, the reporting rates of item 8 (search) and item 15 (study bias) were 82%, the item 27 (funding) reporting rate was 73%. The results of the PRISMA checklist assessment are shown in Table 5 .

Table 5 . Results of the PRISMA checklist.

3.3.4 Results of the quality of the evidence

A meta-analysis of the 42 outcomes in the study was performed, and the GRADE system was used to assess the quality of each outcome.1 outcome measure was high evidence quality, 14 moderate, 14 low and 13 very low evidence quality. Longitudinal analysis was performed for each degradation factor while horizontal evaluation was performed for the included systematic review/meta-analysis outcome indicators. We found that inconsistency ( n  = 29) was the main factor in reducing the quality of evidence, followed by publication bias ( n  = 28) and limitation ( n  = 23). Imprecision ( n  = 10) also has an impact on the evidence quality of the article. The results of evidence quality are shown in Table 6 .

Table 6 . Results of evidence quality.

3.3.5 Summary of results

The outcome measures extracted from the included studies are listed in Table 6 .

6 SRs/MAs ( 21 – 25 , 28 ) reported the BI. 1 SR/MA ( 21 ) conducted subgroup analyses, yielding the following results: (1) The therapeutic efficacy of acupuncture as a standalone treatment was superior to that of other conventional rehabilitation therapies; (2) The therapeutic efficacy of acupuncture as a standalone treatment was superior to non-acupuncture treatments; (3) The combined application of acupuncture and other conventional rehabilitation treatments demonstrated superior therapeutic efficacy compared to other conventional rehabilitation treatments; (4) Acupuncture as a standalone treatment exhibited superior therapeutic efficacy compared to conventional western medical treatments. Furthermore, 5 SRs/MAs ( 22 – 25 , 28 ) reported that the combined application of acupuncture and non-acupuncture treatments showed superior therapeutic efficacy compared to non-acupuncture treatments.

3.3.5.2 Clinical effectiveness

6 SRs/MAs ( 21 – 23 , 25 – 27 ) reported the Clinical effectiveness. 1 SR/MA ( 21 ) conducted subgroup analyses, revealing the following results: (1) The combined application of acupuncture and other conventional rehabilitation treatments demonstrated superior therapeutic efficacy compared to other conventional rehabilitation treatments; (2) Acupuncture as a standalone treatment exhibited superior therapeutic efficacy compared to conventional western medical treatments. Furthermore, 5 SRs/MAs ( 22 , 23 , 25 – 27 ) reported that the combined application of acupuncture and non-acupuncture treatments showed superior therapeutic efficacy compared to non-acupuncture treatments.

3.3.5.3 NDs

3 SRs/MAs ( 21 , 22 , 31 ) reported the NDs. 1 SR/MA ( 21 ) conducted subgroup analyses, yielding the following results: (1) The combined application of acupuncture and other conventional rehabilitation treatments demonstrated superior therapeutic efficacy compared to other conventional rehabilitation treatments; (2) Acupuncture as a standalone treatment exhibited superior therapeutic efficacy compared to conventional western medical treatments. Additionally, 1 SR/MA ( 22 ) reported that the combined application of acupuncture and non-acupuncture treatments showed superior therapeutic efficacy compared to non-acupuncture treatments. Furthermore, 1 SR/MA ( 31 ) reported that acupuncture as a standalone treatment exhibited superior therapeutic efficacy compared to conventional western medical treatments.

3.3.5.4 MAS

4 SRs/MAs ( 21 – 24 ) reported the MAS. 1 SR/MA ( 21 ) reported that acupuncture as a standalone treatment demonstrated superior efficacy compared to other conventional rehabilitation therapies. 3 SRs/MAs ( 22 – 24 ) reported that the combined approach of acupuncture with non-acupuncture treatments exhibited superior efficacy over non-acupuncture treatments alone.

3.3.5.5 The Berg Balance Scale score

1 SR/MA ( 21 ) reported the Berg Balance Scale score. The results indicated that the efficacy of acupuncture as a standalone treatment surpassed that of other conventional rehabilitation therapies.

3.3.5.6 FMA-L

7 SRs/MAs ( 21 – 24 , 28 – 30 ) reported the FMA-L. 1 SR/MA ( 21 ) conducted subgroup analyses, yielding the following results: (1) The therapeutic efficacy of acupuncture as a standalone treatment was superior to that of other conventional rehabilitation therapies; (2) The combined application of acupuncture and other conventional rehabilitation treatments demonstrated superior therapeutic efficacy compared to using other conventional rehabilitation treatments alone; (3) Acupuncture as a standalone treatment exhibited superior therapeutic efficacy compared to conventional western medical treatments. 5 SRs/MAs ( 22 – 24 , 28 , 29 ) reported that the combined approach of acupuncture with non-acupuncture treatments exhibited superior efficacy compared to non-acupuncture treatments alone. Additionally, 1 SR/MA ( 30 ) reported that acupuncture as a standalone treatment demonstrated superior efficacy compared to non-acupuncture treatments.

3.3.5.7 Brunnstorm Stage score

3 SRs/MAs ( 22 , 29 , 31 ) reported the Brunnstorm Stage score. 2 SRs/MAs ( 22 , 29 ) reported that the combined approach of acupuncture with non-acupuncture treatments exhibited superior efficacy compared to non-acupuncture treatments alone. Additionally, 1 SR/MA ( 31 ) reported that acupuncture as a standalone treatment demonstrated superior efficacy compared to conventional western medical treatments.

3.3.5.8 CSI

1 SR/MA ( 22 ) reported the CSI. The results indicate that the combined approach of acupuncture with non-acupuncture treatments shows superior efficacy compared to non-acupuncture treatments alone.

3.3.5.9 FCA

1 SR/MA ( 29 ) reported the FCA. The results indicate that the combined approach of acupuncture with non-acupuncture treatments shows superior efficacy compared to non-acupuncture treatments alone.

4 Discussion

High-quality SRs/MAs is an important source for evidence-based medicine to obtain the best evidence, and is recognized as the cornerstone for evaluating clinical efficacy and formulating clinical guidelines and specifications ( 20 ). Overview of SRs/MAs is a comprehensive research method to comprehensively collect the treatment and diagnosis of the same disease or health problems, which can provide more concentrated users with high-quality evidence and provide better guidance for future clinical work ( 32 , 33 ). In this study, the included 11 SRs/MAs of acupuncture for stroke hemiplegia were reevaluated, and the included articles were evaluated by multiple quality evaluation methods to provide high-quality evidence-based evidence and decision basis for the clinical efficacy of acupuncture for hemiplegia, and provide further reference and evidence support for future clinical applications. At present, acupuncture has been widely used for the treatment of hemiparesis caused by stroke. When stroke patients are in a hemiparetic state, acupuncture can promote the recovery of muscle strength and muscle tone, improve limb motor function, and effectively prevent various complications ( 34 , 35 ). However, the current research on acupuncture therapy for stroke is extensive, with studies primarily focusing on spasticity and the recovery phase in investigating the mechanisms underlying acupuncture’s improvement of limb motor function.

In this study, the methodology of quality was evaluated by the included articles using AMSTAR-2, and all the 7 key items had some defects: (1) Only 2 of the 11 included research protocols were formulated in advance, which may affect the rigor of the systematic review formulation; (2) 9 articles do not manually search the grey literature or provide a complete search form, and cannot check whether the included literature search is comprehensive and accurate, and whether the data extraction is accurate and not reproducible; (3) None of the articles provides a list of excluded articles, which may reduce the credibility of the SRs/MAs; (4) 3 articles did not report any potential conflicts of interest or funding, which would bias the systematic evaluation. The absence of the above key items is the main factor leading to the lower results of the methodological quality evaluation.

The SRs/MAs bias risk results included in the ROBIS scale evaluation showed that incomplete literature search and data synthesis and incomplete presentation of results were the main factors leading to high bias. This study evaluated the quality of the report using the PRISMA checklist, which showed that lack of protocol registration, incomplete search strategy, and uncertainty about funding sources affected the rigor of the systematic overview as the highest evidence for diagnosis and treatment.

Quality grade evaluation of the included articles using the GRADE rating scale found that all the outcome measures were mostly of low quality (1) Some studies also have great risks in terms of heterogeneity and inaccuracy, mainly because the overlap of confidence intervals is too small or the sample size does not meet the optimal information sample size, or the confidence interval is too wide, which may be unreasonably related to the literature inclusion criteria and retrieval method; (2) More than half of the outcome indicators were downgraded, mainly because of the heterogeneity of the included SRs/MAs, and only a few conducted sensitivity and subgroup analyses; (3) In the evaluation of limitations, 23 outcome indicators were downgraded, because the randomization method and allocation of the included studies were not clear, only a small part of them described whether they was blind, and there was a high possibility of implementation and measurement bias; (4) Publication bias is also widespread.

To the best of our knowledge, this overview is the first article to use the SRs/MAs for clinical evaluation of acupuncture for stroke hemiplegia, which can provide comprehensive evidence reference for clinical practice. The assessment processes using tools such as AMSTAR-2, ROBIS, PRISMA, and grading have highlighted the limitations of SRs/MAs and RCTs, which can guide future high-quality clinical research. However, we must also acknowledge the limitations of this overview. Due to language restrictions, this study only included systematic reviews published in Chinese and English, and did not include Korean and Japanese databases that have similar backgrounds in traditional Chinese medicine research. Additionally, the search process actually overlooked manual searching, leading to some degree of selection bias. Furthermore, the literature screening and quality assessment were conducted by two researchers and were somewhat subjective. The number of included SRs/MAs was small, and the overall quality was not high.

4.1 Implications for future research

To reduce various biases such as selection bias, implementation bias, and measurement bias, further original studies should be conducted using large-sample, multicenter, long-term clinical randomized controlled trials based on evidence-based medicine standards. Attention should be given to properly and rationally implementing randomization, concealing allocation, and blinding. Additionally, to improve the quality of evidence, authors should register their study protocols before conducting SRs/MAs to ensure the rigor of their procedures. During the literature search and screening process, the excluded literature information and complete search strategies for all databases should be provided to ensure replicability. When quantitatively calculating effect sizes, individual study results should be systematically excluded one by one to ensure the stability of the results. Furthermore, a comprehensive assessment of publication bias will also improve the accuracy of the meta-analysis results. In order to develop a more effective treatment prescription and evaluation system, it is advisable for future studies to report detailed information regarding acupuncture treatment. This includes the number of needles used in each treatment session, specific acupuncture techniques employed, needle depth, needle reactions, treatment process, qualifications of the acupuncturist, years of experience of the assessors and clinicians involved, and the provision of a standardized and explicit treatment plan. Reporting these details will contribute to a better understanding of the acupuncture intervention and facilitate the replication and comparison of studies, ultimately leading to improved treatment outcomes and enhanced evaluation of acupuncture efficacy.

5 Conclusion

In conclusion, acupuncture treatment of stroke hemiplegia has certain efficacy, which can effectively improve the clinical manifestations of patients and reduce the disability rate. However, there are widespread problems of low methodological quality and low quality of evidence for SRs/MAs, which limits the reliability of the results and requires more high-quality original studies to provide evidence to support them.

Data availability statement

The original contributions presented in the study are included in the article/supplementary materials, further inquiries can be directed to the corresponding author.

Author contributions

MF: Writing – original draft, Writing – review & editing. BZ: Writing – original draft, Writing – review & editing, Methodology. CC: Data curation, Methodology, Writing – original draft. RL: Methodology, Validation, Writing – original draft. WG: Writing – original draft, Writing – review & editing.

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This work was supported by National Natural Science Foundation of China (No. 82204942); Natural Science Foundation of Shandong Province (No. ZR2022QH123); China Postdoctoral Science Foundation (No. 2022M721998).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Abbreviations

SRs, Systematic reviews; MAs, Meta-analyses; RCTs, Randomized controlled trials; AMSTAR-2, Assessment System for Evaluating Methodological Quality 2; ROBIS, Risk of Bias in Systematic reviews; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; GRADE, Grading of Recommendations Assessment Development and Evaluation; CNKI, Chinese National Knowledge Infrastructure; CBM, Chinese Biological Medicine Database; FMA-L, Fugl-meyer as-sessment of low limb; BI, Barthel Index; NDS, Clinic Neurological Function Deficit Scale; MAS, The Modified Ashworth Spasticity Rating Scale score; CSI, The Clinical Spasticity Index score; FCA, Functional comprehensive assessment score

1. Jiang, LS, Wang, Y, Chen, SF, Cui, LL, and Wang, ZY. Clinical observation on treatment of apoplectic hemiplegia by Tongdu Tiaoshen acupuncture combined with Buyang Huanwu decoction. J Chin Med . (2023) 41:206–8.

Google Scholar

2. Wang, ZW, Ma, ZY, Xue, SF, Li, W, Zuo, HJ, Wu, QN, et al. Consensus of experts on co-morbidity management of coronary heart disease and ischemic stroke at grass-roots level 2022. Chin J Cardiov Dis . (2022) 20:772–93.

3. Wang, YJ, Li, ZX, Gu, HQ, Zhai, Y, Jiang, Y, Zhou, Q, et al. Chinese stroke report 2020 (Chinese version) (3). Chin J Stroke . (2022) 17:675–82.

4. Li, JY, Zhang, JJ, An, Y, He, WW, Dong, XL, Zhang, GM, et al. Quality evaluation study of TCM guidelines and consensus for stroke disease. Chin J Tradit Chin Med Inform . (2022) 29:30–6.

5. Zhou, W, Hu, LD, Kong, FS, Li, PP, Zhuang, SJ, Cheng, YS, et al. Clinical observation on Qigui Tongluo formula combined with acupuncture in improving hemiplegia after stroke and its effect on endoplasmic reticulum stress-autophagy. J Chin Med . (2023) 41:89–93.

6. He, XP, and Hu, LL. Clinical study on the treatment of acupuncture combined with exercise imagination therapy for stroke hemiplegia. New Tradit Chin Med . (2021) 53:191–3.

7. Ouyang, YJ. Analysis of the effect of TCM acupuncture combined with rehabilitation training in treating stroke hemiplegia. China Pract Med . (2022) 17:166–8.

8. He, Y, and Jin, RJ. Progress in the treatment of hemiplegia limb spasm in stroke. Rehabil Theory Pract China . (2006) 10:863–6.

9. Hu, CH, Pan, M, and Zhou, M. Research progress of acupuncture treatment for stroke spastic hemiplegia. New Chin Med . (2021) 53:176–9.

10. Zhang, JH, Wang, D, and Liu, M. Overview of systematic reviews and meta-analyses of acupuncture for stroke. Neuroepidemiology . (2014) 42:50–8. doi: 10.1159/000355435

PubMed Abstract | Crossref Full Text | Google Scholar

11. Xie, M, and Wu, HD. Research progress and thinking on acupuncture treatment of hemiplegia after stroke. Sichuan Tradit Chin Med . (2020) 38:216–8.

12. Tao, H, Yang, LT, Ping, A, Quan, LL, Yang, X, Zhang, YG, et al. Interpretation of AMSTAR-2, a quality assessment tool for systematic evaluation of randomized or nonrandomized control studies. Chin J Evid Based Med . (2018) 18:101–8.

13. Shea, BJ, Reeves, BC, Wells, G, Thuku, M, Hamel, C, Moran, J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ . (2017) 21:j4008. doi: 10.1136/bmj.j4008

Crossref Full Text | Google Scholar

14. Wu, QF, Ding, HF, Deng, W, Yang, N, Wang, Q, Yao, L, et al. ROBIS: a new tool for assessing the risk of systematic reviews bias. Chin J Evid Based Med . (2015) 15:1454–7.

15. Page, MJ, Moher, D, Bossuyt, PM, Boutron, I, Hoffmann, TC, Mulrow, CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ . (2021) 29:n160. doi: 10.1136/bmj.n160

16. Gao, Y, Liu, M, Yang, KL, Ge, L, Li, L, Li, J, et al. System reviews report specification: comparative analysis and example interpretation of PRISMA 2020 and PRISMA 2009. Chin J Evid Based Med . (2021) 21:606–16.

17. Atkins, D, Best, D, Briss, PA, Eccles, M, Falck-Ytter, Y, Flottorp, S, et al. Grading quality of evidence and strength of recommendations. BMJ . (2004) 328:1490. doi: 10.1136/bmj.328.7454.1490

18. Technical Specifications Development Group for Real World Study in Traditional Chinese Medicine, China Association of Chinese Medicine. J Tradit Chin Med . (2022) 63:293–300.

19. Peng, XX, and Liu, YL. Evaluation of evidence quality in diagnostic accuracy studies: inconsistency, imprecision, publication bias, and others. Chin J Evid Based Pediatr . (2021) 16:472–4.

20. Zhao, C, Tian, GH, Zhang, XY, Wang, YP, Zhang, ZX, Shang, HC, et al. Connotation and thinking of evidence-based medicine developing to evidence-based science. Chin J Evid Based Med . (2019) 19:510–4.

21. Tu, Y, Peng, W, Wang, J, Hao, QH, Wang, Y, Li, H, et al. Acupuncture therapy on patients with flaccid hemiplegia after stroke: a systematic review and meta-analysis. Evid Based Complement Alternat Med . (2022) 2022:1–17. doi: 10.1155/2022/2736703

22. Li, HP, Zhai, YB, Xing, J, and Wang, JL. Meta-analysis of acupuncture treatment for spastic hemiplegia of upper limbs after stroke. World J Tradit Chin Med . (2022) 17:196–207+214.

23. Zhu, JM, Zhuang, R, He, J, Ding, YQ, Jiang, LL, et al. Meta-analysis of acupuncture combined with rehabilitation training for upper limb spasticity after stroke. J Integr Tradit Chin Western Med Cardiovasc Dis . (2021) 19:1892–8.

24. Fan, W, Kuang, X, Hu, J, Chen, X, Yi, W, Lu, L, et al. Acupuncture therapy for poststroke spastic hemiplegia: a systematic review and meta-analysis of randomized controlled trials. Complement Ther Clin Pract . (2020) 40:101176. doi: 10.1016/j.ctcp.2020.101176

25. Gou, Y, Wu, JG, Mo, XN, Xun, LX, et al. Meta-analysis of randomized controlled trial of Buyang Huanwu decoction combined with acupuncture versus Buyang Huanwu decoction alone in treating stroke hemiplegia. Asia-Pac Tradit Med . (2019) 15:153–6.

26. Wang, YY, Dai, LW, Qin, YP, Li, F, Yu, J, et al. Meta-analysis of therapeutic effect of acupuncture on stroke hemiplegia. Clin Res Tradit Chin Med . (2017) 9:8–11.

27. Wang, YQ, Yang, HS, Feng, HH, Li, YQ, Peng, XY, Xiao, CX, et al. Meta-analysis of randomized controlled trials of acupuncture for stroke hemiplegia. Chin J Basic Med . (2016) 22:1670–1672+1712.

28. Jin, D, and Wang, J. Meta-analysis of the clinical efficacy evaluation of acupuncture therapy for hemiplegia in stroke. New Tradit Chin Med . (2016) 48:284–6.

29. Song, Y, Li, Q, Wu, ZJ, and Wang, ZY. Meta-analysis of the effect of acupuncture therapy combined with rehabilitation training on the recovery of lower limb motor function in stroke patients with hemiplegia. J Nanjing Inst Phys Educ (Nat Sci Ed) . (2016) 15:49–57.

30. Lin, MQ, and Liu, WL. Systematic review of acupuncture treatment for hemiplegic gait in stroke patients. J Rehabil . (2015) 25:54–62.

31. Li, N, Feng, B, Zou, J, Liu, Y, et al. Meta-analysis of acupuncture for hemiplegia of stroke. J Chengdu Univ Tradit Chin Med . (2002) 2:37–39+64.

32. Wang, LJ, Zeng, Q, Xie, YJ, Chen, D, Yao, DY, Chen, X, et al. Normative evaluation of re-evaluation reports on acupuncture systems at home and abroad based on PRIO-harms scale. Chin Acupunct . (2020) 40:793–8.

33. Yang, KH, Liu, YL, Yuan, JQ, and Jiang, HL. Re-evaluation of system review in development and perfection. Chin J Evid-Based Pediatr . (2011) 6:54–7.

34. Lee, SH, and Lim, SM. Acupuncture for insomnia after stroke: a systematic review and meta-analysis. BMC Complement Altern Med . (2016) 16:228. doi: 10.1186/s12906-016-1220-z

35. Liu, S, Zhang, CS, Cai, Y, Guo, XF, Zhang, L, Xue, CL, et al. Acupuncture for post-stroke shoulder-hand syndrome: a systematic review and meta-analysis. Front Neurol . (2019) 26:433.

36. Du, YH, Li, J, Sun, DW, Liu, WH, Li, GP, Lin, X, et al. Study on the spectrum of modern acupuncture and moxibustion in China. Chin Acupunct . (2007) 5:373–8.

Keywords: acupuncture, stroke hemiplegia, systematic reviews, meta-analyses, overview

Citation: Fan M, Zhang B, Chen C, Li R and Gao W (2024) Evidence quality assessment of acupuncture intervention for stroke hemiplegia: an overview of systematic reviews and meta-analyses. Front. Neurol . 15:1375880. doi: 10.3389/fneur.2024.1375880

Received: 26 January 2024; Accepted: 08 April 2024; Published: 30 April 2024.

Reviewed by:

Copyright © 2024 Fan, Zhang, Chen, Li and Gao. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Wulin Gao, [email protected]

† These authors share first authorship

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.