scope and limitation of the study qualitative research

How to Write Limitations of the Study (with examples)

This blog emphasizes the importance of recognizing and effectively writing about limitations in research. It discusses the types of limitations, their significance, and provides guidelines for writing about them, highlighting their role in advancing scholarly research.

Updated on August 24, 2023

No matter how well thought out, every research endeavor encounters challenges. There is simply no way to predict all possible variances throughout the process.

These uncharted boundaries and abrupt constraints are known as limitations in research . Identifying and acknowledging limitations is crucial for conducting rigorous studies. Limitations provide context and shed light on gaps in the prevailing inquiry and literature.

This article explores the importance of recognizing limitations and discusses how to write them effectively. By interpreting limitations in research and considering prevalent examples, we aim to reframe the perception from shameful mistakes to respectable revelations.

What are limitations in research?

In the clearest terms, research limitations are the practical or theoretical shortcomings of a study that are often outside of the researcher’s control . While these weaknesses limit the generalizability of a study’s conclusions, they also present a foundation for future research.

Sometimes limitations arise from tangible circumstances like time and funding constraints, or equipment and participant availability. Other times the rationale is more obscure and buried within the research design. Common types of limitations and their ramifications include:

• Theoretical: limits the scope, depth, or applicability of a study.
• Methodological: limits the quality, quantity, or diversity of the data.
• Empirical: limits the representativeness, validity, or reliability of the data.
• Analytical: limits the accuracy, completeness, or significance of the findings.
• Ethical: limits the access, consent, or confidentiality of the data.

Regardless of how, when, or why they arise, limitations are a natural part of the research process and should never be ignored . Like all other aspects, they are vital in their own purpose.

Why is identifying limitations important?

Whether to seek acceptance or avoid struggle, humans often instinctively hide flaws and mistakes. Merging this thought process into research by attempting to hide limitations, however, is a bad idea. It has the potential to negate the validity of outcomes and damage the reputation of scholars.

By identifying and addressing limitations throughout a project, researchers strengthen their arguments and curtail the chance of peer censure based on overlooked mistakes. Pointing out these flaws shows an understanding of variable limits and a scrupulous research process.

Showing awareness of and taking responsibility for a project’s boundaries and challenges validates the integrity and transparency of a researcher. It further demonstrates the researchers understand the applicable literature and have thoroughly evaluated their chosen research methods.

Presenting limitations also benefits the readers by providing context for research findings. It guides them to interpret the project’s conclusions only within the scope of very specific conditions. By allowing for an appropriate generalization of the findings that is accurately confined by research boundaries and is not too broad, limitations boost a study’s credibility .

Limitations are true assets to the research process. They highlight opportunities for future research. When researchers identify the limitations of their particular approach to a study question, they enable precise transferability and improve chances for reproducibility. 

Simply stating a project’s limitations is not adequate for spurring further research, though. To spark the interest of other researchers, these acknowledgements must come with thorough explanations regarding how the limitations affected the current study and how they can potentially be overcome with amended methods.

How to write limitations

Typically, the information about a study’s limitations is situated either at the beginning of the discussion section to provide context for readers or at the conclusion of the discussion section to acknowledge the need for further research. However, it varies depending upon the target journal or publication guidelines. 

Don’t hide your limitations

It is also important to not bury a limitation in the body of the paper unless it has a unique connection to a topic in that section. If so, it needs to be reiterated with the other limitations or at the conclusion of the discussion section. Wherever it is included in the manuscript, ensure that the limitations section is prominently positioned and clearly introduced.

While maintaining transparency by disclosing limitations means taking a comprehensive approach, it is not necessary to discuss everything that could have potentially gone wrong during the research study. If there is no commitment to investigation in the introduction, it is unnecessary to consider the issue a limitation to the research. Wholly consider the term ‘limitations’ and ask, “Did it significantly change or limit the possible outcomes?” Then, qualify the occurrence as either a limitation to include in the current manuscript or as an idea to note for other projects. 

Writing limitations

Once the limitations are concretely identified and it is decided where they will be included in the paper, researchers are ready for the writing task. Including only what is pertinent, keeping explanations detailed but concise, and employing the following guidelines is key for crafting valuable limitations:

1) Identify and describe the limitations : Clearly introduce the limitation by classifying its form and specifying its origin. For example:

• An unintentional bias encountered during data collection
• An intentional use of unplanned post-hoc data analysis

2) Explain the implications : Describe how the limitation potentially influences the study’s findings and how the validity and generalizability are subsequently impacted. Provide examples and evidence to support claims of the limitations’ effects without making excuses or exaggerating their impact. Overall, be transparent and objective in presenting the limitations, without undermining the significance of the research. 

3) Provide alternative approaches for future studies : Offer specific suggestions for potential improvements or avenues for further investigation. Demonstrate a proactive approach by encouraging future research that addresses the identified gaps and, therefore, expands the knowledge base.

Whether presenting limitations as an individual section within the manuscript or as a subtopic in the discussion area, authors should use clear headings and straightforward language to facilitate readability. There is no need to complicate limitations with jargon, computations, or complex datasets.

Examples of common limitations

Limitations are generally grouped into two categories , methodology and research process .

Methodology limitations

Methodology may include limitations due to:

• Sample size
• Lack of available or reliable data
• Lack of prior research studies on the topic
• Measure used to collect the data
• Self-reported data

The researcher is addressing how the large sample size requires a reassessment of the measures used to collect and analyze the data.

Research process limitations

Limitations during the research process may arise from:

• Access to information
• Longitudinal effects
• Cultural and other biases
• Language fluency
• Time constraints

The author is pointing out that the model’s estimates are based on potentially biased observational studies.

Final thoughts

Successfully proving theories and touting great achievements are only two very narrow goals of scholarly research. The true passion and greatest efforts of researchers comes more in the form of confronting assumptions and exploring the obscure.

In many ways, recognizing and sharing the limitations of a research study both allows for and encourages this type of discovery that continuously pushes research forward. By using limitations to provide a transparent account of the project's boundaries and to contextualize the findings, researchers pave the way for even more robust and impactful research in the future.

Charla Viera, MS

See our "Privacy Policy"

Ensure your structure and ideas are consistent and clearly communicated

Pair your Premium Editing with our add-on service Presubmission Review for an overall assessment of your manuscript.

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• Limitations of the Study
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The limitations of the study are those characteristics of design or methodology that impacted or influenced the interpretation of the findings from your research. Study limitations are the constraints placed on the ability to generalize from the results, to further describe applications to practice, and/or related to the utility of findings that are the result of the ways in which you initially chose to design the study or the method used to establish internal and external validity or the result of unanticipated challenges that emerged during the study.

Price, James H. and Judy Murnan. “Research Limitations and the Necessity of Reporting Them.” American Journal of Health Education 35 (2004): 66-67; Theofanidis, Dimitrios and Antigoni Fountouki. "Limitations and Delimitations in the Research Process." Perioperative Nursing 7 (September-December 2018): 155-163. .

Importance of...

Always acknowledge a study's limitations. It is far better that you identify and acknowledge your study’s limitations than to have them pointed out by your professor and have your grade lowered because you appeared to have ignored them or didn't realize they existed.

Keep in mind that acknowledgment of a study's limitations is an opportunity to make suggestions for further research. If you do connect your study's limitations to suggestions for further research, be sure to explain the ways in which these unanswered questions may become more focused because of your study.

Acknowledgment of a study's limitations also provides you with opportunities to demonstrate that you have thought critically about the research problem, understood the relevant literature published about it, and correctly assessed the methods chosen for studying the problem. A key objective of the research process is not only discovering new knowledge but also to confront assumptions and explore what we don't know.

Claiming limitations is a subjective process because you must evaluate the impact of those limitations . Don't just list key weaknesses and the magnitude of a study's limitations. To do so diminishes the validity of your research because it leaves the reader wondering whether, or in what ways, limitation(s) in your study may have impacted the results and conclusions. Limitations require a critical, overall appraisal and interpretation of their impact. You should answer the question: do these problems with errors, methods, validity, etc. eventually matter and, if so, to what extent?

Price, James H. and Judy Murnan. “Research Limitations and the Necessity of Reporting Them.” American Journal of Health Education 35 (2004): 66-67; Structure: How to Structure the Research Limitations Section of Your Dissertation. Dissertations and Theses: An Online Textbook. Laerd.com.

Descriptions of Possible Limitations

All studies have limitations . However, it is important that you restrict your discussion to limitations related to the research problem under investigation. For example, if a meta-analysis of existing literature is not a stated purpose of your research, it should not be discussed as a limitation. Do not apologize for not addressing issues that you did not promise to investigate in the introduction of your paper.

Here are examples of limitations related to methodology and the research process you may need to describe and discuss how they possibly impacted your results. Note that descriptions of limitations should be stated in the past tense because they were discovered after you completed your research.

Possible Methodological Limitations

• Sample size -- the number of the units of analysis you use in your study is dictated by the type of research problem you are investigating. Note that, if your sample size is too small, it will be difficult to find significant relationships from the data, as statistical tests normally require a larger sample size to ensure a representative distribution of the population and to be considered representative of groups of people to whom results will be generalized or transferred. Note that sample size is generally less relevant in qualitative research if explained in the context of the research problem.
• Lack of available and/or reliable data -- a lack of data or of reliable data will likely require you to limit the scope of your analysis, the size of your sample, or it can be a significant obstacle in finding a trend and a meaningful relationship. You need to not only describe these limitations but provide cogent reasons why you believe data is missing or is unreliable. However, don’t just throw up your hands in frustration; use this as an opportunity to describe a need for future research based on designing a different method for gathering data.
• Lack of prior research studies on the topic -- citing prior research studies forms the basis of your literature review and helps lay a foundation for understanding the research problem you are investigating. Depending on the currency or scope of your research topic, there may be little, if any, prior research on your topic. Before assuming this to be true, though, consult with a librarian! In cases when a librarian has confirmed that there is little or no prior research, you may be required to develop an entirely new research typology [for example, using an exploratory rather than an explanatory research design ]. Note again that discovering a limitation can serve as an important opportunity to identify new gaps in the literature and to describe the need for further research.
• Measure used to collect the data -- sometimes it is the case that, after completing your interpretation of the findings, you discover that the way in which you gathered data inhibited your ability to conduct a thorough analysis of the results. For example, you regret not including a specific question in a survey that, in retrospect, could have helped address a particular issue that emerged later in the study. Acknowledge the deficiency by stating a need for future researchers to revise the specific method for gathering data.
• Self-reported data -- whether you are relying on pre-existing data or you are conducting a qualitative research study and gathering the data yourself, self-reported data is limited by the fact that it rarely can be independently verified. In other words, you have to the accuracy of what people say, whether in interviews, focus groups, or on questionnaires, at face value. However, self-reported data can contain several potential sources of bias that you should be alert to and note as limitations. These biases become apparent if they are incongruent with data from other sources. These are: (1) selective memory [remembering or not remembering experiences or events that occurred at some point in the past]; (2) telescoping [recalling events that occurred at one time as if they occurred at another time]; (3) attribution [the act of attributing positive events and outcomes to one's own agency, but attributing negative events and outcomes to external forces]; and, (4) exaggeration [the act of representing outcomes or embellishing events as more significant than is actually suggested from other data].

Possible Limitations of the Researcher

• Access -- if your study depends on having access to people, organizations, data, or documents and, for whatever reason, access is denied or limited in some way, the reasons for this needs to be described. Also, include an explanation why being denied or limited access did not prevent you from following through on your study.
• Longitudinal effects -- unlike your professor, who can literally devote years [even a lifetime] to studying a single topic, the time available to investigate a research problem and to measure change or stability over time is constrained by the due date of your assignment. Be sure to choose a research problem that does not require an excessive amount of time to complete the literature review, apply the methodology, and gather and interpret the results. If you're unsure whether you can complete your research within the confines of the assignment's due date, talk to your professor.
• Cultural and other type of bias -- we all have biases, whether we are conscience of them or not. Bias is when a person, place, event, or thing is viewed or shown in a consistently inaccurate way. Bias is usually negative, though one can have a positive bias as well, especially if that bias reflects your reliance on research that only support your hypothesis. When proof-reading your paper, be especially critical in reviewing how you have stated a problem, selected the data to be studied, what may have been omitted, the manner in which you have ordered events, people, or places, how you have chosen to represent a person, place, or thing, to name a phenomenon, or to use possible words with a positive or negative connotation. NOTE :   If you detect bias in prior research, it must be acknowledged and you should explain what measures were taken to avoid perpetuating that bias. For example, if a previous study only used boys to examine how music education supports effective math skills, describe how your research expands the study to include girls.
• Fluency in a language -- if your research focuses , for example, on measuring the perceived value of after-school tutoring among Mexican-American ESL [English as a Second Language] students and you are not fluent in Spanish, you are limited in being able to read and interpret Spanish language research studies on the topic or to speak with these students in their primary language. This deficiency should be acknowledged.

Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Senunyeme, Emmanuel K. Business Research Methods. Powerpoint Presentation. Regent University of Science and Technology; ter Riet, Gerben et al. “All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies.” PLOS One 8 (November 2013): 1-6.

Structure and Writing Style

Information about the limitations of your study are generally placed either at the beginning of the discussion section of your paper so the reader knows and understands the limitations before reading the rest of your analysis of the findings, or, the limitations are outlined at the conclusion of the discussion section as an acknowledgement of the need for further study. Statements about a study's limitations should not be buried in the body [middle] of the discussion section unless a limitation is specific to something covered in that part of the paper. If this is the case, though, the limitation should be reiterated at the conclusion of the section.

If you determine that your study is seriously flawed due to important limitations , such as, an inability to acquire critical data, consider reframing it as an exploratory study intended to lay the groundwork for a more complete research study in the future. Be sure, though, to specifically explain the ways that these flaws can be successfully overcome in a new study.

But, do not use this as an excuse for not developing a thorough research paper! Review the tab in this guide for developing a research topic . If serious limitations exist, it generally indicates a likelihood that your research problem is too narrowly defined or that the issue or event under study is too recent and, thus, very little research has been written about it. If serious limitations do emerge, consult with your professor about possible ways to overcome them or how to revise your study.

When discussing the limitations of your research, be sure to:

• Describe each limitation in detailed but concise terms;
• Explain why each limitation exists;
• Provide the reasons why each limitation could not be overcome using the method(s) chosen to acquire or gather the data [cite to other studies that had similar problems when possible];
• Assess the impact of each limitation in relation to the overall findings and conclusions of your study; and,
• If appropriate, describe how these limitations could point to the need for further research.

Remember that the method you chose may be the source of a significant limitation that has emerged during your interpretation of the results [for example, you didn't interview a group of people that you later wish you had]. If this is the case, don't panic. Acknowledge it, and explain how applying a different or more robust methodology might address the research problem more effectively in a future study. A underlying goal of scholarly research is not only to show what works, but to demonstrate what doesn't work or what needs further clarification.

Aguinis, Hermam and Jeffrey R. Edwards. “Methodological Wishes for the Next Decade and How to Make Wishes Come True.” Journal of Management Studies 51 (January 2014): 143-174; Brutus, Stéphane et al. "Self-Reported Limitations and Future Directions in Scholarly Reports: Analysis and Recommendations." Journal of Management 39 (January 2013): 48-75; Ioannidis, John P.A. "Limitations are not Properly Acknowledged in the Scientific Literature." Journal of Clinical Epidemiology 60 (2007): 324-329; Pasek, Josh. Writing the Empirical Social Science Research Paper: A Guide for the Perplexed. January 24, 2012. Academia.edu; Structure: How to Structure the Research Limitations Section of Your Dissertation. Dissertations and Theses: An Online Textbook. Laerd.com; What Is an Academic Paper? Institute for Writing Rhetoric. Dartmouth College; Writing the Experimental Report: Methods, Results, and Discussion. The Writing Lab and The OWL. Purdue University.

Writing Tip

Don't Inflate the Importance of Your Findings!

After all the hard work and long hours devoted to writing your research paper, it is easy to get carried away with attributing unwarranted importance to what you’ve done. We all want our academic work to be viewed as excellent and worthy of a good grade, but it is important that you understand and openly acknowledge the limitations of your study. Inflating the importance of your study's findings could be perceived by your readers as an attempt hide its flaws or encourage a biased interpretation of the results. A small measure of humility goes a long way!

Another Writing Tip

Negative Results are Not a Limitation!

Negative evidence refers to findings that unexpectedly challenge rather than support your hypothesis. If you didn't get the results you anticipated, it may mean your hypothesis was incorrect and needs to be reformulated. Or, perhaps you have stumbled onto something unexpected that warrants further study. Moreover, the absence of an effect may be very telling in many situations, particularly in experimental research designs. In any case, your results may very well be of importance to others even though they did not support your hypothesis. Do not fall into the trap of thinking that results contrary to what you expected is a limitation to your study. If you carried out the research well, they are simply your results and only require additional interpretation.

Lewis, George H. and Jonathan F. Lewis. “The Dog in the Night-Time: Negative Evidence in Social Research.” The British Journal of Sociology 31 (December 1980): 544-558.

Yet Another Writing Tip

Sample Size Limitations in Qualitative Research

Sample sizes are typically smaller in qualitative research because, as the study goes on, acquiring more data does not necessarily lead to more information. This is because one occurrence of a piece of data, or a code, is all that is necessary to ensure that it becomes part of the analysis framework. However, it remains true that sample sizes that are too small cannot adequately support claims of having achieved valid conclusions and sample sizes that are too large do not permit the deep, naturalistic, and inductive analysis that defines qualitative inquiry. Determining adequate sample size in qualitative research is ultimately a matter of judgment and experience in evaluating the quality of the information collected against the uses to which it will be applied and the particular research method and purposeful sampling strategy employed. If the sample size is found to be a limitation, it may reflect your judgment about the methodological technique chosen [e.g., single life history study versus focus group interviews] rather than the number of respondents used.

Boddy, Clive Roland. "Sample Size for Qualitative Research." Qualitative Market Research: An International Journal 19 (2016): 426-432; Huberman, A. Michael and Matthew B. Miles. "Data Management and Analysis Methods." In Handbook of Qualitative Research . Norman K. Denzin and Yvonna S. Lincoln, eds. (Thousand Oaks, CA: Sage, 1994), pp. 428-444; Blaikie, Norman. "Confounding Issues Related to Determining Sample Size in Qualitative Research." International Journal of Social Research Methodology 21 (2018): 635-641; Oppong, Steward Harrison. "The Problem of Sampling in qualitative Research." Asian Journal of Management Sciences and Education 2 (2013): 202-210.

• << Previous: 8. The Discussion
• Next: 9. The Conclusion >>
• Last Updated: Apr 16, 2024 10:20 AM
• URL: https://libguides.usc.edu/writingguide

Setting Limits and Focusing Your Study: Exploring scope and delimitation

As a researcher, it can be easy to get lost in the vast expanse of information and data available. Thus, when starting a research project, one of the most important things to consider is the scope and delimitation of the study. Setting limits and focusing your study is essential to ensure that the research project is manageable, relevant, and able to produce useful results. In this article, we will explore the importance of setting limits and focusing your study through an in-depth analysis of scope and delimitation.

Company Name 123

Lorem ipsum dolor sit amet, cu usu cibo vituperata, id ius probo maiestatis inciderint, sit eu vide volutpat.

Sign Up for More Insights

Table of Contents

Scope and Delimitation – Definition and difference

Scope refers to the range of the research project and the study limitations set in place to define the boundaries of the project and delimitation refers to the specific aspects of the research project that the study will focus on.

In simpler words, scope is the breadth of your study, while delimitation is the depth of your study.

Scope and delimitation are both essential components of a research project, and they are often confused with one another. The scope defines the parameters of the study, while delimitation sets the boundaries within those parameters. The scope and delimitation of a study are usually established early on in the research process and guide the rest of the project.

Types of Scope and Delimitation

Significance of Scope and Delimitation

Setting limits and focusing your study through scope and delimitation is crucial for the following reasons:

• It allows researchers to define the research project’s boundaries, enabling them to focus on specific aspects of the project. This focus makes it easier to gather relevant data and avoid unnecessary information that might complicate the study’s results.
• Setting limits and focusing your study through scope and delimitation enables the researcher to stay within the parameters of the project’s resources.
• A well-defined scope and delimitation ensure that the research project can be completed within the available resources, such as time and budget, while still achieving the project’s objectives.

5 Steps to Setting Limits and Defining the Scope and Delimitation of Your Study

There are a few steps that you can take to set limits and focus your study.

1. Identify your research question or topic

The first step is to identify what you are interested in learning about. The research question should be specific, measurable, achievable, relevant, and time-bound (SMART). Once you have a research question or topic, you can start to narrow your focus.

2. Consider the key terms or concepts related to your topic

What are the important terms or concepts that you need to understand in order to answer your research question? Consider all available resources, such as time, budget, and data availability, when setting scope and delimitation.

The scope and delimitation should be established within the parameters of the available resources. Once you have identified the key terms or concepts, you can start to develop a glossary or list of definitions.

3. Consider the different perspectives on your topic

There are often different perspectives on any given topic. Get feedback on the proposed scope and delimitation. Advisors can provide guidance on the feasibility of the study and offer suggestions for improvement.

It is important to consider all of the different perspectives in order to get a well-rounded understanding of your topic.

4. Narrow your focus

Be specific and concise when setting scope and delimitation. The parameters of the study should be clearly defined to avoid ambiguity and ensure that the study is focused on relevant aspects of the research question.

This means deciding which aspects of your topic you will focus on and which aspects you will eliminate.

5. Develop the final research plan

Revisit and revise the scope and delimitation as needed. As the research project progresses, the scope and delimitation may need to be adjusted to ensure that the study remains focused on the research question and can produce useful results. This plan should include your research goals, methods, and timeline.

Examples of Scope and Delimitation

To better understand scope and delimitation, let us consider two examples of research questions and how scope and delimitation would apply to them.

Research question: What are the effects of social media on mental health?

Scope: The scope of the study will focus on the impact of social media on the mental health of young adults aged 18-24 in the United States.

Delimitation: The study will specifically examine the following aspects of social media: frequency of use, types of social media platforms used, and the impact of social media on self-esteem and body image.

Research question: What are the factors that influence employee job satisfaction in the healthcare industry?

Scope: The scope of the study will focus on employee job satisfaction in the healthcare industry in the United States.

Delimitation: The study will specifically examine the following factors that influence employee job satisfaction: salary, work-life balance, job security, and opportunities for career growth.

Setting limits and defining the scope and delimitation of a research study is essential to conducting effective research. By doing so, researchers can ensure that their study is focused, manageable, and feasible within the given time frame and resources. It can also help to identify areas that require further study, providing a foundation for future research.

So, the next time you embark on a research project, don’t forget to set clear limits and define the scope and delimitation of your study. It may seem like a tedious task, but it can ultimately lead to more meaningful and impactful research. And if you still can’t find a solution, reach out to Enago Academy using #AskEnago and tag @EnagoAcademy on Twitter , Facebook , and Quora .

Frequently Asked Questions

The scope in research refers to the boundaries and extent of a study, defining its specific objectives, target population, variables, methods, and limitations, which helps researchers focus and provide a clear understanding of what will be investigated.

Delimitation in research defines the specific boundaries and limitations of a study, such as geographical, temporal, or conceptual constraints, outlining what will be excluded or not within the scope of investigation, providing clarity and ensuring the study remains focused and manageable.

To write a scope; 1. Clearly define research objectives. 2. Identify specific research questions. 3. Determine the target population for the study. 4. Outline the variables to be investigated. 5. Establish limitations and constraints. 6. Set boundaries and extent of the investigation. 7. Ensure focus, clarity, and manageability. 8. Provide context for the research project.

To write delimitations; 1. Identify geographical boundaries or constraints. 2. Define the specific time period or timeframe of the study. 3. Specify the sample size or selection criteria. 4. Clarify any demographic limitations (e.g., age, gender, occupation). 5. Address any limitations related to data collection methods. 6. Consider limitations regarding the availability of resources or data. 7. Exclude specific variables or factors from the scope of the study. 8. Clearly state any conceptual boundaries or theoretical frameworks. 9. Acknowledge any potential biases or constraints in the research design. 10. Ensure that the delimitations provide a clear focus and scope for the study.

What is an example of delimitation of the study?

Thank you 💕

Thank You very simplified🩷

Thanks, I find this article very helpful

Rate this article Cancel Reply

Your email address will not be published.

Enago Academy's Most Popular Articles

• Publishing Research
• Reporting Research

How to Optimize Your Research Process: A step-by-step guide

For researchers across disciplines, the path to uncovering novel findings and insights is often filled…

• Industry News
• Trending Now

Breaking Barriers: Sony and Nature unveil “Women in Technology Award”

Sony Group Corporation and the prestigious scientific journal Nature have collaborated to launch the inaugural…

Achieving Research Excellence: Checklist for good research practices

Academia is built on the foundation of trustworthy and high-quality research, supported by the pillars…

• Promoting Research

Plain Language Summary — Communicating your research to bridge the academic-lay gap

Science can be complex, but does that mean it should not be accessible to the…

Science under Surveillance: Journals adopt advanced AI to uncover image manipulation

Journals are increasingly turning to cutting-edge AI tools to uncover deceitful images published in manuscripts.…

Choosing the Right Analytical Approach: Thematic analysis vs. content analysis for…

Research Recommendations – Guiding policy-makers for evidence-based decision making

Demystifying the Role of Confounding Variables in Research

Sign-up to read more

Subscribe for free to get unrestricted access to all our resources on research writing and academic publishing including:

• 2000+ blog articles
• 50+ Webinars
• 10+ Expert podcasts
• 50+ Infographics
• 10+ Checklists
• Research Guides

We hate spam too. We promise to protect your privacy and never spam you.

I am looking for Editing/ Proofreading services for my manuscript Tentative date of next journal submission:

What should universities' stance be on AI tools in research and academic writing?

Research Limitations & Delimitations

What they are and how they’re different (with examples)

By: Derek Jansen (MBA) | Expert Reviewed By: David Phair (PhD) | September 2022

If you’re new to the world of research, you’ve probably heard the terms “ research limitations ” and “ research delimitations ” being thrown around, often quite loosely. In this post, we’ll unpack what both of these mean, how they’re similar and how they’re different – so that you can write up these sections the right way.

Overview: Limitations vs Delimitations

• Are they the same?
• What are research limitations
• What are research delimitations
• Limitations vs delimitations

First things first…

Let’s start with the most important takeaway point of this post – research limitations and research delimitations are not the same – but they are related to each other (we’ll unpack that a little later). So, if you hear someone using these two words interchangeably, be sure to share this post with them!

Research Limitations

Research limitations are, at the simplest level, the weaknesses of the study, based on factors that are often outside of your control as the researcher. These factors could include things like time , access to funding, equipment , data or participants . For example, if you weren’t able to access a random sample of participants for your study and had to adopt a convenience sampling strategy instead, that would impact the generalizability of your findings and therefore reflect a limitation of your study.

Research limitations can also emerge from the research design itself . For example, if you were undertaking a correlational study, you wouldn’t be able to infer causality (since correlation doesn’t mean certain causation). Similarly, if you utilised online surveys to collect data from your participants, you naturally wouldn’t be able to get the same degree of rich data that you would from in-person interviews .

Simply put, research limitations reflect the shortcomings of a study , based on practical (or theoretical) constraints that the researcher faced. These shortcomings limit what you can conclude from a study, but at the same time, present a foundation for future research . Importantly, all research has limitations , so there’s no need to hide anything here – as long as you discuss how the limitations might affect your findings, it’s all good.

Research Delimitations

Alright, now that we’ve unpacked the limitations, let’s move on to the delimitations .

Research delimitations are similar to limitations in that they also “ limit ” the study, but their focus is entirely different. Specifically, the delimitations of a study refer to the scope of the research aims and research questions . In other words, delimitations reflect the choices you, as the researcher, intentionally make in terms of what you will and won’t try to achieve with your study. In other words, what your research aims and research questions will and won’t include.

As we’ve spoken about many times before, it’s important to have a tight, narrow focus for your research, so that you can dive deeply into your topic, apply your energy to one specific area and develop meaningful insights. If you have an overly broad scope or unfocused topic, your research will often pull in multiple, even opposing directions, and you’ll just land up with a muddy mess of findings .

So, the delimitations section is where you’ll clearly state what your research aims and research questions will focus on – and just as importantly, what they will exclude . For example, you might investigate a widespread phenomenon, but choose to focus your study on a specific age group, ethnicity or gender. Similarly, your study may focus exclusively on one country, city or even organization. As long as the scope is well justified (in other words, it represents a novel, valuable research topic), this is perfectly acceptable – in fact, it’s essential. Remember, focus is your friend.

Need a helping hand?

Conclusion: Limitations vs Delimitations

Ok, so let’s recap.

Research limitations and research delimitations are related in that they both refer to “limits” within a study. But, they are distinctly different. Limitations reflect the shortcomings of your study, based on practical or theoretical constraints that you faced.

Contrasted to that, delimitations reflect the choices that you made in terms of the focus and scope of your research aims and research questions. If you want to learn more about research aims and questions, you can check out this video post , where we unpack those concepts in detail.

Psst… there’s more (for free)

This post is part of our dissertation mini-course, which covers everything you need to get started with your dissertation, thesis or research project. 

You Might Also Like:

18 Comments

Good clarification of ideas on how a researcher ought to do during Process of choice

Thank you so much for this very simple but explicit explanation on limitation and delimitation. It has so helped me to develop my masters proposal. hope to recieve more from your site as time progresses

Thank you for this explanation – very clear.

Thanks for the explanation, really got it well.

This website is really helpful for my masters proposal

Thank you very much for helping to explain these two terms

I spent almost the whole day trying to figure out the differences

when I came across your notes everything became very clear

thanks for the clearly outlined explanation on the two terms, limitation and delimitation.

Very helpful Many thanks 🙏

Excellent it resolved my conflict .

I would like you to assist me please. If in my Research, I interviewed some participants and I submitted Questionnaires to other participants to answered to the questions, in the same organization, Is this a Qualitative methodology , a Quantitative Methodology or is it a Mixture Methodology I have used in my research? Please help me

How do I cite this article in APA format

Really so great ,finally have understood it’s difference now

Getting more clear regarding Limitations and Delimitation and concepts

I really appreciate your apt and precise explanation of the two concepts namely ; Limitations and Delimitations.

This is a good sources of research information for learners.

thank you for this, very helpful to researchers

Very good explained

Great and clear explanation, after a long confusion period on the two words, i can now explain to someone with ease.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Community Blog

Keep up-to-date on postgraduate related issues with our quick reads written by students, postdocs, professors and industry leaders.

How to Write the Scope of the Study

• By DiscoverPhDs
• August 26, 2020

What is the Scope of the Study?

The scope of the study refers to the boundaries within which your research project will be performed; this is sometimes also called the scope of research. To define the scope of the study is to define all aspects that will be considered in your research project. It is also just as important to make clear what aspects will not be covered; i.e. what is outside of the scope of the study.

Why is the Scope of the Study Important?

The scope of the study is always considered and agreed upon in the early stages of the project, before any data collection or experimental work has started. This is important because it focuses the work of the proposed study down to what is practically achievable within a given timeframe.

A well-defined research or study scope enables a researcher to give clarity to the study outcomes that are to be investigated. It makes clear why specific data points have been collected whilst others have been excluded.

Without this, it is difficult to define an end point for a research project since no limits have been defined on the work that could take place. Similarly, it can also make the approach to answering a research question too open ended.

How do you Write the Scope of the Study?

In order to write the scope of the study that you plan to perform, you must be clear on the research parameters that you will and won’t consider. These parameters usually consist of the sample size, the duration, inclusion and exclusion criteria, the methodology and any geographical or monetary constraints.

Each of these parameters will have limits placed on them so that the study can practically be performed, and the results interpreted relative to the limitations that have been defined. These parameters will also help to shape the direction of each research question you consider.

The term limitations’ is often used together with the scope of the study to describe the constraints of any parameters that are considered and also to clarify which parameters have not been considered at all. Make sure you get the balance right here between not making the scope too broad and unachievable, and it not being too restrictive, resulting in a lack of useful data.

The sample size is a commonly used parameter in the definition of the research scope. For example, a research project involving human participants may define at the start of the study that 100 participants will be recruited. This number will be determined based on an understanding of the difficulty in recruiting participants to studies and an agreement of an acceptable period of time in which to recruit this number.

Any results that are obtained by the research group can then be interpreted by others with the knowledge that the study was capped to 100 participants and an acceptance of this as a limitation of the study. In other words, it is acknowledged that recruiting 100 rather than 1,000 participants has limited the amount of data that could be collected, however this is an acceptable limitation due to the known difficulties in recruiting so many participants (e.g. the significant period of time it would take and the costs associated with this).

Example of a Scope of the Study

The follow is a (hypothetical) example of the definition of the scope of the study, with the research question investigating the impact of the COVID-19 pandemic on mental health.

Whilst the immediate negative health problems related to the COVID-19 pandemic have been well documented, the impact of the virus on the mental health (MH) of young adults (age 18-24 years) is poorly understood. The aim of this study is to report on MH changes in population group due to the pandemic.

The scope of the study is limited to recruiting 100 volunteers between the ages of 18 and 24 who will be contacted using their university email accounts. This recruitment period will last for a maximum of 2 months and will end when either 100 volunteers have been recruited or 2 months have passed. Each volunteer to the study will be asked to complete a short questionnaire in order to evaluate any changes in their MH.

From this example we can immediately see that the scope of the study has placed a constraint on the sample size to be used and/or the time frame for recruitment of volunteers. It has also introduced a limitation by only opening recruitment to people that have university emails; i.e. anyone that does not attend university will be excluded from this study.

This may be an important factor when interpreting the results of this study; the comparison of MH during the pandemic between those that do and do not attend university, is therefore outside the scope of the study here. We are also told that the methodology used to assess any changes in MH are via a questionnaire. This is a clear definition of how the outcome measure will be investigated and any other methods are not within the scope of research and their exclusion may be a limitation of the study.

The scope of the study is important to define as it enables a researcher to focus their research to within achievable parameters.

The scope of the study is defined at the start of the study. It is used by researchers to set the boundaries and limitations within which the research study will be performed.

Find out the differences between a Literature Review and an Annotated Bibliography, whey they should be used and how to write them.

The title page of your dissertation or thesis conveys all the essential details about your project. This guide helps you format it in the correct way.

Join thousands of other students and stay up to date with the latest PhD programmes, funding opportunities and advice.

Browse PhDs Now

This post explains the difference between the journal paper status of In Review and Under Review.

Stay up to date with current information being provided by the UK Government and Universities about the impact of the global pandemic on PhD research studies.

Christine is entering the 4th year of her PhD Carleton University, researching worker’s experiences of the changing conditions in the Non Profit and Social Service sector, pre and during COVID-19.

Amy recently entered her third and final year of her PhD at the University of Strathclyde. Her research has focussed on young people’s understanding of mental health stigma in Scotland.

Join Thousands of Students

How To Write Scope and Delimitation of a Research Paper (With Examples)

An effective research paper or thesis has a well-written Scope and Delimitation.  This portion specifies your study’s coverage and boundaries.

Not yet sure about how to write your research’s Scope and Delimitation? Fret not, as we’ll guide you through the entire writing process through this article.

Related: How To Write Significance of the Study (With Examples)

Table of Contents

What is the scope and delimitation of a research paper.

The “Scope and Delimitation” section states the concepts and variables your study covered. It tells readers which things you have included and excluded in your analysis.

This portion tells two things: 1

• The study’s “Scope” – concepts and variables you have explored in your research and;
• The study’s “Delimitation” – the “boundaries” of your study’s scope. It sets apart the things included in your analysis from those excluded.

For example, your scope might be the effectiveness of plant leaves in lowering blood sugar levels. You can “delimit” your study only to the effect of gabi leaves on the blood glucose of Swiss mice.

Where Should I Put the Scope and Delimitation?

This portion is in Chapter 1, usually after the “Background of the Study.”

Why Should I Write the Scope and Delimitation of My Research Paper?

There’s a lot to discover in a research paper or thesis. However, your resources and time dedicated to it are scarce. Thus, given these constraints, you have to narrow down your study. You do this in the Scope and Delimitation.

Suppose you’re studying the correlation between the quantity of organic fertilizer and plant growth . Experimenting with several types of plants is impossible because of several limitations. So, you’ve decided to use one plant type only. 

Informing your readers about this decision is a must. So, you have to state it in your Scope and Delimitation. It also acts as a “disclaimer” that your results are inapplicable to the entire plant kingdom.

What Is the Difference Between Delimitation and Limitation?

People often use the terms “Delimitation” and “Limitation” interchangeably. However, these words differ 2 .

Delimitation refers to factors you set to limit your analysis. It delineates those that are included in your research and those that are excluded. Remember, delimitations are within your control. 

Meanwhile, limitations are factors beyond your control that may affect your research’s results.  You can think of limitations as the “weaknesses” of your study. 

Let’s go back to our previous example. Due to some constraints, you’ve only decided to examine one plant type: dandelions. This is an example of a delimitation since it limits your analysis to dandelions only and not other plant types. Note that the number of plant types used is within your control. 

Meanwhile, your study cannot state that a higher quantity of organic fertilizer is the sole reason for plant growth. That’s because your research’s focus is only on correlation. Since this is already beyond your control, then this is a limitation. 

How To Write Scope and Delimitation: Step-by-Step Guide

To write your research’s Scope and Delimitation section, follow these steps:

1. Review Your Study’s Objectives and Problem Statement

Your study’s coverage relies on its objectives. Thus, you can only write this section if you know what you’re researching. Furthermore, ensure that you understand the problems you ought to answer. 

Once you understand the abovementioned things, you may start writing your study’s Scope and Delimitation.

2. State the Key Information To Explain Your Study’s Coverage and Boundaries

a. The Main Objective of the Research

This refers to the concept that you’re focusing on in your research. Some examples are the following:

• level of awareness or satisfaction of a particular group of people
• correlation between two variables
• effectiveness of a new product
• comparison between two methods/approaches
• lived experiences of several individuals

It’s helpful to consult your study’s Objectives or Statement of the Problem section to determine your research’s primary goal.

b. Independent and Dependent Variables Included

Your study’s independent variable is the variable that you manipulate. Meanwhile, the dependent variable is the variable whose result depends upon the independent variable. Both of these variables must be clear and specific when indicated. 

Suppose you study the relationship between social media usage and students’ language skills. These are the possible variables for the study:

• Independent Variable: Number of hours per day spent on using Facebook
• Dependent Variable: Grade 10 students’ scores in Quarterly Examination in English. 

Note how specific the variables stated above are. For the independent variable, we narrow it down to Facebook only. Since there are many ways to assess “language skills,” we zero in on the students’ English exam scores as our dependent variable. 

c. Subject of the Study

This refers to your study’s respondents or participants. 

In our previous example, the research participants are Grade 10 students. However, there are a lot of Grade 10 students in the Philippines. Thus, we have to select from a specific school only—for instance, Grade 10 students from a national high school in Manila. 

d. Timeframe and Location of the Study

Specify the month(s), quarter(s), or year(s) as the duration of your study. Also, indicate where you will gather the data required for your research. 

e. Brief Description of the Study’s Research Design and Methodology

You may also include whether your research is quantitative or qualitative, the sampling method (cluster, stratified, purposive) applied, and how you conducted the experiment.

Using our previous example, the Grade 10 students can be selected using stratified sampling. Afterward, the researchers may obtain their English quarterly exam scores from their respective teachers. You can add these things to your study’s Scope and Delimitation. 

3. Indicate Which Variables or Factors Are Not Covered by Your Research

Although you’ve already set your study’s coverage and boundaries in Step 2, you may also explicitly mention things you’ve excluded from your research. 

Returning to our previous example, you can state that your assessment will not include the vocabulary and oral aspects of the English proficiency skill. 

Examples of Scope and Delimitation of a Research Paper

1. scope and delimitation examples for quantitative research.

a. Example 1

Research Title

    A Study on the Relationship of the Extent of Facebook Usage on the English Proficiency Level of Grade 10 Students of Matagumpay High School

Scope and Delimitation

(Main Objective)

This study assessed the correlation between the respondents’ duration of Facebook usage and their English proficiency level. 

(Variables used)

The researchers used the number of hours per day of using Facebook and the activities usually performed on the platform to assess the respondents’ extent of Facebook usage. Meanwhile, the respondents’ English proficiency level is limited to their quarterly English exam scores. 

(Subject of the study)

A sample of fifty (50) Grade 10 students of Matagumpay High School served as the study’s respondents. 

(Timeframe and location)

This study was conducted during the Second Semester of the School Year 2018 – 2019 on the premises of Matagumpay High School in Metro Manila. 

(Methodology)

The respondents are selected by performing stratified random sampling to ensure that there will be ten respondents from five Grade 10 classes of the school mentioned above. The researchers administered a 20-item questionnaire to assess the extent of Facebook usage of the selected respondents. Meanwhile, the data for the respondents’ quarterly exam scores were acquired from their English teachers. The collected data are handled with the utmost confidentiality. Spearman’s Rank Order Correlation was applied to quantitatively assess the correlation between the variables.

(Exclusions)

This study didn’t assess other aspects of the respondents’ English proficiency, such as English vocabulary and oral skills. 

Note: The words inside the parentheses in the example above are guides only. They are not included in the actual text.

b. Example 2

  Level of Satisfaction of Grade 11 Students on the Implementation of the Online Learning Setup of Matagumpay High School for SY 2020 – 2021

This study aims to identify students’ satisfaction levels with implementing online learning setups during the height of the COVID-19 pandemic.

Students’ satisfaction was assessed according to teachers’ pedagogy, school policies, and learning materials used in the online learning setup. The respondents included sixty (60) Grade 11 students of Matagumpay High School who were randomly picked. The researchers conducted the study from October 2020 to February 2021. 

Online platforms such as email and social media applications were used to reach the respondents. The researchers administered a 15-item online questionnaire to measure the respondents’ satisfaction levels. Each response was assessed using a Likert Scale to provide a descriptive interpretation of their answers. A weighted mean was applied to determine the respondents’ general satisfaction. 

This study did not cover other factors related to the online learning setup, such as the learning platform used, the schedule of synchronous learning, and channels for information dissemination.

2. Scope and Delimitation Examples for Qualitative Research

  Lived Experiences of Public Utility Vehicle (PUV) Drivers of Antipolo City Amidst the Continuous June 2022 Oil Price Hikes

This research focused on the presentation and discussion of the lived experiences of PUV drivers during the constant oil price hike in June 2022.

The respondents involved are five (5) jeepney drivers from Antipolo City who agreed to be interviewed. The researchers assessed their experiences in terms of the following: (1) daily net income; (2) duration and extent of working; (3) alternative employment opportunity considerations; and (4) mental and emotional status. The respondents were interviewed daily at their stations on June 6 – 10, 2022. 

In-depth one-on-one interviews were used for data collection.  Afterward, the respondents’ first-hand experiences were drafted and annotated with the researchers’ insights. 

The researchers excluded some factors in determining the respondents’ experiences, such as physical and health conditions and current family relationship status. 

 A Study on the Perception of the Residents of Mayamot, Antipolo City on the Political and Socioeconomic Conditions During the Post-EDSA Period (1986 – 1996)

This research aims to discuss the perception of Filipinos regarding the political and socioeconomic economic conditions during the post-EDSA period, specifically during the years 1986 – 1996. 

Ten (10) residents of Mayamot, Antipolo City, who belonged to Generation X (currently 40 – 62 years old), were purposively selected as the study’s respondents. The researchers asked them about their perception of the following aspects during the period mentioned above (1) performance of national and local government; (2) bureaucracy and government services; (3) personal economic and financial status; and (4) wage purchasing power. 

The researchers conducted face-to-face interviews in the respondents’ residences during the second semester of AY 2018 – 2019. The responses were written and corroborated with the literature on the post-EDSA period. 

The following factors were not included in the research analysis: political conflicts and turmoils, the status of the legislative and judicial departments, and other macroeconomic indicators. 

Tips and Warnings

1. use the “5ws and 1h” as your guide in understanding your study’s coverage.

• Why did you write your study?  
• What variables are included?
• Who are your study’s subject
• Where did you conduct the study?
• When did your study start and end?
• How did you conduct the study?

2. Use key phrases when writing your research’s scope

• This study aims to … 
• This study primarily focuses on …
• This study deals with … 
• This study will cover …
• This study will be confined…

3. Use key phrases when writing factors beyond your research’s delimitations

• The researcher(s) decided to exclude …
• This study did not cover….
• This study excluded … 
• These variables/factors were excluded from the study…

4. Don’t forget to ask for help

Your research adviser can assist you in selecting specific concepts and variables suitable to your study. Make sure to consult him/her regularly. 

5. Make it brief

No need to make this section wordy. You’re good to go if you meet the “5Ws and 1Hs”. 

Frequently Asked Questions

1. what are scope and delimitation in tagalog.

In a Filipino research ( pananaliksik ), Scope and Delimitation is called “ Saklaw at Delimitasyon”. 

Here’s an example of Scope and Delimitation in Filipino:

Pamagat ng Pananaliksik

Epekto Ng Paggamit Ng Mga Digital Learning Tools Sa Pag-Aaral Ng Mga Mag-Aaral Ng Mataas Na Paaralan Ng Matagumpay Sa General Mathematics

Sakop at Delimitasyon ng Pag-aaral

Nakatuon ang pananaliksik na ito sa epekto ng paggamit ng mga digital learning aids sa pag-aaral ng mga mag-aaral.

Ang mga digital learning tools na kinonsidera sa pag-aaral na ito ay Google Classroom, Edmodo, Kahoot, at mga piling bidyo mula YouTube. Samantala, ang epekto sa pag-aaral ng mga mag-aaral ng mga nabanggit na digital learning tools ay natukoy sa pamamagitan ng kanilang (1) mga pananaw hinggil sa benepisyo nito sa kanilang pag-aaral sa General Mathematics at (2) kanilang average grade sa asignaturang ito.

Dalawampu’t-limang (25) mag-aaral mula sa Senior High School ng Mataas na Paaralan ng Matagumpay ang pinili para sa pananaliksik na ito. Sila ay na-interbyu at binigyan ng questionnaire noong Enero 2022 sa nasabing paaralan. Sinuri ang resulta ng pananaliksik sa pamamagitan ng mga instrumentong estadistikal na weighted mean at Analysis of Variance (ANOVA). Hindi saklaw ng pananaliksik na ito ang ibang mga aspeto hinggil sa epekto ng online learning aids sa pag-aaral gaya ng lebel ng pag-unawa sa aralin at kakayahang iugnay ito sa araw-araw na buhay. 

2. The Scope and Delimitation should consist of how many paragraphs?

Three or more paragraphs will suffice for your study’s Scope and Delimitation. Here’s our suggestion on what you should write for each paragraph:

Paragraph 1: Introduction (state research objective) Paragraph 2: Coverage and boundaries of the research (you may divide this section into 2-3 paragraphs) Paragraph 3 : Factors excluded from the study

• University of St. La Salle. Unit 3: Lesson 3 Setting the Scope and Limitation of a Qualitative Research [Ebook] (p. 12). Retrieved from https://www.studocu.com/ph/document/university-of-st-la-salle/senior-high-school/final-sg-pr1-11-12-unit-3-lesson-3-setting-the-scope-and-limitation-of-a-qualitative-research/24341582
• Theofanidis, D., & Fountouki, A. (2018). Limitations and Delimitations in the Research Process. Perioperative Nursing (GORNA), 7(3), 155–162. doi: 10.5281/zenodo.2552022

Written by Jewel Kyle Fabula

in Career and Education , Juander How

Last Updated May 6, 2023 09:59 AM

Jewel Kyle Fabula

Jewel Kyle Fabula is a Bachelor of Science in Economics student at the University of the Philippines Diliman. His passion for learning mathematics developed as he competed in some mathematics competitions during his Junior High School years. He loves cats, playing video games, and listening to music.

Browse all articles written by Jewel Kyle Fabula

Copyright Notice

All materials contained on this site are protected by the Republic of the Philippines copyright law and may not be reproduced, distributed, transmitted, displayed, published, or broadcast without the prior written permission of filipiknow.net or in the case of third party materials, the owner of that content. You may not alter or remove any trademark, copyright, or other notice from copies of the content. Be warned that we have already reported and helped terminate several websites and YouTube channels for blatantly stealing our content. If you wish to use filipiknow.net content for commercial purposes, such as for content syndication, etc., please contact us at legal(at)filipiknow(dot)net

• Social Science
• Social Psychology

Assumptions, Limitations, Delimitations, and Scope of the Study By

Related documents

Add this document to collection(s)

You can add this document to your study collection(s)

Add this document to saved

You can add this document to your saved list

Suggest us how to improve StudyLib

(For complaints, use another form )

Input it if you want to receive answer

CRO Platform

Test your insights. Run experiments. Win. Or learn. And then win.

eCommerce Customer Analytics Platform

Customer Survey Platform

• Managed Services
• How we Help

Acquisition matters. But retention matters more. Understand, monitor & nurture the best customers.

• Certified Partners
• Case Studies
• Ebooks, Tools, Templates
• Digital Marketing Glossary
• eCommerce Growth Stories
• eCommerce Growth Show
• Help & Technical Documentation

CRO Guide   >  Chapter 3.1

Qualitative Research: Definition, Methodology, Limitation, Examples

These are tips and tricks on how to use qualitative research to better understand your audience and improve your ROI. Also learn the difference between qualitative and quantitative data.

Table of Contents

There is a fundamental distinction between data types: qualitative and quantitative. Typically, we call data ‘quantitative’ if it is in numerical form, and ‘qualitative’ if it’s not.

Marketers love to get into customers’ minds. But for that, they need to do a qualitative research. Face-to-face interviews, focus groups, or qualitative observations can provide valuable insights about your products, your market, and your customers’ opinions and motivations.

What is Qualitative Research

Qualitative research is a market research method that focuses on obtaining data through open-ended and conversational communication. This method focuses on the “why” rather than the “what” people think about you.

Let’s say you have an online shop that addresses a general audience. You do a demographic analysis and you find out that most of your customers are male. Naturally, you will want to find out  why  women are not buying from you. And that’s what qualitative research will help you find out.

Quantitative vs. Qualitative Research

Image source

Quantitative research is concerned with measurement and numbers, while qualitative research is concerned with understanding and words.

Quantitative research is used to quantify the problem. Its main goal is to generate numerical data or data that can be turned into statistics. It uses measurable data to formulate facts and uncover patterns in research.

Quantitative data collection methods include various forms of surveys (online surveys, paper surveys, mobile surveys, kiosk surveys, etc.), face-to-face interviews, telephone interviews, longitudinal studies, website interceptors, online polls, and systematic observations.

On the other hand, qualitative research is used to gain an understanding of underlying reasons, opinions, and motivations. It provides insights into the problem or helps to develop ideas or hypotheses for potential quantitative research.

Qualitative data collection methods include focus groups (group discussions), individual interviews, and participation/observation.

The statistical data of quantitative methods obtained from many people reveal a broad, generalizable set of findings. In contrast, qualitative methods produce a large amount of detailed information about a smaller number of people that results in rich understanding but reduces generalizability.

Qualitative Research Methodology

Once the marketer has decided that their research questions will provide data that is qualitative in nature, the next step is to choose the appropriate qualitative approach.

The approach chosen will take into account the purpose of the research, the role of the researcher, the data collected, the method of data analysis  and how the results will be presented. The most common approaches include:

• Narrative : explores the life of an individual, tells their story;
• Phenomenology : attempts to understand or explain life experiences or phenomena;
• Grounded theory : investigates the process, action, or interaction with the goal of developing a theory “grounded” in observations;
• Ethnography : describes and interprets an ethnic, cultural, or social group;
• Case study : examines episodic events in a definable framework, develops in-depth analyses of single or multiple cases, generally explains “how”.

Types of Qualitative Research Methods

Qualitative research methods are designed in a manner that they help reveal the behavior and perception of a target audience regarding a particular topic.

The most frequently used qualitative research methods are one-on-one interviews, focus groups, ethnographic research, case study research, record keeping, and qualitative observation.

1. One-on-one interviews

Conducting one-on-one interviews is one of the most common qualitative research methods. One of the advantages of this method is that it provides a great opportunity to gather precise data about what people think and their motivations.

Spending time talking to customers not only helps marketers understand who their clients are, but it also helps with customer care: clients love hearing from brands. This strengthens the relationship between a brand and its clients and paves the way for customer testimonials.

These interviews can be performed face-to-face or on the phone and usually last between half an hour and two hours or more.

When a one-on-one interview is conducted face-to-face, it also gives the marketer the opportunity to read the body language of the respondent and match the responses.

2. Focus groups

Focus groups are another commonly used qualitative research method. The ideal size of a focus group is usually between five and eight participants.

If the topic is of minor concern to participants, and if they have little experience with the topic, then a group size of 10 could be productive.

As the topic becomes more important, if people have more expertise on the topic, or if they are likely to have strong feelings about the topic, then the group size should be restricted to five or six people.

The main goal of a focus group is to find answers to the “why”, “what”, and “how” questions.

One advantage that focus groups have is that the marketer doesn’t necessarily have to interact with the group in person. Nowadays focus groups can be sent as online surveys on various devices.

Focus groups are an expensive option compared to the other qualitative research methods, which is why they are typically used to explain complex processes. Focus groups are especially useful when it comes to market research on new products and testing new concepts.

3. Ethnographic research

Ethnographic research is the most in-depth observational method that studies individuals in their naturally occurring environment.

This method aims at understanding the cultures, challenges, motivations, and settings that occur.

Ethnographic research requires the marketer to adapt to the target audiences’ environments (a different organization, a different city, or even a remote location), which is why geographical constraints can be an issue while collecting data.

This type of research can last from a few days to a few years. It’s challenging and time-consuming and solely depends on the expertise of the marketer to be able to analyze, observe, and infer the data.

4. Case study research

The case study method has grown into a valuable qualitative research method. This type of research method is usually used in education or social sciences.

Case study research may seem difficult to operate, but it’s actually one of the simplest ways of conducting research as it involves a deep dive and thorough understanding of the data collection methods and inferring the data.

5. Record keeping

Record keeping is similar to going to the library: you go over books or any other reference material to collect relevant data. This method uses already existing reliable documents and similar sources of information as a data source.

6. Qualitative observation

Qualitative observation is a method that uses subjective methodologies to gather systematic information or data. This method deals with the five major sensory organs and their functioning, sight, smell, touch, taste, and hearing.

Qualitative observation doesn’t involve measurements or numbers but instead characteristics.

Examples of Qualitative Research

1. online grocery shop with a predominantly male audience.

Let’s go back to the previous example. You have an online grocery shop. By nature, it addresses a general audience, but after you do a demographic analysis you find out that most of your customers are male.

One good method to determine why women are not buying from you is to hold one-on-one interviews with potential customers in the category.

Interviewing a sample of potential female customers should reveal why they don’t find your store appealing. The reasons could range from not stocking enough products for women to the fact that you also sell sex toys for example.

Tapping into different market segments will have a positive impact on your revenue.

2. Software company launching a new product

Focus groups are great for establishing product-market fit.

Let’s assume you are a software company who wants to launch a new product and you hold a focus group with 12 people. Although getting their feedback regarding users’ experience with the product is a good thing, this sample is too small to define how the entire market will react to your product.

So what you can do instead is holding multiple focus groups in 20 different geographic regions. Each region should be hosting a group of 12 for each market segment; you can even segment your audience based on age. This would be a better way to establish credibility in the feedback you receive.

3. Alan Peshkin’s “God’s Choice: The Total World of a Fundamentalist Christian School”

Moving from a fictional example to a real-life one, let’s analyze Alan Peshkin’s 1986 book “God’s Choice: The Total World of a Fundamentalist Christian School”.

Peshkin studied the culture of Bethany Baptist Academy by interviewing the students, parents, teachers, and members of the community alike, and spending eighteen months observing them to provide a comprehensive and in-depth analysis of Christian schooling as an alternative to public education.

Peshkin described Bethany Baptist Academy as having institutional unity of purpose, a dedicated faculty, an administration that backs teachers in enforcing classroom disciplines, cheerful students, rigorous homework, committed parents, and above all grounded in positive moral values and a character building environment.

However, it lacked cultural diversity, which meant that students were trained in one-dimensional thought, entirely cut off from viewpoints that differ with their teacher’s biblical interpretations, and a heavily censored library.

Even after discovering all this, Peshkin still presented the school in a positive light and stated that public schools have much to learn from such schools.

Peshkin’s in-depth study represents a qualitative research that uses observations and unstructured interviews, without any assumptions or hypothesis. He utilizes descriptive or non-quantifiable data on Bethany Baptist Academy specifically, without attempting to generalize the findings to other Christian schools.

4. Understanding buyers’ trends

Another way marketers can use quality research is to understand buyers’ trends. To do this, marketers need to look at historical data for both their company and their industry and identify where buyers are purchasing items in higher volumes.

For example, electronics distributors know that the holiday season is a peak market for sales while life insurance agents find that spring and summer wedding months are good seasons for targeting new clients.

5. Determining products/services missing from the market

Conducting your own research isn’t always necessary. If there are significant breakthroughs in your industry, you can use industry data and adapt it to your marketing needs.

The influx of hacking and hijacking of cloud-based information has made Internet security a topic of many industry reports lately. A software company could use these reports to better understand the problems his clients are facing.

As a result, the company can provide solutions prospects already know they need.

Real-time Customer Lifetime Value (CLV) Benchmark Report

See where your business stands compared to 1,000+ e-stores in different industries.

Limitations of qualitative research

The disadvantages of qualitative research are quite unique. The techniques of the data collector and their own unique observations can alter the information in subtle ways. That being said, these are the qualitative research’ limitations:

1. It’s a time-consuming process

The main drawback of qualitative research is that the process is time-consuming. Another problem is that the interpretations are limited. Personal experience and knowledge influence observations and conclusions.

Thus, a qualitative research might take several weeks or months. Also, since this process delves into personal interaction for data collection, discussions often tend to deviate from the main issue to be studied.

2. You can’t verify the results of qualitative research

Because qualitative research is open-ended, participants have more control over the content of the data collected. So the marketer is not able to verify the results objectively against the scenarios stated by the respondents.

3. It’s a labor-intensive approach

Qualitative research requires a labor-intensive analysis process such as categorization, recoding, etc. Similarly, qualitative research requires well-experienced marketers to obtain the needed data from a group of respondents.

4. It’s difficult to investigate causality

Qualitative research requires thoughtful planning to ensure the obtained results are accurate. There is no way to analyze qualitative data mathematically. This type of research is based more on opinion and judgment rather than results. Because all qualitative studies are unique they are difficult to replicate.

5. Qualitative research is not statistically representative

Because qualitative research is a perspective-based method of research, the responses given are not measured.

Comparisons can be made and this can lead toward duplication, but for the most part, quantitative data is required for circumstances which need statistical representation and that is not part of the qualitative research process.

While doing a qualitative research, it’s important to cross-reference the data obtained with the quantitative data. By continuously surveying prospects and customers marketers can build a stronger database of useful information.

Do Conversion Rate Optimization the Right way.

Explore helps you make the most out of your CRO efforts through advanced A/B testing, surveys, advanced segmentation and optimised customer journeys.

If you haven’t subscribed yet to our newsletter, now is your chance!

Like what you’re reading?

Join the informed ecommerce crowd.

Stay connected to what’s hot in eCommerce. We will never bug you with irrelevant info.

By clicking the Button, you confirm that you agree with our Terms and Conditions .

Continue your Conversion Rate Optimization Journey

• Last modified: January 3, 2023
• Conversion Rate Optimization , User Research

Valentin Radu

We’re a team of people that want to empower marketers around the world to create marketing campaigns that matter to consumers in a smart way. Meet us at the intersection of creativity, integrity, and development, and let us show you how to optimize your marketing.

Our Software

• > Book a Demo
• > Partner Program
• Blog Sitemap
• Terms and Conditions
• Privacy & Security
• Cookies Policy
• REVEAL Terms and Conditions

Scope and Delimitations in Academic Research

Table of contents

• 1.1 Examples of Elements Included in the Scope
• 2.1 Examples of Delimitations in Research
• 3 Determining the Scope and Delimitation
• 4 Writing the Scope and Delimitations Section
• 5 Conclusion

Understanding the scope and delimitations of a study is crucial for defining its parameters and ensuring focused research efforts. What are delimitations in a research study? These components establish the boundaries within which the research will operate and clarify what the study aims to explore and achieve. This article delves into the significance of clearly defining the scope and every delimitation, how they guide the research focus, and their roles in shaping the research process. Additionally, it provides insights into determining these aspects and articulating them effectively in a research proposal or paper. Transitioning smoothly into the main discussion, let’s explore the importance of scope in research, guiding the focus.

The importance of Clearly Defining the Scope of the Study for Guiding Research Focus

The scope of research delineates its extent or range of inquiry, setting clear parameters for what the study will cover. It’s a foundational aspect that guides every step of the research process, from the formulation of research questions to the interpretation of results. Defining the scope helps in focusing the research efforts, ensuring that the study remains manageable and within realistic bounds.

Understanding the scope and limitation of the study allows researchers to allocate resources efficiently, ensuring that every aspect of the study receives adequate attention. It also helps in avoiding the common pitfall of overreaching, which can dilute the research’s impact and make findings less actionable. By setting a defined scope, researchers can more easily communicate their work’s relevance, limitations and delimitations in the research process to stakeholders, enhancing the credibility and applicability of their findings. Furthermore, a well-defined scope can facilitate a more targeted and effective literature review, laying a solid foundation for the research study.

When navigating the complexities of defining a study’s scope, researchers might seek external support to ensure their research is concise, well-structured, and impactful. A writing service , PapersOwl offers a spectrum tailored to meet academic research’s unique demands. Their expertise can be particularly beneficial in refining research proposals, ensuring the scope is clearly communicated and aligned with academic standards. Engaging with such a service allows researchers to benefit from professional insights, which can enhance the coherence and focus of their work. This collaboration can be instrumental in identifying the most relevant study areas and avoiding unnecessary diversions. With PapersOwl’s support, researchers can ensure their project’s scope is well-defined and compellingly presented, making a strong case for its significance and feasibility. This partnership can be a strategic step towards achieving a study’s specific objectives, ensuring it contributes valuable insights within its defined boundaries.

Examples of Elements Included in the Scope

Defining the scope of a research project is akin to drawing a map for a journey; it outlines the terrain to be explored and the boundaries within which the exploration will occur. This clarity is essential for guiding the research process, ensuring the investigation remains focused and relevant. The scope encompasses various elements, each contributing to the overall direction and integrity of the study. Let’s delve into some of these key elements:

• Research Objectives : The specific aim the study is designed to achieve.
• Geographical Coverage: The physical or virtual locations where the research is conducted.
• Time Frame: The period during which the study takes place, which could range from a few days to several years.
• Subject: The specific topics or issues the research intends to address.
• Population Being Studied: The group of individuals, organizations, or phenomena being investigated.

These components of the scope serve as critical navigational tools in the research journey. They ensure that the study remains grounded in its objectives, relevant to its intended audience or population, and manageable within its temporal and geographical constraints. By carefully defining these elements at the outset, researchers can avoid common pitfalls such as scope creep, where the study’s focus broadens uncontrollably, potentially diluting its impact and significance. A well-defined scope is instrumental in crafting a focused, coherent, and impactful research project.

Role of Delimitations in Qualitative Research

Delimitations in research examples specify the boundaries set by the investigator on what the study will not cover, distinguishing them from limitations, which are potential weaknesses in the study not controlled by the researcher. Delimitations are choices made to narrow the scope of a study, focusing on specific aspects while excluding others. In the intricate tapestry of research design, delimitations play a pivotal role in sharpening the focus and enhancing the clarity of a study. By explicitly stating what the research will not explore, delimitations help prevent the dispersion of the research efforts across too broad an area, thereby increasing the depth and specificity of the investigation. This strategic narrowing allows researchers to concentrate their inquiries on areas most likely to yield impactful insights, making efficient use of available resources and time.

One might wonder how to establish these boundaries effectively without compromising the potential breadth of discovery. Here, the expertise provided by platforms like PapersOwl, particularly their research paper help service, becomes invaluable. Their seasoned professionals can offer guidance on crafting a research design that is both focused and flexible, assisting in identifying and justifying delimitations that enhance the study’s relevance and feasibility. Through such collaboration, researchers can balance the scope and delimitation of the study, ensuring that it remains grounded in its objectives while open to unforeseen insights.

Furthermore, acknowledging delimitations in a research paper demonstrates a researcher’s critical understanding of their study’s context and constraints, enhancing the credibility of their work. It shows a mindful engagement with the research process, recognizing that by setting deliberate boundaries, the study can delve more deeply and meaningfully into its chosen area of inquiry. Thus, when thoughtfully articulated with support from research paper writing help, like that offered by PapersOwl, delimitation in research becomes a testament to the rigor and integrity of its effort.

Examples of Delimitations in Research

Delimitations in research are akin to the guardrails on a highway; they keep the investigation on track and prevent it from veering into less relevant or overly broad territories. Below are some examples of how researchers can apply delimitations to fine-tune their investigations:

• Restricting the Study to Certain Age Groups: Focusing on a specific demographic, such as teenagers or the elderly.
• Geographic Locations: Limiting the research to a particular country, city, or region.
• Specific Periods: Studying a phenomenon during a particular time frame, ignoring other periods.

Setting these research delimitations is not about narrowing the vision of the research, but rather about sharpening its focus. It allows for a more thorough and nuanced exploration of the chosen subjects, leading to more precise findings and general delimitation meaning in research. Delimitations highlight the researcher’s awareness of the study’s scope and commitment to conducting a focused, manageable investigation.

Determining the Scope and Delimitation

Identifying the scope and delimitations of your research involves understanding the research problem deeply and recognizing what is feasible within the constraints of time, resources, and data availability. Strategies for determining these include:

• Reviewing existing literature to identify gaps and opportunities.
• Consulting with experts or advisors to refine research questions.
• Considering data availability and methodological constraints.

Balancing the scope and delimitations involves ensuring the research is neither too broad, unmanageable, nor too narrow, limiting its significance. Crafting a research project that strikes the right balance between breadth and depth is a nuanced task. It requires a researcher to be acutely aware of where their study begins and ends, what it encompasses, and what it intentionally leaves out. This equilibrium is not found in isolation but through a diligent exploration of the field and an understanding of how to best position one’s work within it. A key step in this process is identifying and sourcing relevant literature and data, which can significantly influence the scope of research.

Leveraging resources such as PapersOwl’s guide on how to find sources for research papers can prove invaluable in this phase. This platform provides insights into locating credible and relevant information, ensuring that researchers build their work upon a solid foundation of existing knowledge. By understanding how to navigate the vast, effective ocean of available data, researchers can make informed decisions about the direction and limits of their study. This meticulous preparation is crucial for defining the scope and delimitations and justifying them within the context of the research proposal or paper. It demonstrates a researcher’s commitment to rigor and depth, showing that their choices are informed by a comprehensive understanding of the subject and its existing body of literature.

Writing the Scope and Delimitations Section

Articulating the scope and delimitations in a research paper or proposal is crucial for setting clear expectations. It should clearly define delimitations and what the study will and will not cover, providing a rationale for these choices. Effective wording and structure involve:

• Stating the research objectives and questions upfront.
• Describing the research methodology , data collection methods and analysis.
• Outlining the geographical coverage, time frame, and subject matter.
• Clearly stating the delimitations and the reasons behind them.

The presentation of the scope and delimitations within a research document not only guides the readers through the intentions of the research but also establishes a framework for evaluating the findings. It’s a critical section where transparency and precision are paramount, allowing the audience to grasp the extent of the study and the rationale behind its boundaries. This transparency is essential for the credibility of the research, as it demonstrates a conscious and deliberate effort to focus the investigation and acknowledges the existence of boundaries that the study does not cross.

To ensure clarity and impact, this section should seamlessly integrate with the overall narrative of the research proposal or paper. Researchers are advised to avoid jargon and overly technical language, making the research scope and delimitations accessible to a broader audience. This includes a layperson who may not have deep expertise in the field but an interest in the study’s outcomes. Additionally, it is beneficial to highlight how the defined study scope and delimitations contribute to addressing the research problem, filling knowledge gaps, or exploring uncharted territories.

Moreover, this part of the document offers an opportunity to discuss how the chosen delimitations enhance the study’s focus and depth. By justifying the exclusions, researchers can address potential critiques head-on, reinforcing the methodological choices and underscoring the study’s contribution to the field. This careful articulation ensures that the research is perceived as a well-thought-out endeavor, grounded in a strategic approach to inquiry.

The scope and delimitations of a study are foundational elements that guide the research process, setting clear boundaries and focusing efforts. By defining these aspects clearly, researchers can provide a clear roadmap for their investigation, ensuring that their work is both manageable and relevant. By consciously deciding what to exclude from the study, researchers can intensify their focus on the chosen subject, ensuring that the research efforts are concentrated where they are most needed and can be most effective. These self-imposed boundaries are critical for maintaining the study’s coherence and depth. This clarity not only aids in conducting the research but also in effectively communicating its implications, limits, and outcomes.

Readers also enjoyed

WHY WAIT? PLACE AN ORDER RIGHT NOW!

Just fill out the form, press the button, and have no worries!

We use cookies to give you the best experience possible. By continuing we’ll assume you board with our cookie policy.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

SCOPE, LIMITATIONS, and DELIMITATIONS

Related Papers

Division of Palawan

Leo Andrew B . Diego

This module was designed and written with you in mind. It is here to help you master the Scope and Delimitation and Benefits and Beneficiaries of Research. The scope of this module permits it to be used in many different learning situations. The language used recognizes the diverse vocabulary level of students. The lessons are arranged to follow the standard sequence of the course. But the order in which you read them can be changed to correspond with the textbook you are now using. The module is divided into Two (2) lessons, namely: Lesson 1- Scope and Delimitation of research Lesson 2- Benefits and Beneficiaries of research After going through this module, you are expected to: a. define scope and delimitation of research; b. appreciate the scope, limitation and delimitation; and, c. write the benefits and beneficiaries of research.

Asfarasin Maricar

Tourism studies

Journal of Museum Education

Journal of Public Health

Belinda Burford

Satyajit Behera

Satish Saluja

Saeed Anwar

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

• Open access
• Published: 10 April 2024

“So at least now I know how to deal with things myself, what I can do if it gets really bad again”—experiences with a long-term cross-sectoral advocacy care and case management for severe multiple sclerosis: a qualitative study

• Anne Müller   ORCID: orcid.org/0000-0002-2456-2492 1 ,
• Fabian Hebben   ORCID: orcid.org/0009-0003-6401-3433 1 ,
• Kim Dillen 1 ,
• Veronika Dunkl 1 ,
• Yasemin Goereci 2 ,
• Raymond Voltz 1 , 3 , 4 ,
• Peter Löcherbach 5 ,
• Clemens Warnke   ORCID: orcid.org/0000-0002-3510-9255 2 &
• Heidrun Golla   ORCID: orcid.org/0000-0002-4403-630X 1

on behalf of the COCOS-MS trial group represented by Martin Hellmich

BMC Health Services Research volume  24 , Article number:  453 ( 2024 ) Cite this article

121 Accesses

Metrics details

Persons with severe Multiple Sclerosis (PwsMS) face complex needs and daily limitations that make it challenging to receive optimal care. The implementation and coordination of health care, social services, and support in financial affairs can be particularly time consuming and burdensome for both PwsMS and caregivers. Care and case management (CCM) helps ensure optimal individual care as well as care at a higher-level. The goal of the current qualitative study was to determine the experiences of PwsMS, caregivers and health care specialists (HCSs) with the CCM.

In the current qualitative sub study, as part of a larger trial, in-depth semi-structured interviews with PwsMS, caregivers and HCSs who had been in contact with the CCM were conducted between 02/2022 and 01/2023. Data was transcribed, pseudonymized, tested for saturation and analyzed using structuring content analysis according to Kuckartz. Sociodemographic and interview characteristics were analyzed descriptively.

Thirteen PwsMS, 12 caregivers and 10 HCSs completed interviews. Main categories of CCM functions were derived deductively: (1) gatekeeper function, (2) broker function, (3) advocacy function, (4) outlook on CCM in standard care. Subcategories were then derived inductively from the interview material. 852 segments were coded. Participants appreciated the CCM as a continuous and objective contact person, a person of trust (92 codes), a competent source of information and advice (on MS) (68 codes) and comprehensive cross-insurance support (128 codes), relieving and supporting PwsMS, their caregivers and HCSs (67 codes).

Conclusions

Through the cross-sectoral continuous support in health-related, social, financial and everyday bureaucratic matters, the CCM provides comprehensive and overriding support and relief for PwsMS, caregivers and HCSs. This intervention bears the potential to be fine-tuned and applied to similar complex patient groups.

Trial registration

The study was approved by the Ethics Committee of the University of Cologne (#20–1436), registered at the German Register for Clinical Studies (DRKS00022771) and in accordance with the Declaration of Helsinki.

Peer Review reports

Introduction

Multiple sclerosis (MS) is the most frequent and incurable chronic inflammatory and degenerative disease of the central nervous system (CNS). Illness awareness and the number of specialized MS clinics have increased since the 1990s, paralleled by the increased availability of disease-modifying therapies [ 1 ]. There are attempts in the literature for the definition of severe MS [ 2 , 3 ]. These include a high EDSS (Expanded disability Status Scale [ 4 ]) of ≥ 6, which we took into account in our study. There are also other factors to consider, such as a highly active disease course with complex therapies that are associated with side effects. These persons are (still) less disabled, but may feel overwhelmed with regard to therapy, side effects and risk monitoring of therapies [ 5 , 6 ].

Persons with severe MS (PwsMS) develop individual disease trajectories marked by a spectrum of heterogeneous symptoms, functional limitations, and uncertainties [ 7 , 8 ] manifesting individually and unpredictably [ 9 ]. This variability can lead to irreversible physical and mental impairment culminating in complex needs and daily challenges, particularly for those with progressive and severe MS [ 5 , 10 , 11 ]. Such challenges span the spectrum from reorganizing biographical continuity and organizing care and everyday live, to monitoring disease-specific therapies and integrating palliative and hospice care [ 5 , 10 ]. Moreover, severe MS exerts a profound of social and economic impact [ 9 , 12 , 13 , 14 ]. PwsMS and their caregivers (defined in this manuscript as relatives or closely related individuals directly involved in patients’ care) often find themselves grappling with overwhelming challenges. The process of organizing and coordinating optimal care becomes demanding, as they contend with the perceived unmanageability of searching for, implementing and coordinating health care and social services [ 5 , 15 , 16 , 17 ].

Case management (CM) proved to have a positive effect on patients with neurological disorders and/or patients with palliative care needs [ 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 ]. However, a focus on severe MS has been missed so far Case managers primarily function as: (1) gatekeeper involving the allocation of necessary and available resources to a case, ensuring the equitable distribution of resources; as (2) broker assisting clients in pursuing their interests, requiring negotiation to provide individualized assistance that aligns as closely as possible with individual needs and (3) advocate working to enhance clients’ individual autonomy, to advocate for essential care offers, and to identify gaps in care [ 25 , 26 , 27 , 28 , 29 ].

Difficulties in understanding, acting, and making decisions regarding health care-related aspects (health literacy) poses a significant challenge for 54% of the German population [ 30 ]. Additionally acting on a superordinate level as an overarching link, a care and case management (CCM) tries to reduce disintegration in the social and health care system [ 31 , 32 ]. Our hypothesis is that a CCM allows PwsMS and their caregivers to regain time and resources outside of disease management and to facilitate the recovery and establishment of biographical continuity that might be disrupted due to severe MS [ 33 , 34 ].

Health care specialists (HCSs) often perceive their work with numerous time and economic constraints, especially when treating complex and severely ill individuals like PwsMS and often have concerns about being blamed by patients when expectations could not be met [ 35 , 36 ]. Our hypothesis is that the CCM will help to reduce time constraints and free up resources for specialized tasks.

To the best of our knowledge there is no long-term cross-sectoral and outreaching authority or service dedicated to assisting in the organization and coordination of the complex care concerns of PwsMS within the framework of standard care addressing needs in health, social, financial, every day and bureaucratic aspects. While some studies have attempted to design and test care programs for persons with MS (PwMS), severely affected individuals were often not included [ 37 , 38 , 39 ]. They often remain overlooked by existing health and social care structures [ 5 , 9 , 15 ].

The COCOS-MS trial developed and applied a long-term cross-sectoral CCM intervention consisting of weekly telephone contacts and monthly re-assessments with PwsMS and caregivers, aiming to provide optimal care. Their problems, resources and (unmet) needs were assessed holistically including physical health, mental health, self-sufficiency and social situation and participation. Based on assessed (unmet) needs, individual care plans with individual actions and goals were developed and constantly adapted during the CCM intervention. Contacts with HCSs were established to ensure optimal care. The CCM intervention was structured through and documented in a CCM manual designed for the trial [ 40 , 41 ].

Our aim was to find out how PwsMS, caregivers and HCSs experienced the cross-sectoral long-term, outreaching patient advocacy CCM.

This study is part of a larger phase II, randomized, controlled clinical trial “Communication, Coordination and Security for people with severe Multiple Sclerosis (COCOS-MS)” [ 41 ]. This explorative clinical trial, employing a mixed-method design, incorporates a qualitative study component with PwsMS, caregivers and HCSs to enrich the findings of the quantitative data. This manuscript focuses on the qualitative data collected between February 2022 and January 2023, following the Consolidated Criteria for Reporting Qualitative Research (COREQ) guidelines [ 42 ].

Research team

Three trained authors AM, KD and FH (AM, female, research associate, M.A. degree in Rehabilitation Sciences; KD, female, researcher, Dr. rer. medic.; FH, male, research assistant, B.Sc. degree in Health Care Management), who had no prior relationship with patients, caregivers or HCSs conducted qualitative interviews. A research team, consisting of clinical experts and health services researchers, discussed the development of the interview guides and the finalized category system.

Theoretical framework

Interview data was analyzed with the structuring content analysis according to Kuckartz. This method enables a deductive structuring of interview material, as well as the integration of new aspects found in the interview material through the inductive addition of categories in an iterative analysis process [ 43 ].

Sociodemographic and interview characteristics were analyzed descriptively (mean, median, range, SD). PwsMS, caregivers and HCSs were contacted by the authors AM, KD or FH via telephone or e-mail after providing full written informed consent. Participants had the option to choose between online interviews conducted via the GoToMeeting 10.19.0® Software or face-to-face. Peasgood et al. (2023) found no significant differences in understanding questions, engagement or concentration between face-to-face and online interviews [ 44 , 45 ]. Digital assessments were familiar to participants due to pandemic-related adjustments within the trial.

Out of 14 PwsMS and 14 caregivers who were approached to participate in interviews, three declined to complete interviews, resulting in 13 PwsMS (5 male, 8 female) and 12 caregiver (7 male, 5 female) interviews, respectively (see Fig.  1 ). Thirty-one HCSs were contacted of whom ten (2 male, 8 female) agreed to be interviewed (see Fig.  2 ).

Flowchart of PwsMS and caregiver participation in the intervention group of the COCOS-MS trial. Patients could participate with and without a respective caregiver taking part in the trial. Therefore, number of caregivers does not correspond to patients. For detailed inclusion criteria see also Table  1 in Golla et al. [ 41 ]

Flowchart of HCSs interview participation

Setting and data collection

Interviews were carried out where participants preferred, e.g. at home, workplace, online, and no third person being present. In total, we conducted 35 interviews whereof 7 interviews face-to-face (3 PwsMS, 3 caregivers, 1 HCS).

The research team developed a topic guide which was meticulously discussed with research and clinical staff to enhance credibility. It included relevant aspects for the evaluation of the CCM (see Tables  1 and 2 , for detailed topic guides see Supplementary Material ). Patient and caregiver characteristics (covering age, sex, marital status, living situation, EDSS (patients only), subgroup) were collected during the first assessment of the COCOS-MS trial and HCSs characteristics (age, sex, profession) as well as interview information (length and setting) were collected during the interviews. The interview guides developed for this study addressed consistent aspects both for PwsMS and caregivers (see Supplementary Material ):

For HCSs it contained the following guides:

Probing questions were asked to get more specific and in-depth information. Interviews were carried out once and recorded using a recording device or the recording function of the GoToMeeting 10.19.0® Software. Data were pseudonymized (including sensitive information, such as personal names, dates of birth, or addresses), audio files were safely stored in a data protection folder. The interview duration ranged from 11 to 56 min (mean: 23.9 min, SD: 11.1 min). Interviews were continued until we found that data saturation was reached. Audio recordings were transcribed verbatim by an external source and not returned to participants.

Data analysis

Two coders (AM, FH) coded the interviews. Initially, the first author (AM) thoroughly reviewed the transcripts to gain a sense of the interview material. Using the topic guide and literature, she deductively developed a category system based on the primary functions of CM [ 25 , 26 , 27 , 28 , 29 ]. Three interviews were coded repeatedly for piloting, and inductive subcategories were added when new themes emerged in the interview material. This category system proved suitable for the interview material. The second coder (FH) familiarized himself with the interview material and category system. Both coders (AM, FH) independently coded all interviews, engaging in discussions and adjusting codes iteratively. The finalized category system was discussed and consolidated in a research workshop and within the COCOS-MS trial group and finally we reached an intercoder agreement of 90% between the two coders AM and FH, computed by the MAXQDA Standard 2022® software.

We analyzed sociodemographic and interview characteristics using IBM SPSS Statistics 27® and Excel 2016®. Transcripts were managed and analyzed using MAXQDA Standard 2022®.

Participants were provided with oral and written information about the trial and gave written informed consent. Ethical approvals were obtained from the Ethics Committee of the University of Cologne (#20–1436). The trial is registered in the German Register for Clinical Studies (DRKS) (DRKS00022771) and is conducted under the Declaration of Helsinki.

Characteristics of participants and interviews

PwsMS participating in an interview were mainly German (84.6%), had a mean EDSS of 6.8 (range: 6–8) and MS for 13.5 years (median: 14; SD: 8.1). For detailed characteristics see Table  3 .

Most of the interviewed caregivers (9 caregivers) were the partners of the PwsMS with whom they lived in the same household. For further details see Table  3 .

HCSs involved in the study comprised various professions, including MS-nurse (3), neurologist (2), general physician with further training in palliative care (1), physician with further training in palliative care and pain therapist (1), housing counselling service (1), outpatient nursing service manager (1), participation counselling service (1).

Structuring qualitative content analysis

The experiences of PwsMS, caregivers and HCSs were a priori deductively assigned to four main categories: (1) gatekeeper function, (2) broker function, (3) advocacy function [ 25 , 26 , 27 , 28 , 29 ] and (4) Outlook on CCM in standard care, whereas the subcategories were developed inductively (see Fig.  3 ).

Category system including main and subcategories of the qualitative thematic content analysis

The most extensive category, housing the highest number of codes and subcodes, was the “ Outlook on CCM in standard care ” (281 codes). Following this, the category “ Advocacy Function ” contained 261 codes. The “ Broker Function ” (150 codes) and the “ Gatekeeper Function ” (160 codes) constituted two smaller categories. The majority of codes was identified in the caregivers’ interviews, followed by those of PwsMS (see Table  4 ). Illustrative quotes for each category and subcategory can be found in Table  5 .

Persons with severe multiple sclerosis

In the gatekeeper function (59 codes), PwsMS particularly valued the CCM as a continuous contact person . They appreciated the CCM as a person of trust who was reliably accessible throughout the intervention period. This aspect, with 41 codes, held significant importance for PwsMS.

Within the broker function (44 codes), establishing contact was most important for PwsMS (22 codes). This involved the CCM as successfully connecting PwsMS and caregivers with physicians and therapists, as well as coordinating and arranging medical appointments, which were highly valued. Assistance in authority and health and social insurance matters (10 codes) was another subcategory, where the CCM encompassed support in communication with health insurance companies, such as improving the level of care, assisting with retirement pension applications, and facilitating rehabilitation program applications. Optimized care (12 codes) resulted in improved living conditions and the provision of assistive devices through the CCM intervention.

The advocacy function (103 codes) emerged as the most critical aspect for PwsMS, representing the core of the category system. PwsMS experienced multidimensional, comprehensive, cross-insurance system support from the CCM. This category, with 43 statements, was the largest within all subcategories. PwsMS described the CCM as addressing their concerns, providing help, and assisting with the challenges posed by the illness in everyday life. The second-largest subcategory, regaining, maintaining and supporting autonomy (25 codes), highlighted the CCM’s role in supporting self-sufficiency and independence. Reviving personal wellbeing (17 codes) involved PwsMSs’ needs of regaining positive feelings, improved quality of life, and a sense of support and acceptance, which could be improved by the CCM. Temporal relief (18 codes) was reported, with the CCM intervention taking over or reducing tasks.

Within the outlook on CCM in standard care (84 codes), eight subcategories were identified. Communications was described as friendly and open (9 codes), with the setting of communication (29 codes) including the frequency of contacts deemed appropriate by the interviewed PwsMS, who preferred face-to-face contact over virtual or telephone interactions. Improvement suggestions for CCM (10 codes) predominantly revolved around the desire for the continuation of the CCM beyond the trial, expressing intense satisfaction with the CCM contact person and program. PwsMS rarely wished for better cooperation with the CCM. With respect to limitations (7 codes), PwsMS distinguished between individual limitations (e.g. when not feeling ready for using a wheelchair) and overriding structural limitations (e.g. unsuccessful search for an accessible apartment despite CCM support). Some PwsMS mentioned needing the CCM earlier in the course of the disease and believed it would beneficial for anyone with a chronic illness (6 codes).

In the gatekeeper function (75 codes), caregivers highly valued the CCM as a continuous contact partner (33 codes). More frequently than among the PwsMS interviewed, caregivers valued the CCM as a source of consultation/ information on essential individual subjects (42 codes). The need for basic information about the illness, its potential course, treatment and therapy options, possible supportive equipment, and basic medical advice/ information could be met by the CCM.

Within the broker function (63 codes), caregivers primarily experienced the subcategory establish contacts (24 codes). They found the CCM as helpful in establishing and managing contact with physicians, therapists and especially with health insurance companies. In the subcategory assistance in authority and health and social insurance matters (22 codes), caregivers highlighted similar aspects as the PwsMS interviewed. However, there was a particular emphasis on assistance with patients' retirement matters. Caregivers also valued the optimization of patients’ care and living environment (17 codes) in various life areas during the CCM intervention, including improved access to assistive devices, home modification, and involvement of a household support and/ or nursing services.

The advocacy function, with 115 codes, was by far the broadest category . The subcategory multidimensional, comprehensive, cross-insurance system support represented the largest subcategory of caregivers, with 70 statements. In summary, caregivers felt supported by the CCM in all domains of life. Regaining, maintaining and supporting autonomy (11 codes) and reviving personal wellbeing (8 codes) in the form of an improved quality of life played a role not only for patients but also for caregivers, albeit to a lower extend. Caregivers experienced temporal relief (26 codes) as the CCM undertook a wide range of organizational tasks, freeing up more needed resources for their own interests.

For the Outlook on CCM in standard care , caregivers provided various suggestions (81 codes). Similar to PwsMS, caregivers felt that setting (home based face-to-face, telephone, virtual) and frequency of contact were appropriate (10 codes, communication setting ) and communications (7 codes) were recognized as open and friendly. However, to avoid conflicts between caregiver and PwsMS, caregivers preferred meeting the CCM separately from the PwsMS in the future. Some caregivers wished the CCM to specify all services it might offer at the beginning, while others emphasized not wanting this. Like PwsMS, caregivers criticized the CCM intervention being (trial-related) limited to one year, regardless of whether further support was needed or processes being incomplete (13 codes, improvement suggestions ). After the CCM intervention time had expired, the continuous contact person and assistance were missed and new problems had arisen and had to be managed with their own resources again (9 codes, effects of CCM discontinuation ), which was perceived as an exhausting or unsolvable endeavor. Caregivers identified analogous limitations (8 codes), both individual and structural. However, the largest subcategory, was the experienced potential of CCM (27 codes), reflected in extremely high satisfaction with the CCM intervention. Like PwsMS, caregivers regarded severe chronically ill persons in general as target groups for a CCM (7 codes) and would implement it even earlier, starting from the time of diagnosis. They considered a CCM to be particularly helpful for patients without caregivers or for caregivers with limited (time) resources, as it was true for most caregivers.

Health care specialists

In the gatekeeper function (26 codes) HCSs particularly valued the CCM as a continuous contact partner (18 codes). They primarily described their valuable collaboration with the CCM, emphasizing professional exchange between the CCM and HCSs.

Within the broker function (43 codes), the CCM was seen as a connecting link between patients and HCSs, frequently establishing contacts (18 codes). This not only improved optimal care on an individual patient level (case management) but also at a higher, superordinate care level (care management). HCSs appreciated the optimized care and living environment (18 codes) for PwsMS, including improved medical and therapeutic access and the introduction of new assistive devices. The CCM was also recognized as providing assistance in authority and health and social matters (7 codes) for PwsMS and their caregivers.

In the advocacy function (43 codes), HCSs primarily reported temporal relief through CCM intervention (23 codes). They experienced this relief, especially as the CCM provided multidimensional, comprehensive, and cross-insurance system support (15 codes) for PwsMS and their caregivers. Through this support, HCSs felt relieved from time intensive responsibilities that may not fall within their area of expertise, freeing up more time resources for their actual professional tasks.

The largest category within the HCSs interviews was the outlook on CCM in standard care (116 codes). In the largest subcategory, HCSs made suggestions for further patient groups who could benefit (38 codes) from a CCM. Chronic neurological diseases like neurodegenerative diseases (e.g. amyotrophic lateral sclerosis), typical and atypical Parkinson syndromes were mentioned. HCSs considered the enrollment of the CCM directly after the diagnosis of these complex chronic diseases. Additionally, chronic progressive diseases in general or oncological diseases, which may also run chronically, were regarded worthwhile for this approach. HCSs also provided suggestions regarding improvement (21 codes). They wished e.g. for information or contact when patients were enrolled to the CCM, regular updates, exchange and collaborative effort. On the other hand, HCSs reported, that their suggestions for improvement would hardly be feasible due to their limited time resources. Similar to patients and caregivers, HCSs experienced structural limits (13 codes), which a CCM could not exceed due to overriding structural limitations (e.g. insufficient supply of (household) aids, lack of outreach services like psychotherapists, and long processing times on health and pension insurers' side). HCSs were also asked about their opinions on financial resources (14 codes) of a CCM in standard care. All interviewed HCSs agreed that CCM would initially cause more costs for health and social insurers, but they were convinced of cost savings in the long run. HCSs particularly perceived the potential of the CCM (20 codes) through the feedback of PwsMS, highlighting the trustful relationship enabling individualized help for PwsMS and their caregivers.

Persons with severe multiple sclerosis and their caregivers

The long-term cross-sectoral CCM intervention implemented in the COCOS-MS trial addressed significant unmet needs of PwsMS and their caregivers which previous research revealed as burdensome and hardly or even not possible to improve without assistance [ 5 , 6 , 9 , 10 , 33 , 35 , 46 ]. Notably, the CCM service met the need for a reliable, continuous contact partner, guiding patients through the complexities of regulations, authorities and the insurance system. Both, PwsMS and their caregivers highly valued the professional, objective perspective provided by the CCM, recognizing it as a source of relief, support and improved care in line with previous studies [ 37 , 47 ]. Caregivers emphasized the CCM’s competence in offering concrete assistance and information on caregiving and the fundamentals of MS, including bureaucratic, authority and insurances matters. On the other hand, PwsMS particularly appreciated the CCMs external reflective and advisory function, along with empathic social support tailored to their individual concerns. Above all, the continuous partnership of trust, available irrespective of the care sector, was a key aspect that both PwsMS and their caregivers highlighted. This consistent support was identified as one of the main components in the care of PwsMS in previous studies [ 5 , 33 , 35 ].

As the health literacy is inadequate or problematic for 54% of the German population and disintegration in the health and social care system is high [ 30 , 31 , 32 ], the CCM approach serves to enhance health literacy and reduce disintegration of PwsMS and their caregivers by providing cross-insurance navigational guidance in the German health and social insurance sector on a superordinate level. Simultaneously PwsMS and caregivers experienced relief and gained more (time) resources for all areas of life outside of the disease and its management, including own interests and establishing biographical continuity. This empowerment enables patients to find a sense of purpose beyond their illness, regain autonomy, and enhance social participation, reducing the feeling of being a burden to those closest to them. Such feelings are often experienced as burdensome and shameful by PwsMS [ 6 , 48 , 49 , 50 ]. Finding a sense of purpose beyond the illness also contributes to caregivers perceiving their loved ones not primarily as patient but as individuals outside of the disease, reinforcing valuable relationships such as partners, siblings, or children, strengthening emotional bonds. These factors are also highly relevant and well-documented in a suicide-preventive context, as the suicide rate is higher in persons diagnosed with neurological disorders [ 19 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ] and the feeling of being a burden to others, loss of autonomy, and perceived loss of dignity are significant factors in patients with severe chronic neurological diseases for suicide [ 50 , 57 ].

The temporal relief experienced by the CCM was particularly significant for HCSs and did not only improve the satisfaction of HCSs but also removed unfulfilled expectations and concerns about being blamed by patients when expectations could not be met, which previous studied elaborated [ 35 , 36 ]. Moreover, the CCM alleviated the burden on HCSs by addressing patients’ concerns, allowing them to focus on their own medical responsibilities. This aspect probably reduced the dissatisfaction that arises when HCSs are expected to address issues beyond their medical expertise, such as assistive devices, health and social insurance, and the organization and coordination of supplementary therapies, appointments, and contacts [ 35 , 36 , 61 ]. Consequently, the CCM reduced difficulties of HCSs treating persons with neurological or chronical illnesses, which previous research identified as problematic.

HCSs perceive their work as increasingly condensed with numerous time and economic constraints, especially when treating complex and severely ill individuals like PwsMS [ 36 ]. This constraint was mentioned by HCSs in the interviews and was one of the main reasons why they were hesitant to participate in interviews and may also be an explanation for a shorter interview duration than initially planned in the interview guides. The CCM’s overarching navigational competence in the health and social insurance system was particularly valued by HCSs. The complex and often small-scale specialties in the health and social care system are not easily manageable or well-known even for HCSs, and dealing with them can exceed their skills and time capacities [ 61 ]. The CCM played a crucial role in keeping (temporal) resources available for what HCSs are professionally trained and qualified to work on. However, there remains a challenge in finding solutions to the dilemma faced by HCSs regarding their wish to be informed about CCM procedures and linked with each other, while also managing the strain of additional requests and contact with the CCM due to limited (time) resources [ 62 ]. Hudon et al. (2023) suggest that optimizing time resources and improving exchange could involve meetings, information sharing via fax, e-mail, secure online platforms, or, prospectively, within the electronic patient record (EPR). The implementation of an EPR has shown promise in improving the quality of health care and time resources, when properly implemented [ 63 , 64 ]. The challenge lies ineffective information exchange between HCSs and CCM for optimal patient care. The prospect of time saving in the long run and at best for a financial incentive, e.g., when anchoring in the Social Security Code, will help best to win over the HCSs.If this crucial factor can be resolved, there is a chance that HCSs will thoroughly accept the CCM as an important pillar, benefiting not only PwsMS but also other complex patient groups, especially those with long-term neurological or complex oncological conditions that might run chronically.

Care and case management and implications for the health care system

The results of our study suggest that the cross-sectoral long-term advocacy CCM in the COCOS-MS trial, with continuous personal contacts at short intervals and constant reevaluation of needs, problems, resources and goals, is highly valued by PwsMS, caregivers, and HCSs. The trial addresses several key aspects that may have been overlooked in previous studies which have shown great potential for the integration of case management [ 17 , 47 , 62 , 65 , 66 ]. However, they often excluded the overriding care management, missed those patient groups with special severity and complexity who might struggle to reach social and health care structures independently or the interventions were not intended for long-term [ 22 , 37 ]. Our results indicate that the CCM intervention had a positive impact on PwsMS and caregivers as HCSs experienced them with benefits such as increased invigoration, reduced demands, and enhanced self-confidence. However, there was a notable loss experienced by PwsMS and caregivers after the completion of the CCM intervention, even if they had stabilized during the intervention period. The experiences of optimized social and health care for the addressed population, both at an individual and superordinate care level, support the integration of this service into standard care. Beyond the quantitatively measurable outcomes and economic considerations reported elsewhere [ 16 , 20 , 21 ], our results emphasize the importance of regaining control, self-efficacy, self-worth, dignity, autonomy, and social participation. These aspects are highlighted as preventive measures in suicidal contexts, which is particularly relevant for individuals with severe and complex illnesses [ 19 , 50 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ]. Our findings further emphasize the societal responsibilities to offer individuals with severe and complex illnesses the opportunity to regain control and meaningful aspects of life, irrespective of purely economic considerations. This underscores the need for a comprehensive evaluation that not only takes into account quantitative measures but also the qualitative aspects of well-being and quality of life when making recommendations of a CCM in standard care.

The study by J. Y. Joo and Huber (2019) highlighted that CM interventions aligned with the standards of the Case Management Society of America varied in duration, ranging from 1 month to 15.9 years, and implemented in community- or hospital-based settings. However, they noted a limitation in understanding how CM processes unfold [ 67 ]. In contrast, our trial addressed this criticism by providing transparent explanations of the CCM process, which also extends to a superordinate care management [ 40 , 41 ]. Our CCM manual [ 40 ] outlines a standardized and structured procedure for measuring and reevaluating individual resources, problems, and unmet needs on predefined dimensions. It also identifies goals and actions at reducing unmet needs and improving the individual resources of PwsMS and caregivers. Importantly, the CCM manual demonstrates that the CCM process can be structured and standardized, while accounting for the unique aspects of each individual’s serious illness, disease courses, complex needs, available resources, and environmental conditions. Furthermore, the adaptability of the CCM manual to other complex chronically ill patient groups suggests the potential for a standardized approach in various health care settings. This standardized procedure allows for consistency in assessing and addressing the individual needs of patients, ensuring that the CCM process remains flexible while maintaining a structured and goal-oriented framework.

The discussion about the disintegration in the social and health care system and the increasing specialization dates back to 2009 [ 31 , 32 ]. Three strategies were identified to address this issue: (a) “driver-minimizing” [Treiberminimierende], (b) “effect-modifying” [Effektmodifizierende] and (c) “disintegration-impact-minimizing” [Desintegrationsfolgenminimierende] strategies. “Driver-minimizing strategies” involve comprehensive and radical changes within the existing health and social care system, requiring political and social pursuit. “Disintegration-impact-minimizing strategies” are strategies like quality management or tele-monitoring, which are limited in scope and effectiveness. “Effect-modifying strategies”, to which CCM belongs, acknowledges the segmentation within the system but aims to overcome it through cooperative, communicative, and integrative measures. CCM, being an “effect-modifying strategy”, operates the “integrated segmentation model” [Integrierte Segmentierung] rather than the “general contractor model” [Generalunternehmer-Modell] or “total service provider model” [Gesamtdienstleister-Modell] [ 31 , 32 ]. In this model, the advantage lies in providing an overarching and coordinating service to link different HCSs and services cross-sectorally. The superordinate care management aspect of the CCM plays a crucial role in identifying gaps in care, which is essential for future development strategies within the health and social care system. It aims to find or develop (regional) alternatives to ensure optimal care [ 17 , 23 , 24 , 68 , 69 ], using regional services of existing health and social care structures. Therefore, superordinate care management within the CCM process is decisive for reducing disintegration in the system.

Strengths and limitations

The qualitative study results of the explorative COCOS-MS clinical trial, which employed an integrated mixed-method design, provide valuable insights into the individual experiences of three leading stakeholders: PwsMS, caregivers and HCSs with a long-term cross-sectoral CCM. In addition to in-depth interviews, patient and caregiver reported outcome measurements were utilized and will be reported elsewhere. The qualitative study’s strengths include the inclusion of patients who, due to the severity of their condition (e.g. EDSS mean: 6.8, range: 6–8, highly active MS), age (mean: 53.9 years, range: 36–73 years) family constellations, are often underrepresented in research studies and often get lost in existing social and health care structures. The study population is specific to the wider district region of Cologne, but the broad inclusion criteria make it representative of severe MS in Germany. The methodological approach of a deductive and inductive structuring content analysis made it possible to include new findings into an existing theoretical framework.

However, the study acknowledges some limitations. While efforts were made to include more HCSs, time constraints on their side limited the number of interviews conducted and might have biased the results. Some professions are underrepresented in the interviews. Complex symptoms (e.g. fatigue, ability to concentrate), medical or therapeutic appointments and organization of the everyday live may have been reasons for the patients’ and caregivers’ interviews lasting shorter than initially planned.

The provision of functions of a CCM, might have pre-structured the answers of the participants.

At current, there is no support system for PwsMS, their caregivers and HCSs that addresses their complex and unmet needs comprehensively and continuously. There are rare qualitative insights of the three important stakeholders: PwsMS, caregivers and HCSs in one analysis about a supporting service like a CCM. In response to this gap, we developed and implemented a long-term cross-sectoral advocacy CCM and analyzed it qualitatively. PwsMS, their caregivers and HCSs expressed positive experiences, perceiving the CCM as a source of relief and support that improved care across various aspects of life. For patients, the CCM intervention resulted in enhanced autonomy, reviving of personal wellbeing and new established contacts with HCSs. Caregivers reported a reduced organizational burden and felt better informed, and HCSs experienced primarily temporal relief, allowing them to concentrate on their core professional responsibilities. At a higher level of care, the study suggests that the CCM contributed to a reduction in disintegration within the social and health care system.

The feedback from participants is seen as valuable for adapting the CCM intervention and the CCM manual for follow-up studies, involving further complex patient groups such as neurological long-term diseases apart from MS and tailoring the duration of the intervention depending on the complexity of evolving demands.

Availability of data and materials

Generated and/or analyzed datasets of participants are available from the corresponding author on reasonable request to protect participants. Preliminary partial results have been presented as a poster during the EAPC World Congress in June 2023 and the abstract has been published in the corresponding abstract booklet [ 70 ].

Abbreviations

Amyotrophic lateral sclerosis

• Care and case management

Case management

Central nervous system

Communication, Coordination and security for people with multiple sclerosis

Consolidated criteria for reporting qualitative research

German register for clinical studies

Extended disability status scale

Electronic patient record

Quality of life

Multiple sclerosis

Koch-Henriksen N, Magyari M. Apparent changes in the epidemiology and severity of multiple sclerosis. Nat Rev Neurol. 2021;17:676–88. https://doi.org/10.1038/s41582-021-00556-y .

Article   PubMed   Google Scholar  

Ellenberger D, Flachenecker P, Fneish F, Frahm N, Hellwig K, Paul F, et al. Aggressive multiple sclerosis: a matter of measurement and timing. Brain. 2020;143:e97. https://doi.org/10.1093/brain/awaa306 .

Article   PubMed   PubMed Central   Google Scholar  

Edmonds P, Vivat B, Burman R, Silber E, Higginson IJ. Loss and change: experiences of people severely affected by multiple sclerosis. Palliat Med. 2007;21:101–7. https://doi.org/10.1177/0269216307076333 .

Kurtzke JF. Rating neurologic impairment in multiple rating neurologic impairment in multiple sclerosis: An expanded disability status scale (EDSS). Neurology. 1983;33(11):1444–52.

Article   CAS   PubMed   Google Scholar  

Galushko M, Golla H, Strupp J, Karbach U, Kaiser C, Ernstmann N, et al. Unmet needs of patients feeling severely affected by multiple sclerosis in Germany: a qualitative study. J Palliat Med. 2014;17:274–81. https://doi.org/10.1089/jpm.2013.0497 .

Borreani C, Bianchi E, Pietrolongo E, Rossi I, Cilia S, Giuntoli M, et al. Unmet needs of people with severe multiple sclerosis and their carers: qualitative findings for a home-based intervention. PLoS One. 2014:e109679. https://doi.org/10.1371/journal.pone.0109679 .

Yamout BI, Alroughani R. Multiple Sclerosis. Semin Neurol. 2018;38:212–25. https://doi.org/10.1055/s-0038-1649502 .

Nissen N, Lemche J, Reestorff CM, Schmidt M, Skjerbæk AG, Skovgaard L, et al. The lived experience of uncertainty in everyday life with MS. Disabil Rehabil. 2022;44:5957–63. https://doi.org/10.1080/09638288.2021.1955302 .

Strupp J, Hartwig A, Golla H, Galushko M, Pfaff H, Voltz R. Feeling severely affected by multiple sclerosis: what does this mean? Palliat Med. 2012;26:1001–10. https://doi.org/10.1177/0269216311425420 .

Strupp J, Voltz R, Golla H. Opening locked doors: Integrating a palliative care approach into the management of patients with severe multiple sclerosis. Mult Scler J. 2016;22:13–8.

Article   CAS   Google Scholar  

Kraft AK, Berger K. Kernaspekte einer bedarfsgerechten Versorgung von Patienten mit Multipler Sklerose : Inanspruchnahme ambulanter Leistungen und „shared decision making“ [Core aspects of a needs-conform care of patients with multiple sclerosis : Utilization of outpatient services and shared decision making]. Nervenarzt. 2020;91:503–10. https://doi.org/10.1007/s00115-020-00906-z .

Doshi A, Chataway J. Multiple sclerosis, a treatable disease. Clin Med (Lond). 2017;17:530–6. https://doi.org/10.7861/clinmedicine.17-6-530 .

Kobelt G, Thompson A, Berg J, Gannedahl M, Eriksson J. New insights into the burden and costs of multiple sclerosis in Europe. Mult Scler. 2017;23:1123–36. https://doi.org/10.1177/1352458517694432 .

Conradsson D, Ytterberg C, Engelkes C, Johansson S, Gottberg K. Activity limitations and participation restrictions in people with multiple sclerosis: a detailed 10-year perspective. Disabil Rehabil. 2021;43:406–13. https://doi.org/10.1080/09638288.2019.1626919 .

Sorensen PS, Giovannoni G, Montalban X, Thalheim C, Zaratin P, Comi G. The Multiple Sclerosis Care Unit. Mult Scler J. 2019;5:627–36.

Article   Google Scholar  

Tan H, Yu J, Tabby D, Devries A, Singer J. Clinical and economic impact of a specialty care management program among patients with multiple sclerosis: a cohort study. Mult Scler. 2010;16:956–63. https://doi.org/10.1177/1352458510373487 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Oeseburg B, Wynia K, Middel B, Reijneveld SA. Effects of case management for frail older people or those with chronic illness: a systematic review. Nurs Res. 2009;58:201–10.

Aiken LS, Butner J, Lockhart CA, Volk-Craft BE, Hamilton G, Williams FG. Outcome evaluation of a randomized trial of the PhoenixCare intervention: program of case management and coordinated care for the seriously chronically ill. J Palliat Med. 2006;9:111–26. https://doi.org/10.1089/jpm.2006.9.111 .

Kuhn U, Düsterdiek A, Galushko M, Dose C, Montag T, Ostgathe C, Voltz R. Identifying patients suitable for palliative care—a descriptive analysis of enquiries using a Case Management Process Model approach. BMC Res Notes. 2012;5:611. https://doi.org/10.1186/1756-0500-5-611 .

Leary A, Quinn D, Bowen A. Impact of proactive case management by multiple sclerosis specialist nurses on use of unscheduled care and emergency presentation in multiple sclerosis: a case study. Int J MS Care. 2015;17:159–63. https://doi.org/10.7224/1537-2073.2014-011 .

Strupp J, Dose C, Kuhn U, Galushko M, Duesterdiek A, Ernstmann N, et al. Analysing the impact of a case management model on the specialised palliative care multi-professional team. Support Care Cancer. 2018;26:673–9. https://doi.org/10.1007/s00520-017-3893-3 .

Wynia K, Annema C, Nissen H, de Keyser J, Middel B. Design of a Randomised Controlled Trial (RCT) on the effectiveness of a Dutch patient advocacy case management intervention among severely disabled Multiple Sclerosis patients. BMC Health Serv Res. 2010;10:142. https://doi.org/10.1186/1472-6963-10-142 .

Ewers M, Schaeffer D, editors. Case Management in Theorie und Praxis. Bern: Huber; 2005.

Google Scholar  

Neuffer M. Case Management: Soziale Arbeit mit Einzelnen und Familien. 5th ed. Weinheim, Basel: Beltz Juventa; 2013.

Case Management Society of America. The standards of practice for case management. 2022.

Deutsche Gesellschaft für Care und Case Management e.V., editor. Case Management Leitlinien: Rahmenempfehlung, Standards und ethische Grundlagen. 2nd ed. Heidelberg: Medhochzwei; 2020.

Monzer M. Case Management Grundlagen. 2nd ed. Heidelberg: Medhochzwei; 2018.

Wissert M. Grundfunktionen und fachliche Standards des Unterstützungsmanagements. Z Gerontol Geriat. 1998;31(5):331–7.

Wissert M. Tools und Werkzeuge beim Case Management: Die Hilfeplanung. Case Manag. 2007;1:35–7.

Schaeffer D, Berens E-M, Vogt D. Health literacy in the German population. Dtsch Arztebl Int. 2017;114:53–60. https://doi.org/10.3238/arztebl.2017.0053 .

Pfaff H, Schulte H. Der onkologische Patient der Zukunft. Onkologe. 2012;18:127–33. https://doi.org/10.1007/s00761-011-2201-y .

Pfaff H, Kowalski C, Ommen O. Modelle zur Analyse von Integration und Koordination im Versorgungssystem. In: Ameldung, Sydow, Windeler, editor. Vernetzung im Gesundheitswesen: Wettbewerb und Kooperation. Stuttgart: Kohlhammer Verlag; 2009. p. 75–90.

Golla H, Mammeas S, Galushko M, Pfaff H, Voltz R. Unmet needs of caregivers of severely affected multiple sclerosis patients: A qualitative study. Palliat Support Care. 2015;13(6):1685–93.

Golla H, Galushko M, Pfaff H, Voltz R. Multiple sclerosis and palliative care - perceptions of severely affected multiple sclerosis patients and their health professionals: a qualitative study. BMC Palliat Care. 2014;13:11. https://doi.org/10.1186/1472-684x-13-11 .

Golla H, Galushko M, Pfaff H, Voltz R. Unmet needs of severely affected multiple sclerosis patients: the health professionals’ view. Palliat Med. 2012;26:139–51. https://doi.org/10.1177/0269216311401465 .

Methley AM, Chew-Graham CA, Cheraghi-Sohi S, Campbell SM. A qualitative study of patient and professional perspectives of healthcare services for multiple sclerosis: implications for service development and policy. Health Soc Care Community. 2017;25:848–57. https://doi.org/10.1111/hsc.12369 .

Kalb R, Costello K, Guiod L. Case management services to meet the complex needs of patients with multiple sclerosis in the community—the successes and challenges of a unique program from the national multiple sclerosis society. US Neurology. 2019;15:27–31.

Krüger K, Fricke LM, Dilger E-M, Thiele A, Schaubert K, Hoekstra D, et al. How is and how should healthcare for people with multiple sclerosis in Germany be designed?-The rationale and protocol for the mixed-methods study Multiple Sclerosis-Patient-Oriented Care in Lower Saxony (MS-PoV). PLoS One. 2021;16:e0259855. https://doi.org/10.1371/journal.pone.0259855 .

Ivancevic S, Weegen L, Korff L, Jahn R, Walendzik A, Mostardt S, et al. Effektivität und Kosteneffektivät von Versorgungsmanagement-Programmen bei Multipler Sklerose in Deutschland – Eine Übersichtsarbeit. Akt Neurol. 2015;42:503–8. https://doi.org/10.1055/s-0035-1564111 .

Müller A, Dillen K, Dojan T, Ungeheuer S, Goereci Y, Dunkl V, et al. Development of a long-term cross-sectoral case and care management manual for patients with severe multiple sclerosis and their caregivers. Prof Case Manag. 2023;28:183–93. https://doi.org/10.1097/NCM.0000000000000608 .

Golla H, Dillen K, Hellmich M, Dojan T, Ungeheuer S, Schmalz P, et al. Communication, Coordination, and Security for People with Multiple Sclerosis (COCOS-MS): a randomised phase II clinical trial protocol. BMJ Open. 2022;12:e049300. https://doi.org/10.1136/bmjopen-2021-049300 .

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19:349–57. https://doi.org/10.1093/intqhc/mzm042 .

Kuckartz U. Qualitative Inhaltsanalyse: Methoden, Praxis, Computerunterstützung. 4th ed. Weinheim: Beltz Juventa; 2018.

Akyirem S, Ekpor E, Aidoo-Frimpong GA, Salifu Y, Nelson LE. Online interviews for qualitative health research in Africa: a scoping review. Int Health. 2023. https://doi.org/10.1093/inthealth/ihad010 .

Peasgood T, Bourke M, Devlin N, Rowen D, Yang Y, Dalziel K. Randomised comparison of online interviews versus face-to-face interviews to value health states. Soc Sci Med. 2023;323:115818. https://doi.org/10.1016/j.socscimed.2023.115818 .

Giordano A, Cimino V, Campanella A, Morone G, Fusco A, Farinotti M, et al. Low quality of life and psychological wellbeing contrast with moderate perceived burden in carers of people with severe multiple sclerosis. J Neurol Sci. 2016;366:139–45. https://doi.org/10.1016/j.jns.2016.05.016 .

Joo JY, Liu MF. Experiences of case management with chronic illnesses: a qualitative systematic review. Int Nurs Rev. 2018;65(1):102–1113.

Freeman J, Gorst T, Gunn H, Robens S. “A non-person to the rest of the world”: experiences of social isolation amongst severely impaired people with multiple sclerosis. Disabil Rehabil. 2020;42:2295–303. https://doi.org/10.1080/09638288.2018.1557267 .

National Institute for Health and Care Excellence. Multiple sclerosis: Management of multiple sclerosis in primary and secondary care. 2014.

Erdmann A, Spoden C, Hirschberg I, Neitzke G. The wish to die and hastening death in amyotrophic lateral sclerosis: A scoping review. BMJ Support Palliat Care. 2021;11:271–87. https://doi.org/10.1136/bmjspcare-2020-002640 .

Erlangsen A, Stenager E, Conwell Y, Andersen PK, Hawton K, Benros ME, et al. Association between neurological disorders and death by suicide in Denmark. JAMA. 2020;323:444–54. https://doi.org/10.1001/jama.2019.21834 .

Kalb R, Feinstein A, Rohrig A, Sankary L, Willis A. Depression and suicidality in multiple sclerosis: red flags, management strategies, and ethical considerations. Curr Neurol Neurosci Rep. 2019;19:77. https://doi.org/10.1007/s11910-019-0992-1 .

Feinstein A, Pavisian B. Multiple sclerosis and suicide. Mult Scler. 2017;23:923–7. https://doi.org/10.1177/1352458517702553 .

Marrie RA, Salter A, Tyry T, Cutter GR, Cofield S, Fox RJ. High hypothetical interest in physician-assisted death in multiple sclerosis. Neurology. 2017;88:1528–34. https://doi.org/10.1212/WNL.0000000000003831 .

Gauthier S, Mausbach J, Reisch T, Bartsch C. Suicide tourism: a pilot study on the Swiss phenomenon. J Med Ethics. 2015;41:611–7. https://doi.org/10.1136/medethics-2014-102091 .

Fischer S, Huber CA, Imhof L, MahrerImhof R, Furter M, Ziegler SJ, Bosshard G. Suicide assisted by two Swiss right-to-die organisations. J Med Ethics. 2008;34:810–4. https://doi.org/10.1136/jme.2007.023887 .

Strupp J, Ehmann C, Galushko M, Bücken R, Perrar KM, Hamacher S, et al. Risk factors for suicidal ideation in patients feeling severely affected by multiple sclerosis. J Palliat Med. 2016;19:523–8. https://doi.org/10.1089/jpm.2015.0418 .

Spence RA, Blanke CD, Keating TJ, Taylor LP. Responding to patient requests for hastened death: physician aid in dying and the clinical oncologist. J Oncol Pract. 2017;13:693–9. https://doi.org/10.1200/JOP.2016.019299 .

Monforte-Royo C, Villavicencio-Chávez C, Tomás-Sábado J, Balaguer A. The wish to hasten death: a review of clinical studies. Psychooncology. 2011;20:795–804. https://doi.org/10.1002/pon.1839 .

Blanke C, LeBlanc M, Hershman D, Ellis L, Meyskens F. Characterizing 18 years of the death with dignity act in Oregon. JAMA Oncol. 2017;3:1403–6. https://doi.org/10.1001/jamaoncol.2017.0243 .

Methley A, Campbell S, Cheraghi-Sohi S, Chew-Graham C. Meeting the mental health needs of people with multiple sclerosis: a qualitative study of patients and professionals. Disabil Rehab. 2017;39(11):1097-105. https://doi.org/10.1080/09638288.2016.1180547 .

Hudon C, Bisson M, Chouinard M-C, Delahunty-Pike A, Lambert M, Howse D, et al. Implementation analysis of a case management intervention for people with complex care needs in primary care: a multiple case study across Canada. BMC Health Serv Res. 2023;23:377. https://doi.org/10.1186/s12913-023-09379-7 .

Beckmann M, Dittmer K, Jaschke J, Karbach U, Köberlein-Neu J, Nocon M, et al. Electronic patient record and its effects on social aspects of interprofessional collaboration and clinical workflows in hospitals (eCoCo): a mixed methods study protocol. BMC Health Serv Res. 2021;21:377. https://doi.org/10.1186/s12913-021-06377-5 .

Campanella P, Lovato E, Marone C, Fallacara L, Mancuso A, Ricciardi W, Specchia ML. The impact of electronic health records on healthcare quality: a systematic review and meta-analysis. Eur J Public Health. 2016;26:60–4. https://doi.org/10.1093/eurpub/ckv122 .

García-Hernández M, González de León B, Barreto-Cruz S, Vázquez-Díaz JR. Multicomponent, high-intensity, and patient-centered care intervention for complex patients in transitional care: SPICA program. Front Med (Lausanne). 2022;9:1033689. https://doi.org/10.3389/fmed.2022.1033689 .

Meisinger C, Stollenwerk B, Kirchberger I, Seidl H, Wende R, Kuch B, Holle R. Effects of a nurse-based case management compared to usual care among aged patients with myocardial infarction: results from the randomized controlled KORINNA study. BMC Geriatr. 2013. https://doi.org/10.1186/1471-2318-13-115 .

Joo JY, Huber DL. Case management effectiveness on health care utilization outcomes: a systematic review of reviews. West J Nurs Res. 2019;41:111–33. https://doi.org/10.1177/0193945918762135 .

Stergiopoulos V, Gozdzik A, Misir V, Skosireva A, Connelly J, Sarang A, et al. Effectiveness of housing first with intensive case management in an ethnically diverse sample of homeless adults with mental illness: a randomized controlled trial. PLoS One. 2015;10:e0130281. https://doi.org/10.1371/journal.pone.0130281 .

Löcherbach P, Wendt R, editors. Care und Case Management: Transprofessionelle Versorgungsstrukturen und Netzwerke. 1st ed. Stuttgart: Kohlhammer; 2020.

EAPC2023 Abstract Book. Palliat Med. 2023;37:1–302. https://doi.org/10.1177/02692163231172891 .

Download references

Acknowledgements

We would like to thank all the patients, caregivers and health care specialists who volunteered their time to participate in an interview and the trial, Carola Janßen for transcribing the interviews, Fiona Brown for translating the illustrative quotes and Beatrix Münzberg, Kerstin Weiß and Monika Höveler for data collection in the quantitative study part.

COCOS-MS Trial Group

Anne Müller 1 , Fabian Hebben 1 , Kim Dillen 1 , Veronika Dunkl 1 , Yasemin Goereci 2 , Raymond Voltz 1,3,4 , Peter Löcherbach 5 , Clemens Warnke 2 , Heidrun Golla 1 , Dirk Müller 6 , Dorthe Hobus 1 , Eckhard Bonmann 7 , Franziska Schwartzkopff 8 , Gereon Nelles 9 , Gundula Palmbach 8 , Herbert Temmes 10 , Isabel Franke 1 , Judith Haas 10 , Julia Strupp 1 , Kathrin Gerbershagen 7 , Laura Becker-Peters 8 , Lothar Burghaus 11 , Martin Hellmich 12 , Martin Paus 8 , Solveig Ungeheuer 1 , Sophia Kochs 1 , Stephanie Stock 6 , Thomas Joist 13 , Volker Limmroth 14

1 Department of Palliative Medicine, Faculty of Medicine and University Hospital, University of Cologne, Cologne, Germany

2 Department of Neurology, Faculty of Medicine and University Hospital, University of Cologne, Cologne, Germany

3 Center for Integrated Oncology Aachen Bonn Cologne Düsseldorf (CIO ABCD), University of Cologne, Cologne, Germany

4 Center for Health Services Research (ZVFK), University of Cologne, Cologne, Germany

5 German Society of Care and Case Management e.V. (DGCC), Münster, Germany

6 Institute for Health Economics and Clinical Epidemiology (IGKE), Faculty of Medicine and University Hospital, University of Cologne, Cologne, Germany

7 Department of Neurology, Klinikum Köln, Cologne, Germany

8 Clinical Trials Centre Cologne (CTCC), Faculty of Medicine and University Hospital, University of Cologne, Cologne, Germany

9 NeuroMed Campus, MedCampus Hohenlind, Cologne, Germany

10 German Multiple Sclerosis Society Federal Association (DMSG), Hannover, Germany

11 Department of Neurology, Heilig Geist-Krankenhaus Köln, Cologne, Germany

12 Institute of Medical Statistics and Computational Biology (IMSB), Faculty of Medicine and University Hospital, University of Cologne, Cologne, Germany

13 Academic Teaching Practice, University of Cologne, Cologne, Germany

14 Department of Neurology, Klinikum Köln-Merheim, Cologne, Germany

Open Access funding enabled and organized by Projekt DEAL. This work was supported by the Innovation Funds of the Federal Joint Committee (G-BA), grant number: 01VSF19029.

Author information

Authors and affiliations.

Department of Palliative Medicine, Faculty of Medicine and University Hospital, University of Cologne, Cologne, Germany

Anne Müller, Fabian Hebben, Kim Dillen, Veronika Dunkl, Raymond Voltz & Heidrun Golla

Department of Neurology, Faculty of Medicine and University Hospital, University of Cologne, Cologne, Germany

Yasemin Goereci & Clemens Warnke

Center for Integrated Oncology Aachen Bonn Cologne Düsseldorf (CIO ABCD), University of Cologne, Cologne, Germany

Raymond Voltz

Center for Health Services Research, University of Cologne, Cologne, Germany

German Society of Care and Case Management E.V. (DGCC), Münster, Germany

Peter Löcherbach

You can also search for this author in PubMed   Google Scholar

• Anne Müller
• , Fabian Hebben
• , Kim Dillen
• , Veronika Dunkl
• , Yasemin Goereci
• , Raymond Voltz
• , Peter Löcherbach
• , Clemens Warnke
• , Heidrun Golla
• , Dirk Müller
• , Dorthe Hobus
• , Eckhard Bonmann
• , Franziska Schwartzkopff
• , Gereon Nelles
• , Gundula Palmbach
• , Herbert Temmes
• , Isabel Franke
• , Judith Haas
• , Julia Strupp
• , Kathrin Gerbershagen
• , Laura Becker-Peters
• , Lothar Burghaus
• , Martin Hellmich
• , Martin Paus
• , Solveig Ungeheuer
• , Sophia Kochs
• , Stephanie Stock
• , Thomas Joist
•  & Volker Limmroth

Contributions

HG, KD, CW designed the trial. HG, KD obtained ethical approvals. HG, KD developed the interview guidelines with help of the CCM (SU). AM was responsible for collecting qualitative data, developing the code system, coding, analysis of the data and writing the first draft of the manuscript, thoroughly revised and partly rewritten by HG. FH supported in collecting qualitative data, coding and analysis of the interviews. KD supported in collecting qualitative data. AM, FH, KD, VD, YG, RV, PL, CW, HG discussed and con-solidated the finalized category system. AM, FH, KD, VD, YG, RV, PL, CW, HG read and commented on the manuscript and agreed to the final version.

Authors’ information

Not applicable.

Corresponding author

Correspondence to Anne Müller .

Ethics declarations

Ethics approval and consent to participate.

Participants were provided with oral and written information about the trial and provided written informed consent. Ethical approval was obtained from the Ethics Committee of the University of Cologne (#20–1436). The trial is registered in the German Register for Clinical Studies (DRKS) (DRKS00022771) and is conducted under the Declaration of Helsinki.

Consent for publication

Competing interests.

Clemens Warnke has received institutional support from Novartis, Alexion, Sanofi Genzyme, Janssen, Biogen, Merck and Roche. The other authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Müller, A., Hebben, F., Dillen, K. et al. “So at least now I know how to deal with things myself, what I can do if it gets really bad again”—experiences with a long-term cross-sectoral advocacy care and case management for severe multiple sclerosis: a qualitative study. BMC Health Serv Res 24 , 453 (2024). https://doi.org/10.1186/s12913-024-10851-1

Download citation

Received : 23 November 2023

Accepted : 11 March 2024

Published : 10 April 2024

DOI : https://doi.org/10.1186/s12913-024-10851-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Cross-sectoral
• Qualitative research
• Health care specialist
• Severe multiple sclerosis

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Perspect Med Educ
• v.8(4); 2019 Aug

Limited by our limitations

Paula t. ross.

Medical School, University of Michigan, Ann Arbor, MI USA

Nikki L. Bibler Zaidi

Study limitations represent weaknesses within a research design that may influence outcomes and conclusions of the research. Researchers have an obligation to the academic community to present complete and honest limitations of a presented study. Too often, authors use generic descriptions to describe study limitations. Including redundant or irrelevant limitations is an ineffective use of the already limited word count. A meaningful presentation of study limitations should describe the potential limitation, explain the implication of the limitation, provide possible alternative approaches, and describe steps taken to mitigate the limitation. This includes placing research findings within their proper context to ensure readers do not overemphasize or minimize findings. A more complete presentation will enrich the readers’ understanding of the study’s limitations and support future investigation.

Introduction

Regardless of the format scholarship assumes, from qualitative research to clinical trials, all studies have limitations. Limitations represent weaknesses within the study that may influence outcomes and conclusions of the research. The goal of presenting limitations is to provide meaningful information to the reader; however, too often, limitations in medical education articles are overlooked or reduced to simplistic and minimally relevant themes (e.g., single institution study, use of self-reported data, or small sample size) [ 1 ]. This issue is prominent in other fields of inquiry in medicine as well. For example, despite the clinical implications, medical studies often fail to discuss how limitations could have affected the study findings and interpretations [ 2 ]. Further, observational research often fails to remind readers of the fundamental limitation inherent in the study design, which is the inability to attribute causation [ 3 ]. By reporting generic limitations or omitting them altogether, researchers miss opportunities to fully communicate the relevance of their work, illustrate how their work advances a larger field under study, and suggest potential areas for further investigation.

Goals of presenting limitations

Medical education scholarship should provide empirical evidence that deepens our knowledge and understanding of education [ 4 , 5 ], informs educational practice and process, [ 6 , 7 ] and serves as a forum for educating other researchers [ 8 ]. Providing study limitations is indeed an important part of this scholarly process. Without them, research consumers are pressed to fully grasp the potential exclusion areas or other biases that may affect the results and conclusions provided [ 9 ]. Study limitations should leave the reader thinking about opportunities to engage in prospective improvements [ 9 – 11 ] by presenting gaps in the current research and extant literature, thereby cultivating other researchers’ curiosity and interest in expanding the line of scholarly inquiry [ 9 ].

Presenting study limitations is also an ethical element of scientific inquiry [ 12 ]. It ensures transparency of both the research and the researchers [ 10 , 13 , 14 ], as well as provides transferability [ 15 ] and reproducibility of methods. Presenting limitations also supports proper interpretation and validity of the findings [ 16 ]. A study’s limitations should place research findings within their proper context to ensure readers are fully able to discern the credibility of a study’s conclusion, and can generalize findings appropriately [ 16 ].

Why some authors may fail to present limitations

As Price and Murnan [ 8 ] note, there may be overriding reasons why researchers do not sufficiently report the limitations of their study. For example, authors may not fully understand the importance and implications of their study’s limitations or assume that not discussing them may increase the likelihood of publication. Word limits imposed by journals may also prevent authors from providing thorough descriptions of their study’s limitations [ 17 ]. Still another possible reason for excluding limitations is a diffusion of responsibility in which some authors may incorrectly assume that the journal editor is responsible for identifying limitations. Regardless of reason or intent, researchers have an obligation to the academic community to present complete and honest study limitations.

A guide to presenting limitations

The presentation of limitations should describe the potential limitations, explain the implication of the limitations, provide possible alternative approaches, and describe steps taken to mitigate the limitations. Too often, authors only list the potential limitations, without including these other important elements.

Describe the limitations

When describing limitations authors should identify the limitation type to clearly introduce the limitation and specify the origin of the limitation. This helps to ensure readers are able to interpret and generalize findings appropriately. Here we outline various limitation types that can occur at different stages of the research process.

Study design

Some study limitations originate from conscious choices made by the researcher (also known as delimitations) to narrow the scope of the study [ 1 , 8 , 18 ]. For example, the researcher may have designed the study for a particular age group, sex, race, ethnicity, geographically defined region, or some other attribute that would limit to whom the findings can be generalized. Such delimitations involve conscious exclusionary and inclusionary decisions made during the development of the study plan, which may represent a systematic bias intentionally introduced into the study design or instrument by the researcher [ 8 ]. The clear description and delineation of delimitations and limitations will assist editors and reviewers in understanding any methodological issues.

Data collection

Study limitations can also be introduced during data collection. An unintentional consequence of human subjects research is the potential of the researcher to influence how participants respond to their questions. Even when appropriate methods for sampling have been employed, some studies remain limited by the use of data collected only from participants who decided to enrol in the study (self-selection bias) [ 11 , 19 ]. In some cases, participants may provide biased input by responding to questions they believe are favourable to the researcher rather than their authentic response (social desirability bias) [ 20 – 22 ]. Participants may influence the data collected by changing their behaviour when they are knowingly being observed (Hawthorne effect) [ 23 ]. Researchers—in their role as an observer—may also bias the data they collect by allowing a first impression of the participant to be influenced by a single characteristic or impression of another characteristic either unfavourably (horns effect) or favourably (halo effort) [ 24 ].

Data analysis

Study limitations may arise as a consequence of the type of statistical analysis performed. Some studies may not follow the basic tenets of inferential statistical analyses when they use convenience sampling (i.e. non-probability sampling) rather than employing probability sampling from a target population [ 19 ]. Another limitation that can arise during statistical analyses occurs when studies employ unplanned post-hoc data analyses that were not specified before the initial analysis [ 25 ]. Unplanned post-hoc analysis may lead to statistical relationships that suggest associations but are no more than coincidental findings [ 23 ]. Therefore, when unplanned post-hoc analyses are conducted, this should be clearly stated to allow the reader to make proper interpretation and conclusions—especially when only a subset of the original sample is investigated [ 23 ].

Study results

The limitations of any research study will be rooted in the validity of its results—specifically threats to internal or external validity [ 8 ]. Internal validity refers to reliability or accuracy of the study results [ 26 ], while external validity pertains to the generalizability of results from the study’s sample to the larger, target population [ 8 ].

Examples of threats to internal validity include: effects of events external to the study (history), changes in participants due to time instead of the studied effect (maturation), systematic reduction in participants related to a feature of the study (attrition), changes in participant responses due to repeatedly measuring participants (testing effect), modifications to the instrument (instrumentality) and selecting participants based on extreme scores that will regress towards the mean in repeat tests (regression to the mean) [ 27 ].

Threats to external validity include factors that might inhibit generalizability of results from the study’s sample to the larger, target population [ 8 , 27 ]. External validity is challenged when results from a study cannot be generalized to its larger population or to similar populations in terms of the context, setting, participants and time [ 18 ]. Therefore, limitations should be made transparent in the results to inform research consumers of any known or potentially hidden biases that may have affected the study and prevent generalization beyond the study parameters.

Explain the implication(s) of each limitation

Authors should include the potential impact of the limitations (e.g., likelihood, magnitude) [ 13 ] as well as address specific validity implications of the results and subsequent conclusions [ 16 , 28 ]. For example, self-reported data may lead to inaccuracies (e.g. due to social desirability bias) which threatens internal validity [ 19 ]. Even a researcher’s inappropriate attribution to a characteristic or outcome (e.g., stereotyping) can overemphasize (either positively or negatively) unrelated characteristics or outcomes (halo or horns effect) and impact the internal validity [ 24 ]. Participants’ awareness that they are part of a research study can also influence outcomes (Hawthorne effect) and limit external validity of findings [ 23 ]. External validity may also be threatened should the respondents’ propensity for participation be correlated with the substantive topic of study, as data will be biased and not represent the population of interest (self-selection bias) [ 29 ]. Having this explanation helps readers interpret the results and generalize the applicability of the results for their own setting.

Provide potential alternative approaches and explanations

Often, researchers use other studies’ limitations as the first step in formulating new research questions and shaping the next phase of research. Therefore, it is important for readers to understand why potential alternative approaches (e.g. approaches taken by others exploring similar topics) were not taken. In addition to alternative approaches, authors can also present alternative explanations for their own study’s findings [ 13 ]. This information is valuable coming from the researcher because of the direct, relevant experience and insight gained as they conducted the study. The presentation of alternative approaches represents a major contribution to the scholarly community.

Describe steps taken to minimize each limitation

No research design is perfect and free from explicit and implicit biases; however various methods can be employed to minimize the impact of study limitations. Some suggested steps to mitigate or minimize the limitations mentioned above include using neutral questions, randomized response technique, force choice items, or self-administered questionnaires to reduce respondents’ discomfort when answering sensitive questions (social desirability bias) [ 21 ]; using unobtrusive data collection measures (e.g., use of secondary data) that do not require the researcher to be present (Hawthorne effect) [ 11 , 30 ]; using standardized rubrics and objective assessment forms with clearly defined scoring instructions to minimize researcher bias, or making rater adjustments to assessment scores to account for rater tendencies (halo or horns effect) [ 24 ]; or using existing data or control groups (self-selection bias) [ 11 , 30 ]. When appropriate, researchers should provide sufficient evidence that demonstrates the steps taken to mitigate limitations as part of their study design [ 13 ].

In conclusion, authors may be limiting the impact of their research by neglecting or providing abbreviated and generic limitations. We present several examples of limitations to consider; however, this should not be considered an exhaustive list nor should these examples be added to the growing list of generic and overused limitations. Instead, careful thought should go into presenting limitations after research has concluded and the major findings have been described. Limitations help focus the reader on key findings, therefore it is important to only address the most salient limitations of the study [ 17 , 28 ] related to the specific research problem, not general limitations of most studies [ 1 ]. It is important not to minimize the limitations of study design or results. Rather, results, including their limitations, must help readers draw connections between current research and the extant literature.

The quality and rigor of our research is largely defined by our limitations [ 31 ]. In fact, one of the top reasons reviewers report recommending acceptance of medical education research manuscripts involves limitations—specifically how the study’s interpretation accounts for its limitations [ 32 ]. Therefore, it is not only best for authors to acknowledge their study’s limitations rather than to have them identified by an editor or reviewer, but proper framing and presentation of limitations can actually increase the likelihood of acceptance. Perhaps, these issues could be ameliorated if academic and research organizations adopted policies and/or expectations to guide authors in proper description of limitations.

Let us take a walk to the sustainable tourism practices: a qualitative study through the lens of tourism experts

• Research Article
• Published: 04 January 2024
• Volume 31 , pages 12892–12915, ( 2024 )

Cite this article

• Vikas Arya 1 ,
• Vilte Auruskeviciene   ORCID: orcid.org/0000-0002-1563-4052 2 ,
• Srishti Agarwal 3 ,
• Priyanka Kokatnur 3 ,
• Harish Kumar 4 &
• Rajeev Verma 5  

360 Accesses

Explore all metrics

The rising opportunities of sustainable tourism have brought many policies to control the exploitation of the environment and increase the reach of luxurious, safe, and authentic experiences to the different segments of tourists. This study seeks to prioritize the variables influencing the development of sustainable tourism and pinpoint key success factors that align with the Sustainable Development Goals (SDGs). It adopts a tri-dimensional framework encompassing economic, social, and environmental aspects, further delineated into eleven sub-dimensions, to provide a quantitative evaluation of sustainable tourism. We conducted interviews with 26 tourism industry experts hailing from eight countries, analyzing their responses using interval type-2 fuzzy sets. The results underscore the critical role of specific components in advancing sustainable tourism. In the economic dimension, “financial resources and tourism costs” emerge as vital factors. In the social dimension, “health and safety” takes center stage, while “green infrastructure” plays a pivotal role in the environmental dimension. These findings underscore the significance of these aspects in promoting sustainable tourism. Furthermore, this study explores the strategic importance of sustainable tourism equity in shaping tourism planning and development for emerging markets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Source: Authors

Similar content being viewed by others

Three pillars of sustainability: in search of conceptual origins

Ben Purvis, Yong Mao & Darren Robinson

Impact of tourism development upon environmental sustainability: a suggested framework for sustainable ecotourism

Qadar Bakhsh Baloch, Syed Naseeb Shah, … Asia Umar Khan

Ecotourism and sustainable development: a scientometric review of global research trends

Lishan Xu, Changlin Ao, … Zhenyu Cai

Data availability

Data will be available on request.

Agarwal SA, Kasliwal N (2019) Indian consumer’s environmental consciousness and decision towards green services of hospitality industry. J Hosp Appl Res 14(2):22–40

Google Scholar  

Ahmad N, Youjin L, Hdia M (2022) The role of innovation and tourism in sustainability: why is environment-friendly tourism necessary for entrepreneurship? J Clean Prod 379:134799. https://doi.org/10.1016/j.jclepro.2022.134799

Article   Google Scholar  

Alfaro Navarro JL, Andrés Martínez ME, Mondéjar Jiménez JA (2020) An approach to measuring sustainable tourism at the local level in Europe. Curr Issue Tour 23(4):423–437. https://doi.org/10.1080/13683500.2019.1579174

Andereck KL, Valentine KM, Knopf RC, Vogt CA, Knollenberg W (2021) Tourists’ willingness to conserve resources and reduce their carbon footprint in national parks. J Sustain Tour 29(4):499–516

Andria J, di Tollo G, Pesenti R (2019) A fuzzy evaluation of tourism sustainability. In Business and consumer analytics: New ideas pp 911–932. Springer, Cham. https://doi.org/10.1007/978-3-030-06222-4_24

Apak ÖC, Gürbüz A (2023) The effect of local food consumption of domestic tourists on sustainable tourism. J Retail Consum Serv 71:103192. https://doi.org/10.1016/j.jretconser.2022.103192

Arya V, Sethi D, Paul J (2019) Does digital footprint act as a digital asset? — enhancing brand experience through remarketing. Int J Inf Manage 49:142–156. https://doi.org/10.1016/j.ijinfomgt.2019.03.013

Arya V, Sharma S, Sethi D, Verma H, Shiva A (2018) Ties that bind tourists: embedding destination motivators to destination attachment: a study in the context of Kumbh Fair India. Asia Pacific J Tour Res 23(12):1160–1172. https://doi.org/10.1080/10941665.2018.1528992

Arya V, Sambyal R, Sharma A, Dwivedi YK (2023) Brands are calling your AVATAR in Metaverse—a study to explore XR‐based gamification marketing activities and consumer‐based brand equity in virtual world. Journal of Consumer Behaviour. https://doi.org/10.1002/cb.2214

Athari SA, Alola UV, Ghasemi M, Alola AA (2021) The (Un) sticky role of exchange and inflation rate in tourism development: insight from the low and high political risk destinations. Curr Issue Tour 24(12):1670–1685. https://doi.org/10.1080/13683500.2020.1798893

Balsalobre-Lorente D, Abbas J, He C, Pilař L, Shah SAR (2023a) Tourism, urbanization and natural resources rents matter for environmental sustainability: the leading role of AI and ICT on sustainable development goals in the digital era. Resour Policy 82:103445. https://doi.org/10.1016/j.resourpol.2023.103445

Balsalobre-Lorente D, Luzon LI, Usman M, Jahanger A (2023) The relevance of international tourism and natural resource rents in economic growth: fresh evidence from MINT countries in the digital era. Environ Sci Pollut Res 1–18. https://doi.org/10.1007/s11356-022-25022-0

Barbieri C, Sotomayor S, Gil Arroyo C (2020) Sustainable tourism practices in indigenous communities: the case of the Peruvian Andes. Tour Plann Dev 17(2):207–224. https://doi.org/10.1080/21568316.2019.1597760

Beames S, Mackenzie SH, Raymond E (2022) How can we adventure sustainably? A systematized review of sustainability guidance for adventure tourism operators. J Hosp Tour Manag 50:223–231. https://doi.org/10.1016/j.jhtm.2022.01.002

Becken S, Loehr J (2022) Tourism governance and enabling drivers for intensifying climate action. Journal of Sustainable Tourism 1–19. https://doi.org/10.1080/09669582.2022.2032099

Becken S, Stantic B, Chen JS (2020) Climate change and tourism: impacts and challenges. Tourism Review 75(1):46–60

Berno T, Rajalingam G, Miranda AI, Ximenes J (2022) Promoting sustainable tourism futures in Timor-Leste by creating synergies between food, place and people. J Sustain Tour 30(2–3):500–514. https://doi.org/10.1080/09669582.2021.1895819

Bodhanwala S, Bodhanwala R (2022) Exploring relationship between sustainability and firm performance in travel and tourism industry: a global evidence. Social Respons J 18(7):1251–1269. https://doi.org/10.1108/SRJ-09-2020-0360

Bramwell B, Lane B (2019) Critical research on the governance and management of tourism in protected areas. Curr Issue Tour 22(5):547–579

Briassoulis H (2002) Sustainable tourism and the question of the commons. Ann Tour Res 29(4):1065–1085. https://doi.org/10.1016/S0160-7383(02)00021-X

Bricker KS, Schultz J (2011) Sustainable tourism in the USA: a comparative look at the Global Sustainable Tourism Criteria. Tour Recreat Res 36(3):215–229

Bramwell B, Lane B (2011) Critical research on the governance of tourism and sustainability. J Sustain Tour 19(4–5):411–421. https://doi.org/10.1080/02508281.2011.11081668

Budeanu A, Miller G, Moscardo G, Ooi CS (2016) Sustainable tourism, progress, challenges and opportunities: an introduction. J Clean Prod 111:285–294. https://doi.org/10.1016/j.jclepro.2015.10.027

Buckley R (2012) Sustainable tourism: research and reality. Ann Tour Res 39(2):528–546. https://doi.org/10.1016/j.annals.2012.02.003

Article   MathSciNet   Google Scholar  

Carlson J, Rahman MM, Rosenberger PJ III, Holzmüller HH (2016) Understanding communal and individual customer experiences in group-oriented event tourism: an activity theory perspective. J Mark Manag 32(9–10):900–925. https://doi.org/10.1080/0267257X.2016.1181099

Chandra P, Kumar J (2021) Strategies for developing sustainable tourism business in the Indian Himalayan Region: insights from Uttarakhand, the Northern Himalayan State of India. J Destin Mark Manag 19:100546. https://doi.org/10.1016/j.jdmm.2020.100546

Choe J, O’Regan M, Kimbu A, Lund NF, Ladkin A (2021) Quality of life perspectives for different social groups in a World Centre of Tourism and Leisure. Tour Stud 21(4):615–637. https://doi.org/10.1177/14687976211038758

Choi HSC, Sirakaya E (2005) Measuring residents’ attitude toward sustainable tourism: development of sustainable tourism attitude scale. J Travel Res 43(4):380–394. https://doi.org/10.1177/0047287505274651

Chong K, Balasingam AS (2019) Tourism sustainability: economic benefits and strategies for preservation and conservation of heritage sites in Southeast Asia. Tourism Review 74(2):268–279. https://doi.org/10.1108/TR-11-2017-0182

Darcy S, Cameron B, Pegg S (2010) Accessible tourism and sustainability: a discussion and case study. J Sustain Tour 18(4):515–537. https://doi.org/10.1080/09669581003690668

Da Silva J, Fernandes V, Limont M, Rauen WB (2020) Sustainable development assessment from a capitals perspective: analytical structure and indicator selection criteria. J Environ Manage 260:110147. https://doi.org/10.1016/j.jenvman.2020.110147

Article   PubMed   Google Scholar  

Sausmarez De (2007) Crisis management, tourism and sustainability: the role of indicators. J Sustain Tour 15(6):700–714

Dolnicar S, Crouch GI, Long P (2008) Environment-friendly tourists: what do we really know about them? J Sustain Tour 16(2):197–210. https://doi.org/10.2167/jost653.0

Dong XD, Nguyen TQT (2022) Power, community involvement, and sustainability of tourism destinations. Tour Stud 23(1):62–79. https://doi.org/10.1177/14687976221144335

Dredge D, Scott N, Lechner AM, Dominey-Howes D (2020) Spatial and coastal tourism. In Tourism and Water (pp. 109–126). Channel View Publications

Durband R (2021) Establishing sustainability standards in tourism. Handbook for sustainable tourism practitioners: the essential toolbox, 233

El Atiek S, Goutte S (2023) Impacts, sustainability, and resilience on the Egyptian tourism and hospitality industry after the Russian airplane crash in 2015. Res Int Bus Financ 64:101866. https://doi.org/10.1016/j.ribaf.2022.101866

El-Masry EA, El-Sayed MK, Awad MA, El-Sammak AA, Sabarouti MAE (2022) Vulnerability of tourism to climate change on the Mediterranean coastal area of El Hammam–EL Alamein Egypt. Environ Dev Sustain 24(1):1145–1165. https://doi.org/10.1007/s10668-021-01488-9

Eslami S, Khalifa Z, Mardani A, Streimikiene D, Han H (2019) Community attachment, tourism impacts, quality of life and residents’ support for sustainable tourism development. J Travel Tour Mark 36(9):1061–1079. https://doi.org/10.1080/10548408.2019.1689224

Falatoonitoosi E, Schaffer V, Kerr D (2022) Does sustainable tourism development enhance destination prosperity? J Hosp Tourism Res 46(5):1056–1082. https://doi.org/10.1177/1096348020988328

Farrell B, Twining-Ward L (2005) Seven steps towards sustainability: tourism in the context of new knowledge. J Sustain Tour 13(2):109–122. https://doi.org/10.1080/09669580508668481

Firman A, Moslehpour M, Qiu R, Lin PK, Ismail T, Rahman FF (2023) The impact of eco-innovation, ecotourism policy and social media on sustainable tourism development: evidence from the tourism sector of Indonesia. Econ Res - Ekonomska Istraživanja 36(2):2143847. https://doi.org/10.1080/1331677X.2022.2143847

Fong SF, Lo MC, Songan P et al (2017) Self-efficacy and sustainable rural tourism development: Local communities’ perspectives from Kuching, Sarawak. Asia Pacific J Tour Res 22(2):147–159. https://doi.org/10.1080/10941665.2016.1208668

Font X, Torres-Delgado A, Crabolu G, Palomo Martinez J, Kantenbacher J, Miller G (2023) The impact of sustainable tourism indicators on destination competitiveness: The European Tourism Indicator System. J Sustain Tour 31(7):1608–1630. https://doi.org/10.1080/09669582.2021.1910281

Garau G, Carboni D, Karim El Meligi A (2022) Economic and environmental impact of the tourism carrying capacity: a local-based approach. J Hosp Tour Res 46(7):1257–1273. https://doi.org/10.1177/10963480211031426

Geoffrey Deladem T, Xiao Z, Siueia TT, Doku S, Tettey I (2021) Developing sustainable tourism through public-private partnership to alleviate poverty in Ghana. Tour Stud 21(2):317–343. https://doi.org/10.1177/1468797620955250

Ghoochani O, Ghanian M, Khosravipour B, Crotts JC (2020) Sustainable tourism development performance in the wetland areas: a proposed composite index. Tourism Review. https://doi.org/10.1108/TR-02-2019-0061

Garrig’os-Sim’on FJ, Gald’on-Salvador JL and Gil-Pechu’an I (2015) The economic sustainability of tourism growth through leakage calculation. Tourism Econ 21(4):721–739. https://doi.org/10.5367/te.2014.0372

Ghahramani L, Khalilzadeh J, Kc B (2018) Tour guides’ communication ecosystems: an inferential social network analysis approach. Inform Technol Tour 20(1):103–130. https://doi.org/10.1007/s40558-018-0114-y

Go H, Kang M (2023) Metaverse tourism for sustainable tourism development: Tourism agenda 2030. Tourism Review 78(2):381–394. https://doi.org/10.1108/TR-02-2022-0102

Gössling S (2021) Tourism, technology and ICT: a critical review of affordances and concessions. J Sustain Tour 29(5):733–750. https://doi.org/10.1080/09669582.2021.1873353

Gössling S, Scott D, Hall CM (2020) Tourism and water: interactions, impacts, and challenges. Annu Rev Environ Resour 45:269–294

Gossling S, Scott D, Hall CM (2018) Tourism and Water (Vol. 13). Channel View Publications

Graci S, Maher PT, Peterson B, Hardy A, Vaugeois N (2021) Thoughts from the think tank: lessons learned from the sustainable Indigenous tourism symposium. J Ecotour 20(2):189–197. https://doi.org/10.1080/14724049.2019.1583754

Grilli G, Tyllianakis E, Luisetti T, Ferrini S, Turner RK (2021) Prospective tourist preferences for sustainable tourism development in Small Island Developing States. Tour Manage 82:104178. https://doi.org/10.1016/j.tourman.2020.104178

GSTC (2021) Who We Are. Global Sustainable Tourism Council. Accessed during October 2023, https://www.gstcouncil.org/who-we-are/

GSTC (2022) The GSTC criteria: the global baseline standards for sustainability in travel and tourism. Global Sustainable Tourism Council. Accessed during October 2023, https://www.gstcouncil.org/the-gstc-criteria/

GSTC (2023) Certification. Global Sustainable Tourism Council. Accessed during October 2023, https://www.gstcouncil.org/certification/

Hall CM (2019) Constructing sustainable tourism development: the 2030 agenda and the managerial ecology of sustainable tourism. J Sustain Tour 27(7):1044–1060. https://doi.org/10.1080/09669582.2018.1560456

Hall CM, Gossling S, Scott D (Eds.) (2015) The Routledge handbook of tourism and sustainability. Routledge

Hardy A, Beeton RJ, Pearson L (2002) Sustainable tourism: an overview of the concept and its position in relation to conceptualisations of tourism. J Sustain Tour 10(6):475–496. https://doi.org/10.1080/09669580208667183

Hamid MA, Isa SM, Kiumarsi S (2021) Sustainable tourism practices and business performance from the tour operators’ perspectives. Anatolia 32(1):23–32. https://doi.org/10.1080/13032917.2020.1830135

Han H (2021) Consumer behavior and environmental sustainability in tourism and hospitality: a review of theories, concepts, and latest research. J Sustain Tour 29(7):1021–1042. https://doi.org/10.1080/09669582.2021.1903019

He LY, Li H, Bi JW, Yang JJ, Zhou Q (2022) The impact of public health emergencies on hotel demand—estimation from a new foresight perspective on the COVID-19. Annals of Tourism Research, 94, May 2022. https://doi.org/10.1016/j.annals.2022.103402

Hamid MA, Isa SM, Kiumarsi S (2020) Sustainable tourism practices and business performance from the tour operators’ perspectives. Anatolia 1–10. https://doi.org/10.1080/13032917.2020.1830135

Higgins-Desbiolles F, Carnicelli S, Krolikowski C, Wijesinghe G, Boluk K (2019) Degrowing tourism: rethinking tourism. J Sustain Tour 27(12):1926–1944. https://doi.org/10.1080/09669582.2019.1601732

Higgins-Desbiolles F (2018) Sustainable tourism: sustaining tourism or something more? Tour Manage Perspect 25:157–160. https://doi.org/10.1016/j.tmp.2017.11.017

Higuchi Y, Yamanaka Y (2019) The potential value of research-based evidence in destination management: the case of Kamikawa. Japan Tour Rev 74(2):166–178. https://doi.org/10.1108/TR-11-2017-0188

Iorio M, Corsale A (2013) Diaspora and tourism: Transylvanian Saxons visiting the homeland. Tour Geogr 15(2):198–232. https://doi.org/10.1080/14616688.2012.647327

Iqbal A, Ramachandran S, Siow ML, Subramaniam T, Afandi SHM (2022) Meaningful community participation for effective development of sustainable tourism: bibliometric analysis towards a quintuple helix model. J Outdoor Recreat Tour 39:100523. https://doi.org/10.1016/j.jort.2022.100523

Ivanov S, Gavrilina M, Webster C, Ralko V (2017) Impacts of political instability on the tourism industry in Ukraine. J Policy Res Tourism Leisure Events 9(1):100–127. https://doi.org/10.1080/19407963.2016.1209677

Ivars-Baidal JA, Vera-Rebollo JF, Perles-Ribes J, Femenia-Serra F, Celdrán-Bernabeu MA (2023) Sustainable tourism indicators: what’s new within the smart city/destination approach? J Sustain Tour 31(7):1556–1582

Iversen R (2022). Cultural exchange in sustainable tourism: a literature review. Current Issues in Tourism, 1–16

Jabbour CJC, Fiorini PDC, Wong CW, Jugend D, Jabbour ABLDS, Seles BMRP, ... et al (2020) First-mover firms in the transition towards the sharing economy in metallic natural resource-intensive industries: implications for the circular economy and emerging industry 4.0 technologies. Resources Policy 66:101596. https://doi.org/10.1016/j.resourpol.2020.101596

Kahraman C (Ed.) (2008) Fuzzy multi-criteria decision making: theory and applications with recent developments (Vol. 16). Springer Science and Business Media

Kahraman C, Öztayşi B, Sarı İU, Turanoğlu E (2014) Fuzzy analytic hierarchy process with interval type-2 fuzzy sets. knowledge-Based Systems 59:48–57. https://doi.org/10.1016/j.knosys.2014.02.001

Karnik NN, Mendel JM, Liang Q (1999) Type-2 fuzzy logic systems. IEEE Trans Fuzzy Syst 7(6):643–658. https://doi.org/10.1109/91.811231

Kaefer F (2022) Linda Veråsdal on ethical travel and responsible tourism in the Gambia. In Sustainability leadership in tourism: interviews, insights, and knowledge from practice pp 405–409. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-031-05314-6_64

Kamata H (2022) Tourist destination residents’ attitudes towards tourism during and after the COVID-19 pandemic. Curr Issue Tour 25(1):134–149

Katemliadis I, Kolongou ES, Drousiotis P (2021) Rising sustainability standards: the Cyprus sustainable tourism initiative. Worldwide Hosp Tour Themes 13(6):754–762. https://doi.org/10.1080/13683500.2021.1881452

Kerdpitak C (2022) The effects of innovative management, digital marketing, service quality and supply chain management on performance in cultural tourism business. Uncertain Supply Chain Manage 10(3):771–778. https://doi.org/10.5267/j.uscm.2022.4.005

Ketter E (2020) Millennial travel: tourism micro-trends of European Generation Y. J Tour Futures 7(2):192–196. https://doi.org/10.1108/JTF-10-2019-0106

Komppula R (2014) The role of individual entrepreneurs in the development of competitiveness for a rural tourism destination—a case study. Tour Manage 40:361–371. https://doi.org/10.1016/j.tourman.2013.07.007

Kim YJ, Palakurthi R, Hancer M (2012) The environmentally friendly programs in hotels and customers’ intention to stay: an online survey approach. Int J Hosp Tour Adm 13(3):195–214. https://doi.org/10.1080/15256480.2012.698169

Kilic M, Kaya I (2015) Investment project evaluation by a decision making methodology based on type-2 fuzzy sets. Appl Soft Comput 27:399–410. https://doi.org/10.1016/j.asoc.2014.11.028

Klintman M (2012) Issues of scale in the global accreditation of sustainable tourism schemes: toward harmonized re-embeddedness?. Sustainability: Science, Practice and Policy 8(1):59–69. https://doi.org/10.1080/15487733.2012.11908085

Kularatne T, Wilson C, Månsson J, Hoang V, Lee (2019) Do environmentally sustainable practices make hotels more efficient? A study of major hotels in Sri Lanka. Tourism Management 71:213-225 https://doi.org/10.1016/j.tourman.2018.09.009

Kumar A, Misra SC, Chan FT (2022) Leveraging AI for advanced analytics to forecast altered tourism industry parameters: a COVID-19 motivated study. Expert Syst Appl 210:118628. https://doi.org/10.1016/j.eswa.2022.118628

Article   PubMed   PubMed Central   Google Scholar  

Kuo FI, Fang WT, LePage BA (2022) Proactive environmental strategies in the hotel industry: eco-innovation, green competitive advantage, and green core competence. J Sustain Tour 30(6):1240–1261. https://doi.org/10.1080/09669582.2021.1931254

Lasso A, Dahles H (2018) Are tourism livelihoods sustainable? Tourism development and economic transformation on Komodo Island, Indonesia. Asia Pacific J Tour Res 23(5):473–485. https://doi.org/10.1080/10941665.2018.1467939

Lee CK, Olya H, Ahmad MS, Kim KH, Oh MJ (2021) Sustainable intelligence, destination social responsibility, and pro-environmental behaviour of visitors: evidence from an eco-tourism site. J Hosp Tour Manag 47:365–376. https://doi.org/10.1016/j.jhtm.2021.04.010

Lee SW, Xue K (2020) A model of destination loyalty: Integrating destination image and sustainable tourism. Asia Pacific J Tour Res 25(4):393–408. https://doi.org/10.1080/10941665.2020.1713185

Liburd JJ, Edwards D (eds) (2010) Understanding the sustainable development of tourism. Goodfellow, Oxford, pp 19–44

Liu CH, Jiang JF, Gan B (2021) The antecedent and consequence behaviour of sustainable tourism: integrating the concepts of marketing strategy and destination image. Asia Pacific J Tour Res 26(8):829–848. https://doi.org/10.1080/10941665.2021.1908384

Liu CR, Lin WR, Wang YC, Chen SP (2019) Sustainability indicators for festival tourism: a multi-stakeholder perspective. J Qual Assur Hosp Tour 20(3):296–316. https://doi.org/10.1080/1528008X.2018.1530165

Liu-Lastres B, Wen H, Huang WJ (2023) A reflection on the Great Resignation in the hospitality and tourism industry. Int J Contemp Hosp Manag 35(1):235–249. https://doi.org/10.1108/IJCHM-05-2022-0551

Liu T, Juvan E, Qiu H, Dolnicar S (2022) Context-and culture-dependent behaviors for the greater good: a comparative analysis of plate waste generation. J Sustain Tour 30(6):1200–1218. https://doi.org/10.1080/09669582.2021.1918132

Liu Z (2003) Sustainable tourism development: a critique. J Sustain Tour 11(6):459–475. https://doi.org/10.1080/09669580308667216

Lopes HS, Remoaldo PC, Ribeiro V, Martin-Vide J (2022) Pathways for adapting tourism to climate change in an urban destination—evidences based on thermal conditions for the Porto Metropolitan Area (Portugal). J Environ Manage 315:115161. https://doi.org/10.1016/j.jenvman.2022.115161

Lu J, Nepal SK (2009) Sustainable tourism research: an analysis of papers published in the Journal of Sustainable Tourism. J Sustain Tour 17(1):5–16. https://doi.org/10.1080/09669580802582480

Mamirkulova G, Mi J, Abbas J, Mahmood S, Mubeen R, Ziapour A (2020) New Silk Road infrastructure opportunities in developing tourism environment for residents better quality of life. Global Ecol Conserv 24:e01194. https://doi.org/10.1016/j.gecco.2020.e01194

Mathew PV (2022) Sustainable tourism development: discerning the impact of responsible tourism on community well-being. J Hosp Tour Insights 5(5):987–1001. https://doi.org/10.1108/JHTI-02-2021-0052

Millar M, Baloglu S (2009) A green room experience: a comparison of business and leisure travelers’ preferences. Hospitality Management 1–12

MacKenzie N, Gannon MJ (2019) Exploring the antecedents of sustainable tourism development. Int J Contemp Hosp Manag 31(6):2411–2427. https://doi.org/10.1108/IJCHM-05-2018-0384

Moayerian N, McGehee NG, Stephenson MO Jr (2022) Community cultural development: exploring the connections between collective art making, capacity building and sustainable community-based tourism. Ann Tour Res 93:103355. https://doi.org/10.1016/j.annals.2022.103355

Molden O, Abrams J, Davis EJ, Moseley C (2017) Beyond localism: the micropolitics of local legitimacy in a community-based organization. J Rural Stud 50:60–69. https://doi.org/10.1016/j.jrurstud.2017.01.001

Monaco S (2018) Tourism and the new generations: emerging trends and social implications in Italy. J Tour Futures 4(1):7–15. https://doi.org/10.1108/JTF-12-2017-0053

Mowforth M, Munt I (2019) Tourism and sustainability: development, globalisation, and new tourism in the third world. Routledge

Moyle BD, Weaver DB, Gössling S, McLennan CL, Hadinejad A (2022) Are water-centric themes in sustainable tourism research congruent with the UN Sustainable Development Goals? J Sustain Tour 30(8):1821–1836. https://doi.org/10.1080/09669582.2022.2083903

Nash D, Butler R (1990) Towards sustainable tourism. Tour Manage 11(3):263–264

Nguyen TQT, Young T, Johnson P, Wearing S (2019) Conceptualising networks in sustainable tourism development. Tour Manage Perspect 32:100575. https://doi.org/10.1016/j.tmp.2019.100575

Nok LC, Suntikul W, Agyeiwaah E, Tolkach D (2017) Backpackers in Hong Kong–motivations, preferences and contribution to sustainable tourism. J Travel Tour Mark 34(8):1058–1070. https://doi.org/10.1080/10548408.2016.1276008

Nunkoo R, Sharma A, Rana NP, Dwivedi YK, Sunnassee VA (2021) Advancing sustainable development goals through interdisciplinarity in sustainable tourism research. Journal of Sustainable Tourism 1–25. https://doi.org/10.1080/09669582.2021.2004416

O’Hare D, Pegoraro A, Scarles C (2020) “Instagrammability”: an exploration of social media engagement and destination image in organic contexts. Curr Issue Tour 23(1):1–18

Pan SY, Gao M, Kim H, Shah KJ, Pei SL, Chiang PC (2018) Advances and challenges in sustainable tourism toward a green economy. Sci Total Environ 635:452–469. https://doi.org/10.1016/j.scitotenv.2018.04.134

Article   CAS   PubMed   ADS   Google Scholar  

Partanen M, Sarkki S (2021) Social innovations and sustainability of tourism: Insights from public sector in Kemi Finland. Tourist Studies 21(4):550–571. https://doi.org/10.1177/14687976211040246

Poudel S, Nyaupane GP, Budruk M (2016) Stakeholders’ perspectives of sustainable tourism development: a new approach to measuring outcomes. J Travel Res 55(4):465–480. https://doi.org/10.1177/0047287514563166

Rahmadian E, Feitosa D, Zwitter A (2022) A systematic literature review on the use of big data for sustainable tourism. Curr Issue Tour 25(11):1711–1730. https://doi.org/10.1080/13683500.2021.1974358

Rasoolimanesh SM, Ramakrishna S, Hall CM, Esfandiar K, Seyfi S (2023) A systematic scoping review of sustainable tourism indicators in relation to the sustainable development goals. J Sustain Tour 31(7):1497–1517. https://doi.org/10.1080/09669582.2020.1775621

Rastegar R, Ruhanen L (2021) A safe space for local knowledge sharing in sustainable tourism: an organisational justice perspective. Journal of Sustainable Tourism 1–17. https://doi.org/10.1080/09669582.2021.1929261

Rodríguez-Díaz B, Pulido-Fernández JI (2020) Sustainability as a key factor in tourism competitiveness: a global analysis. Sustainability 12(1):51. https://doi.org/10.3390/su12010051

Roxas FMY, Rivera JPR, Gutierrez ELM (2020-a) Framework for creating sustainable tourism using systems thinking. Current Issues in Tourism 23(3):280–296. https://doi.org/10.1080/13683500.2018.1534805

Roxas FMY, Rivera JPR, Gutierrez ELM (2020b) Mapping stakeholders’ roles in governing sustainable tourism destinations. Journal of Hospitality and Tourism Management 45:387–398. https://doi.org/10.1016/j.jhtm.2020.09.005

Saaty TL (1980) The analytic hierarchy process: planning, priority setting, resources allocation. McGraw-Hill, London

Šagovnović I, Stamenković I (2022) Investigating values of green marketing tools in predicting tourists’ eco-friendly attitudes and behavior. Journal of Ecotourism 1–23. https://doi.org/10.1080/14724049.2022.2075003

Sahoo BK, Nayak R, Mahalik MK (2022) Factors affecting domestic tourism spending in India. Ann Tour Res Empirical Insights 3(2):100050. https://doi.org/10.1016/j.annale.2022.100050

Santos M, Veiga C, Santos JAC, Águas P (2022) Sustainability as a success factor for tourism destinations: a systematic literature review. Worldwide Hosp Tour Themes 14(1):20–37. https://doi.org/10.1108/WHATT-10-2021-0139

Sarı IU, Öztayşi B, Kahraman C (2013) Fuzzy analytic hierarchy process using type-2 fuzzy sets: an application to warehouse location selection. In Multicriteria decision aid and artificial intelligence pp285–308. John Wiley and Sons, Ltd. https://doi.org/10.1002/9781118522516.ch12

Schmude J, Zavareh S, Schwaiger KM, Karl M (2020) Micro-level assessment of regional and local disaster impacts in tourist destinations. In Tourism in changing natural environments pp. 98–116. Routledge

Scott D, Gössling S (2022) A review of research into tourism and climate change—launching the annals of tourism research curated collection on tourism and climate change. Ann Tour Res 95:103409. https://doi.org/10.1016/j.annals.2022.103409

Shafiee S, Ghatari AR, Hasanzadeh A, Jahanyan S (2019) Developing a model for sustainable smart tourism destinations: a systematic review. Tour Manage Perspect 31:287–300

Shafiee S, Jahanyan S, Ghatari AR, Hasanzadeh A (2023) Developing sustainable tourism destinations through smart technologies: a system dynamics approach. J Simul 17(4):477–498. https://doi.org/10.1080/17477778.2022.2030656

Shang Y, Lian Y, Chen H, Qian F (2023) The impacts of energy resource and tourism on green growth: evidence from Asian economies. Resour Policy 81:103359. https://doi.org/10.1016/j.resourpol.2023.103359

Sharif A, Afshan S, Nisha N (2017) Impact of tourism on CO 2 emission: evidence from Pakistan. Asia Pacific J Tour Res 22(4):408–421. https://doi.org/10.1080/10941665.2016.1273960

Sharma K, Arya V, Mathur HP (2023) New higher education policy and strategic plan: commensurate India’s higher education in global perspective. FIIB Business Review 23197145221125351

Sharpley R (2023) Sustainable tourism governance: local or global? Tour Recreat Res 48(5):809–812. https://doi.org/10.1080/02508281.2022.2040295

Sharpley R (2021) On the need for sustainable tourism consumption. Tour Stud 21(1):96–107. https://doi.org/10.1177/1468797620986087

Sharpley R (2000) Tourism and sustainable development: Exploring the theoretical divide. J Sustain Tour 8(1):1–19. https://doi.org/10.1080/09669580008667346

Shu H, Yu Q, Liu K, Wang A, Zha J (2022) Understanding wage differences across tourism-characteristic sectors: insights from an extended input-output analysis. J Hosp Tour Manag 51:88–104. https://doi.org/10.1016/j.jhtm.2022.02.030

Sisneros-Kidd AM, Monz C, Hausner V, Schmidt J, Clark D (2019) Nature-based tourism, resource dependence, and resilience of Arctic communities: framing complex issues in a changing environment. J Sustain Tour 27(8):1259–1276. https://doi.org/10.1080/09669582.2019.1612905

Sobaih AEE, Elshaer I, Hasanein AM, Abdelaziz AS (2021) Responses to COVID-19: The role of performance in the relationship between small hospitality enterprises’ resilience and sustainable tourism development. Int J Hosp Manag 94:102824. https://doi.org/10.1016/j.ijhm.2020.102824

Sørensen EB, Hjalager AM (2020) Conspicuous non-consumption in tourism: non-innovation or the innovation of nothing? Tour Stud 20(2):222–247. https://doi.org/10.1177/1468797619894463

Streimikiene D, Svagzdiene B, Jasinskas E, Simanavicius A (2021) Sustainable tourism development and competitiveness: the systematic literature review. Sustain Dev 29(1):259–271. https://doi.org/10.1002/sd.2133

Strzelecka M, Dąbrowski JM, Uramowska M (2021) A review of environmental sustainability in tourism. J Clean Prod 279:123748

Sullivan K, Thomas S, Rosano M (2018) Using industrial ecology and strategic management concepts to pursue the Sustainable Development Goals. J Clean Prod 174:237–246. https://doi.org/10.1016/j.jclepro.2017.10.201

Sun L, Wang Y (2021) Global economic performance and natural resources commodity prices volatility: evidence from pre and post COVID-19 era. Resour Policy 74:102393. https://doi.org/10.1016/j.resourpol.2021.102393

Sun YY, Li M, Lenzen M, Malik A, Pomponi F (2022) Tourism, job vulnerability and income inequality during the COVID-19 pandemic: a global perspective. Ann Tour Res Empirical Insights 3(1):100046. https://doi.org/10.1016/j.annale.2022.100046

Terry Mok Connie Lam (1998) Hotel and tourism development in Vietnam. Journal of Travel and Tourism Marketing 7(1):85-91. https://doi.org/10.1300/J073v07n01_06

Thapa K, King D, Banhalmi-Zakar Z, Diedrich A (2022) Nature-based tourism in protected areas: a systematic review of socio-economic benefits and costs to local people. International Journal of Sustainable Development and World Ecology 29(7):625–640 https://doi.org/10.1080/13504509.2022.2073616

Tolkach D, King B, Pearlman M (2013) An attribute-based approach to classifying community-based tourism networks. Tour Plann Dev 10(3):319–337. https://doi.org/10.1080/21568316.2012.747985

Torres-Delgado A, Saarinen J (2017) Using indicators to assess sustainable tourism development: a review. New research paradigms in tourism geography, 31–47

Tuan VK, Rajagopal P (2019) Analyzing factors affecting tourism sustainable development towards Vietnam in the new era. Eur J Business Innov Res 7(1):30–42

Tzschentke N, Kirk D, Lynch PA (2004) Reasons for going green in serviced accommodation establishments. Int J Contemp Hosp Manag 16(2):116–124. https://doi.org/10.1108/09596110410520007

UNWTO (2020) Sustainable Development Goals. United Nations World Tourism Organization. Accessed during October 2023, https://www.unwto.org/sustainable-development-goals

UNWTO (2018) Tourism and the SDGs: Journey to 2030. World Tourism Organization

Valenti WC, Kimpara JM, Preto BDL, Moraes-Valenti P (2018) Indicators of sustainability to assess aquaculture systems. Ecol Ind 88:402–413. https://doi.org/10.1016/j.ecolind.2017.12.068

Verma R, Arya V, Thomas A, Bolognesi E, Mueller J (2023) Does startup culture in the emerging country grow around societal sustainability? An empirical study through the lens of co-creational capital and green intellect. J Intellect Cap 24(4):1047–1074

Vespestad MK, Lindberg F, Mossberg L (2019) Value in tourist experiences: How nature-based experiential styles influence value in climbing. Tour Stud 19(4):453–474. https://doi.org/10.1177/1468797619837966

Weaver DB, Moyle B, McLennan CLJ (2022) The citizen within: positioning local residents for sustainable tourism. J Sustain Tour 30(4):897–914. https://doi.org/10.1080/09669582.2021.1903017

Westoby R, Gardiner S, Carter RW, Scott N (2021) Sustainable livelihoods from tourism in the “10 New Balis” in Indonesia. Asia Pacific J Tour Res 26(6):702–716. https://doi.org/10.1080/10941665.2021.1908386

World Tourism Organization (WTO) (2019), International tourism results 2018 and outlook 2019. http://cf.cdn.unwto.org/sites/all/files/pdf/unwto_barometer_jan19_presentation_en.pdf (accessed 24 August 2019)

World Travel and Tourism Council (WTTC) represents the Travel and Tourism Sector globally (2020), Accessed at https://wttc.org/ retrieved on 15 January 2020

Wu JS, Barbrook-Johnson P, Font X (2021) Participatory complexity in tourism policy: understanding sustainability programmes with participatory systems mapping. Ann Tour Res 90:103269. https://doi.org/10.1016/j.annals.2021.103269

Zadeh L (1975) The concept of a linguistic variable and its application to approximate reasoning. Prt I. Inform Sci 8(3):199–249. https://doi.org/10.1016/0020-0255%2875%2990036-5

Zhang J (2016) Weighing and realizing the environmental, economic and social goals of tourism development using an analytic network process-goal programming approach. J Clean Prod 127:262–273. https://doi.org/10.1016/j.jclepro.2016.03.131

Download references

Author information

Authors and affiliations.

Rabat Business School, International University of Rabat, Rabat, Morocco

ISM University of Management and Economics: ISM Vadybos Ir Ekonomikos Universitetas, Gedimino St. 7, 01103, Vilnius, Lithuania

Vilte Auruskeviciene

School of Management, MIT World Peace University, Pune, India

Srishti Agarwal & Priyanka Kokatnur

Great Lakes Institute of Management, Gurgaon Campus, Gurgaon, India

Harish Kumar

Indian Institute of Management, Ranchi, India

Rajeev Verma

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to the study’s conception and design. Vikas Arya conducted the analysis of data and provided an interpretation of the findings. Vilte Auruskeviciene wrote the manuscript draft and ensured consistency in referencing and citation formatting. Srishti Agarwal collected data, collaborated with Vikas Arya to analyze the data, and contributed to the discussion of the findings. Priyanka Kokatnur contributed to the research methodology development and data collection. Harish Kumar provided insights to the theoretical and managerial aspects of the study and contributed to the discussion section. Rajeev Verma conducted a literature review and collaborated with Vilte Auruskeviciene to integrate the literature review into the manuscript. All authors provided comments on previous versions of the manuscript, and they all read and approved the final manuscript.

Corresponding author

Correspondence to Vilte Auruskeviciene .

Ethics declarations

Ethics approval.

Not applicable.

Consent to participate

Consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Responsible Editor: Philippe Garrigues

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Arya, V., Auruskeviciene, V., Agarwal, S. et al. Let us take a walk to the sustainable tourism practices: a qualitative study through the lens of tourism experts. Environ Sci Pollut Res 31 , 12892–12915 (2024). https://doi.org/10.1007/s11356-023-31503-7

Download citation

Received : 15 June 2023

Accepted : 08 December 2023

Published : 04 January 2024

Issue Date : February 2024

DOI : https://doi.org/10.1007/s11356-023-31503-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Sustainable tourism
• Eco-tourism
• AHP-type 2 fuzzy test
• Responsible consumers
• Tourism sustainable equity
• Tourism industry
• Green services
• SDGs in tourism
• Find a journal
• Publish with us
• Track your research

ORIGINAL RESEARCH article

Validity and reliability of anxiety literacy (a-lit) and its relationship with demographic variables in the iranian general population.

• 1 Department of Health Education and Health Promotion, School of Health, Social Development and Health Promotion Research Center, Gonabad University of Medical Sciences, Gonabad, Iran
• 2 Department of Health Education and Health Promotion, School of Health, Social Determinants of Health Research Center, Gonabad University of Medical Sciences, Gonabad, Iran
• 3 Department of Epidemiology and Biostatistics, School of Health, Social Development and Health Promotion Research Center, Gonabad University of Medical Sciences, Gonabad, Iran
• 4 Student Research Committee, Gonabad University of Medical Sciences, Gonabad, Iran
• 5 Social Determinants of Health Research Center, Gonabad University of Medical Sciences, Gonabad, Iran

Introduction: Anxiety disorder is one of the most common mental disorders. This cross-sectional research aimed to determine anxiety literacy (A-Lit) psychometric properties among the Iranian population in 2022.

Methods: This research was conducted on 690 people in Iran in 2022. In this study, people were selected by proportional stratified sampling, and the validity and reliability of the A-Lit designed by Griffiths were assessed. Validity of A-Lit was assessed by face validity, content validity, and confirmatory factor analysis. Reliability of A-Lit was evaluated by the McDonald’s omega coefficient, Cronbach’s alpha coefficient, and test– retest. In analytical sections, the tests of One-way ANOVA, Chi-squared test, and independent samples t -test were used.

Results: The rates of S-CVI/Ave and CVR for A-Lit were 0.922 and 0.774, respectively. In confirmatory factor analysis, three items were deleted because the factor loading was less than 0.4, and goodness-of-fit indexes (Some of goodness-of-fit indexes: χ2/df = 4.175, GFI: 0.909, RMSEA = 0.068, PCFI = 0.745, AGFI = 0.883) were confirmed as the final model with 19 items. For all items, the Cronbach’s alpha coefficient was 0.832, the McDonald’s omega coefficient was 0.835, and the intraclass correlation coefficient was 0.874. According to the results of this study, 1.3% ( n = 9) did not answer any questions correctly and 8.4% ( n = 58) were able to answer 1–6 questions correctly. Approximately 72% ( n = 495) were able to answer 7–12 questions, and eventually only 18.6% ( n = 128) were able to answer 13 questions and more. There was a significant relationship between sex, age group, occupation status, marital status, and get information related to mental illness with A-Lit level ( p < 0.05).

Conclusion: The Persian version of A-Lit was confirmed with 19 items, and this scale is a reliable tool for measuring A-Lit in the general population. The results also showed that a few people have a higher level of anxiety literacy and that educational and intervention programs need to be designed and implemented for the public population.

Introduction

Anxiety disorder is one of the most common mental disorders ( 1 ). Anxiety disorder is characterized by negative feelings and severe arousal, with the main characteristics of fear, doubt, and concern ( 2 ). These people often estimate the risk above the actual level and are constantly concerned without the cause ( 2 ). Sometimes this concern increases so much that it disrupts one’s daily performance in different areas of life, including job, educational, social, and marital relationships, and may even restrict the individual as a crippling physical illness ( 2 ).

Systematic review in 2016 reported that the prevalence of anxiety disorder in the world was between 3.8 and 25% ( 3 ). Based on the results of a global study on 204 countries in 2020, before the COVID-19 pandemic, the prevalence of anxiety was 3,824 per 100,000, which was 4,802 per 100,000 after the COVID-19 pandemic, which showed an increase of 25% ( 4 ). The findings of a systematic review in the Iranian population showed that 42% had anxiety disorders, 36% in women and 27% in men. In addition, the rate of obvious anxiety was 21% and the hidden anxiety rate was 24% ( 5 ).

Anxiety disorder over time can cause other disorders in the body, and there is a relationship between anxiety and physical disorders, and increased anxiety can increase and aggravate these physical disorders ( 6 ). Anxiety disorder, if untreated, can be associated with significant personal and social costs due to repeated referrals to receive primary care, reduced productivity in the workplace, unemployment, and increased risk of mood disorders, drug use, and social relationship disorder ( 7 , 8 ). Severe anxiety also has a strong correlation with depression, and if not considered and no cure for it, the risk of depression increases ( 9 ).

Health literacy (HL) is one of the most effective and predictive factors of people’s health ( 10 , 11 ) and mental health literacy (MHL) is a main strategy in early recognition of mental illnesses and promoting help seeking ( 12 , 13 ). Anxiety Literacy (A-Lit) is a subsidiary of MHL that examines people’s literacy in the field of anxiety ( 14 ). According to the definition of Jorm et al. ( 15 ), regarding MHL, A-Lit refers to the set of beliefs and knowledge about anxiety disorders that helps in their recognition, prevention, and management. Therefore, recognizing anxiety disorders and seeking help to prevent and manage them are two key elements of A-Lit.

Whereas, the recognition of social anxiety disorder was low even in developed countries such as the United Kingdom ( 16 ). In the Furnham and Lousley ( 14 ) study in UK, people’s A-Lit was poor and people were not aware of A-Lit. Also in the Thai Quynh-Chi et al. ( 16 ) study in Vietnam, most students had low A-Lit and could not detect anxiety symptoms.

In addition, long delays in seeking mental health services were observed in most people with anxiety disorders. In social anxiety disorder, the interval from the onset of symptoms to seeking treatment is estimated to be between 9 and 28 years and between 6 and 10 years in generalized anxiety disorder ( 17 ). As a result, considering the widespread prevalence of anxiety disorders in Iran, measuring A-Lit of society and implementing interventions to promote A-Lit can lead to early diagnosis of anxiety disorders and promotion of help seeking. The prerequisite for implementing these interventions is the existence of a specific, valid, and reliable tool.

In fact, based on the our knowledge, in Iran despite numerous studies that have examined MHL ( 18 , 19 ), no study has specifically examined A-Lit. The lack of valid and reliable tools may be a possible reason for this problem. After searching and reviewing numerous sources, the only specific tool developed in this regard is (A-Lit, developed by KG) ( 20 ). This scale consists of 22 questions that examine the level of literacy in relation to anxiety disorder ( 20 ). Due to the lack of an appropriate scale for measuring A-Lit, this research was performed to determine A-Lit psychometric properties among the public population in 2022.

This cross-sectional research aimed to determine the psychometric properties of A-Lit among the Iranian population in 2022. Data was collected within 5 months (October 4, 2022 to March 6, 2023).

Sample size

A sample size of 500 or more is very good for factor analysis ( 21 , 22 ). In this research, to run the confirmatory factor analysis, a sample size of 800 participants was considered, and finally, a sample size of 690 was analyzed (110 questionnaires were removed in analysis section due to more missing information).

Sampling method

Proportional stratified sampling was used for selecting participants. First, the number of health centers with their population was determined ( n = 3). Each health center was considered as a stratum and based on the population of each stratum, the required sample size for each health center was determined. Subsequently, we referred to the health centers and people who had the health record in health centers and had the inclusion criteria were determined. Then, the require samples were selected by simple random sampling method and the questionnaires were given to participants and completed by self-report. People who were unable to completed the questionnaire due to lack of literacy or vision problem, the questionnaire was completed by the interview method and by the questioner. People who were 18 or over the age of 18, having satisfaction and informed consent, had not cognitive problem, and living in Gonabad city were the inclusion criteria.

Instruments

(1) Demographic part: This section includes characteristics such as age, education level, economic status, sex, occupational status, method of obtaining health information, marital status, and so on.

(2) A-Lit scale: This scale contains 22 questions that examine the A-Lit level. The questions are measured as “False,” “True,” and “I do not know.” Each correct response gets 1 score, and a high score shows high levels of people’s A-Lit. The Cronbach’s alpha and test–test in the study of Gulliver et al. ( 20 ) were 0.76 and 0.83.

Translation and cultural adaptation

At first, permission was obtained from the original designer of the scale. The process of this section was conducted using the World Health Organization Guideline in three parts ( 23 ). In the first part, the scale was translated by two experts from English to Persian version and after comparing and reviewing one version of the two Persian versions was created. After that, the Persian version was back translated to the English version and compared with the original English version. In the third part, the back-translated version of the scale was translated to the Persian version, and the final version of the scale was created.

Validity of A-Lit was assessed by qualitative face validity, quantitative and qualitative content validity methods, and structural validity. Qualitative face validity of A-Lit was assessed by 9 specialists (experts of Psychology and experts of Health Education and Health Promotion) and 12 participants of target group. Qualitative and quantitative content validity of A-Lit was assessed by 7 specialists (experts of Psychology and experts of Health Education and Health Promotion). For evaluation the quantitative content validity, S-CVI/Ave (scale content validity index averaging) and CVR (content validity ratio) were assessed ( 24 ). The value of >0.90 is acceptable for S-CVI/Ave ( 25 ). According to the Lawshe table, because the number of evaluators was 7 specialists, the value of above 0.75 is acceptable for CVR ( 26 ).

To perform CFA, the software AMOS V.24 was used. In this section, the data of outliers were determined and deleted by the Mahalanobis test. Then, data normality was evaluated using kurtosis and skewness tests. To evaluate the model in CFA, the following goodness of fit indexes were used: chi-square ratio to degree of freedom (χ 2 /df), adjusted goodness of fit index (AGFI), root mean square error of approximation (RMSEA), parsimony comparative fit index (PCFI), root mean square residual (RMR), parsimonious normed fit index (PNFI), parsimony goodness of fit index (PGFI), and goodness of fit index (GFI) ( 27 – 29 ). The acceptable goodness of fit indexes is χ 2 /df less than 5, RMR and RMSEA less than 0.08, AGFI more than 0.8, PCFI, PGFI, and PNFI more than 0.5, and GFI more than 0.9 ( 27 – 31 ).

Reliability

The internal consistency was surveyed using Cronbach’s alpha coefficient using SPSS V.24 software. A score ranging from 0.70 to 0.95 is considered acceptable for internal reliability (Cronbach’s alpha) ( 32 , 33 ). The McDonald’s omega coefficient was evaluated using JASP version 0.11.1. To evaluate the test–retest reliability, in this study, 30 participants were selected and examined twice with a period of 1 month. For test–retest reliability, the intraclass correlation coefficient (ICC) was checked (ICC > 0.80 is acceptable) ( 34 ).

Statistical analysis in analytical sections

In this study the tests of One-way ANOVA, Chi-square, and independent samples t -test were used to check the relationship between the qualitative variables with A-Lit by SPSS version 24. Also, in the one-way ANOVA test, post hoc test (Tukey’s honestly significant difference) was used to evaluate the relationships between variables.

Demographic characteristics

In this study, most participants were female ( n = 370, 53.6%), in the age group 18 to 30 ( n = 354, 51.3%), university students ( n = 274, 39.7%), had an academic education level ( n = 463, 67.1%), and medium economic status ( n = 474, 68.7%). The most common ways to obtain health information were the Internet ( n = 351, 50.9%) and physician/health care providers ( n = 148, 21.4%). The most common way to get information related to mental illness was the Internet ( n = 204, 29.6%) ( Table 1 ).

Table 1 . Frequency distribution of demographic variables and its relationship with A-Lit ( n = 690).

Psychometric section results

Validity assessment.

In the face validity and content validity, four items were modified and the rates of S-CVI/Ave and CVR for A-Lit were 0.922 and 0.774, respectively.

In this section, the goodness of fit indices confirmed the final model ( Table 2 ). The factor loading of all items was above 0.4, and only three questions were deleted because the factor loading was less than 0.4 ( Table 3 and Figure 1 ).

Table 2 . The model fit indicators of the A-Lit scale.

Table 3 . Factor loadings of the A-Lit scale in the CFA among general population.

Figure 1 . Standardized parameter estimates for the factor structure of the A-Lit scale (all items loadings are significant at p < 0.001).

Reliability assessment

Cronbach’s alpha was 0.832 for A-Lit. McDonald’s omega coefficient of A-Lit was 0.835. In the test–retest, the rate of intraclass correlation coefficient of A-Lit was 0.874 (95% Confidence Interval, lower bound = 0.751, upper bound = 0.937, p < 0.001).

Descriptive and analytical sections of the results

There was a significant relationship between sex and A-Lit, and the level of A-Lit was higher among women than men ( p < 0.001). Also, compared to men, more percentage of women were able to answer 13 questions and more ( p = 0.010). There was a significant relationship between the age group and the A-Lit level, and the age group of 18–30 years gained a higher score on A-Lit ( p < 0.001). There was a significant statistical relationship between job status and HL, and university students had a higher A-Lit level ( p < 0.001). Based on the Tukey’s post hoc , the age group of 18–30 had more A-Lit level than other groups ( p < 0.05) ( Supplementary Table S1 ). Other results are visible in Supplementary Table S1 .

There was no significant relationship between education level and A-Lit ( p = 0.126), but those with academic education levels were able to answer 13 questions and more compared to other people ( p = 0.010). The results also showed that people who obtained general health information from books and the Internet had higher A-Lit scores ( p = 0.001). People who reported information about mental disorders had a higher level of A-Lit ( p < 0.001). People who obtained information about mental disorders from the book had higher A-Lit scores ( p = 0.047). Other information can be found in Table 1 .

In this study, 1.3% ( n = 9) of participants did not answer any questions correctly 8.4% ( n = 58) were able to answer 1–6 questions correctly. Approximately 72% ( n = 495) were able to answer 7–12 questions, and eventually only 18.6% ( n = 128) were able to answer 13 questions and more. The results in Table 4 are related to the accountability status of the participants for each question.

Table 4 . Participants’ response to the A-Lit scale.

Psychometric section

Because the psychometric characteristics of A-Lit have not been examined in Iran, the validity and reliability of the Persian version of the A-Lit scale in the Iranian general population was examined. The original version of the A-Lit scale contains 22 items, and a high score indicates a high level of A-Lit. After evaluating the psychometric properties of A-Lit, 3 questions were removed from the Persian version of the questionnaire because of a factor loading of less than 0.4, and 19 items were confirmed for the modified version.

According to the CFA test, factor loading values were calculated and all questions were larger than 0.4, except for three items. Therefore, at this stage, only three questions were deleted and the final model was confirmed with 19 questions. The cultural and social differences of Iranian society could have been the reason for the elimination of these questions. For example, one of the questions deleted from the Persian version was about acupuncture, which seems to be not well known in the Iranian population. Based on the literature review, there were no studies on the psychometric properties of the A-Lit scale in Iran.

Another study examined the psychometric characteristics of the mental health literacy questionnaire among Iranian soldiers ( 35 ). The study showed that two of the 33 cases that entered the content analysis phase were eliminated by a factor loading below 0.35, and the questionnaire was confirmed with 31 items. The results of the psychometric properties of depression literacy (D-Lit) in the Iranian community showed that only one question was eliminated in the confirmatory factor analysis stage, and D-Lit was confirmed with 21 questions ( 36 ).

In the reliability stage of the A-lit scale, the Cronbach’s alpha coefficient was 0.832 and McDonald’s omega coefficient was 0.835. In addition, the test–retest results indicate the appropriateness of the intraclass correlation coefficient (ICC = 0.874), which agrees with previous studies. In a study of young Australian athletes, the Cronbach’s alpha coefficient of the A-Lit scale was calculated 0.76, which was acceptable ( 20 ). The reliability of the questionnaire was also evaluated by the test–retest, with an acceptable value of 0.83 ( 20 ). Another study in Australia on adults with symptoms of untreated social anxiety disorder found that the results of the test–retest for the social A-Lit in the control group was 0.85 ( 37 ).

As can be seen, the Persian version of the questionnaire is similar to the original version. Therefore, this scale can be used to measure the level of A-Lit of Iranian population, and if A-Lit is low, an intervention program can be implemented in this regard. Educational interventions can be a way to increase A-Lit and improve help seeking attitudes ( 38 ). A clinical trial study showed that if people’s A-Lit is low, intervention to increase social A-lit can be helpful ( 37 ). Therefore, the first step in determining the status of A-Lit in any population is the existence of a valid and reliable tool to plan and implement subsequent preventive programs.

Descriptive and analytical sections

According to the findings of this study, there was a significant relationship between A-Lit and sex, and A-Lit was higher in women than in men. In addition, the percentage of answering to 13 questions and more in women was significantly higher than that in men. There was also a significant statistical relationship about job status, and students had higher A-Lit levels than others. The findings of this study are very similar to those of other studies. A study of Irish teenagers with anxiety disorders showed that girls had better A-Lit than boys ( 39 ). Another study of Singapore students 18 to 24 years old and through social media platforms ( 40 ) showed that the A-Lit level was significantly higher in female students, with no difference from the present study. In addition, students in medical sciences had higher A-Lit levels than others. In another study conducted in Turkey on medical students showed that girls had higher mental health literacy ( 41 ).

In the study of age groups, it was found that the A-Lit level was significantly higher in the age group of 18–30 years than in the others, and they gained a higher score in A-Lit. Because the prevalence of anxiety disorders is usually more from late adolescence, the present study is somewhat justified ( 42 ).

There was no significant relationship between the level of education and A-Lit in the present study; however, the percentage of answers to 13 questions and more was higher in academic education than in other people. Although no similar study was found in this field, another study conducted in Turkey and medical students showed that with the increase in students’ academic activities, the level of mental health literacy also increased ( 41 ).

In this study, the most important source of health information and the main source of information about mental disorders was Internet. The results of this study showed that people who obtained their health information and their information related to mental illness from books had higher A-Lit scores. In another study on medical students in Turkey, the Internet social media was the main source of health information ( 41 ). The results of a study on Iranian students showed that the Internet and health care providers were the main sources of mental health literacy information and that health care providers were the most reliable source of information ( 43 ). The results of our study were in line with the results of the Mahmoodi et al. ( 43 ) study, and the Internet was the most important source of information. However, there is a difference in the second source of health information that can be due to the differences in the groups being studied because the present study was performed on the general population, whereas the Mahmoodi et al. ( 43 ) study was only performed on university students.

While most of the studies found in this field have been conducted in adolescents and young people, in the present study in order to reduce possible sources of bias, We tried to select the participants as much as possible from different population groups. For example, the age difference between the participants and the target population can reduce the generalizability of the study results. Therefore, in the current study, individuals with different ages were examined to increase the generalizability of the results.

Anxiety disorder can be affected by various factors and affects the different functions of individuals. Few people with this disorder are looking for help and receiving help, and they usually have little information about the disorder ( 37 ). Therefore, knowing the status of A-Lit and its related factors can help design more effective preventive programs. Also, since anxiety disorder can cause other problems such as physical disorders and as a result impose subsequent costs, with timely identification and treatment of these disorders in the clinical setting, it is possible to prevent the mentioned problems or reduce their severity.

The application of this study in clinical settings

Cardiovascular disease is the main cause of death in the world and Iran ( 44 , 45 ). Anxiety is one of the most common problems in patients with cardiovascular disease, including coronary artery disease and heart failure. Anxiety causes coronary artery disease and consequently causes adverse effects and mortality in these patients ( 46 ). On the other hand, anxiety disorder is related with weaker glycemic control in diabetic patients, increased night and morning hypertension in patients with hypertension, increased risk of stroke and heart attack ( 47 – 50 ). The existence of A-Lit tools can examine the status of A-Lit in these patients, and based on their condition, appropriate intervention programs can be designed to increase A-Lit, improve help seeking and treatment, improve and implement anxiety management. Therefore, this tool can be used in various studies and among general population and population of patients used by both researchers and health care providers.

Strengths and limitations

One of the limitations of this study was the lack of similar studies to compare the results. The strengths of this study include the high volume of sample size, conducting the study in the general population, and determination of the status of A-Lit in different groups. While most of the studies found in this field have been conducted in adolescents and young people, in the present study in order to reduce possible sources of bias, We tried to select the participants as much as possible from different population groups. For example, the age difference between the participants and the target population can reduce the generalizability of the study results. Therefore, in the current study, individuals with different ages were examined to increase the generalizability of the results.

The Persian version of A-Lit was confirmed with 19 items, and this scale is a reliable tool for measuring A-Lit in the general population of Iranian society. The results also showed that a few people have a higher level of A-Lit and that educational and intervention programs need to be designed and implemented for populations with insufficient A-Lit.

Data availability statement

The original contributions presented in the study are included in the article/ Supplementary material , further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving humans were approved by Ethics Committee of Gonabad University of Medical Sciences (code of ethics: IR.GMU.REC.1401.070). The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

AJ: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing. MM: Conceptualization, Formal analysis, Investigation, Methodology, Software, Supervision, Validation, Writing – original draft, Writing – review & editing. AM: Conceptualization, Formal analysis, Investigation, Methodology, Software, Validation, Writing – original draft, Writing – review & editing. FN: Conceptualization, Investigation, Writing – original draft, Writing – review & editing. MN: Conceptualization, Investigation, Methodology, Project administration, Resources, Supervision, Visualization, Writing – original draft, Writing – review & editing.

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Acknowledgments

We would like to thank to Social Development and Health Promotion Research Center, Gonabad University of Medical Sciences. Also, we would like to thanks all people who assisted the authors to run this research project.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2024.1359146/full#supplementary-material

Abbreviations

A-Lit, anxiety literacy; D-Lit, depression literacy; MHL, mental health literacy; HL, health literacy; CFA, confirmatory factor analysis; PCFI, parsimony comparative fit index; AGFI, adjusted goodness of fit index; GFI, goodness of fit index; RMSEA, root mean square error of approximation; RMR, root mean square residual; PNFI, parsimonious normed fit index; PGFI, parsimony goodness-of-fit index; X 2 /df, chi-square ratio to degree of freedom; ICC, intraclass correlation coefficient.

1. Collaborators, GMD. Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990–2019: a systematic analysis for the global burden of disease study 2019. Lancet Psychiatry . (2022) 9:137–50. doi: 10.1016/S2215-0366(21)00395-3

Crossref Full Text | Google Scholar

2. Jalongo, MR, and Hirsh, RA. Understanding reading anxiety: new insights from neuroscience. Early Childhood Educ J . (2010) 37:431–5. doi: 10.1007/s10643-010-0381-5

3. Remes, O, Brayne, C, Van Der Linde, R, and Lafortune, L. A systematic review of reviews on the prevalence of anxiety disorders in adult populations. Brain Behav . (2016) 6:e00497. doi: 10.1002/brb3.497

PubMed Abstract | Crossref Full Text | Google Scholar

4. Santomauro, DF, Herrera, AMM, Shadid, J, Zheng, P, Ashbaugh, C, Pigott, DM, et al. Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic. Lancet . (2021) 398:1700–12. doi: 10.1016/S0140-6736(21)02143-7

5. Valizadeh, R, Sarokhani, D, Sarokhani, M, Sayehmiri, K, Ostovar, R, Angh, P, et al. A study of prevalence of anxiety in Iran: systematic review and meta-analysis. Der Pharma Chemica . (2016) 8:48–57.

Google Scholar

6. Sareen, J, Cox, BJ, Clara, I, and Asmundson, GJ. The relationship between anxiety disorders and physical disorders in the US National Comorbidity Survey. Depress Anxiety . (2005) 21:193–202. doi: 10.1002/da.20072

7. Simpson, HB, Neria, Y, Lewis-Fernández, R, and Schneier, F. Anxiety disorders: Theory, research and clinical perspectives . England: Cambridge University Press (2010).

8. Murphy, TK, Bengtson, MA, Tan, JY, Carbonell, E, and Levin, GM. Selective serotonin reuptake inhibitors in the treatment of paediatric anxiety disorders: a review. Int Clin Psychopharmacol . (2000) 15:S47–63. doi: 10.1097/00004850-200008002-00008

9. Coryell, W, Fiedorowicz, JG, Solomon, D, Leon, AC, Rice, JP, and Keller, MB. Effects of anxiety on the long-term course of depressive disorders. Br J Psychiatry . (2012) 200:210–5. doi: 10.1192/bjp.bp.110.081992

10. Wei, Y, McGrath, PJ, Hayden, J, and Kutcher, S. Measurement properties of mental health literacy tools measuring help-seeking: a systematic review. J Ment Health . (2017) 26:543–55. doi: 10.1080/09638237.2016.1276532

11. Hossaini, FA, Darya, F, Rahmanian, V, and Sharifi, N. Health literacy and “COVID-19 anxiety” in Iranian adults: a cross-sectional study. J Health Lit . (2023) 8:52–61. doi: 10.22038/jhl.2022.67387.1342

12. Wei, Y, McGrath, PJ, Hayden, J, and Kutcher, S. Mental health literacy measures evaluating knowledge, attitudes and help-seeking: a scoping review. BMC Psychiatry . (2015) 15:1–20. doi: 10.1186/s12888-015-0681-9

13. Olyani, S, Gholian Aval, M, Tehrani, H, and Mahdiadeh, M. School-based mental health literacy educational interventions in adolescents: a systematic review. J Health Lit . (2021) 6:69–77. doi: 10.22038/jhl.2021.58551.1166

14. Furnham, A, and Lousley, C. Mental health literacy and the anxiety disorders. Health . (2013) 5:521–31. doi: 10.4236/health.2013.53A071

15. Jorm, AF, Korten, AE, Jacomb, PA, Christensen, H, Rodgers, B, and Pollitt, P. “Mental health literacy”: a survey of the public’s ability to recognise mental disorders and their beliefs about the effectiveness of treatment. Med J Aust . (1997) 166:182–6. doi: 10.5694/j.1326-5377.1997.tb140071.x

16. Thai Quynh-Chi, N, Hoang-Minh, D, and Thi, K-AL. Recognition of anxiety disorder and depression and literacy of first-aid support: a cross-sectional study among undergraduate students in ha Noi, Viet Nam 2018. Health Psychol Open . (2021) 8:20551029211015116. doi: 10.1177/20551029211015116

17. Paulus, DJ, Wadsworth, LP, and Hayes-Skelton, SA. Mental health literacy for anxiety disorders: how perceptions of symptom severity might relate to recognition of psychological distress. J Public Ment Health . (2015) 14:94–106. doi: 10.1108/JPMH-09-2013-0064

18. Jafari, A, Nejatian, M, Momeniyan, V, Barsalani, FR, and Tehrani, H. Mental health literacy and quality of life in Iran: a cross-sectional study. BMC Psychiatry . (2021) 21:1–11. doi: 10.1186/s12888-021-03507-5

19. Arezoo, B, Alireza, S, and Hossein, MV. Mental health literacy in Iranian adolescents. PJMHS . (2021) 15:1551–6. doi: 10.53350/pjmhs211561551

20. Gulliver, A, Griffiths, KM, Christensen, H, Mackinnon, A, Calear, AL, Parsons, A, et al. Internet-based interventions to promote mental health help-seeking in elite athletes: an exploratory randomized controlled trial. J Med Internet Res . (2012) 14:e69. doi: 10.2196/jmir.1864

21. Tabatchnick, BG, and Fidell, LS. Using multivariate statistics . MA: Needham Heights (2001).

22. Williams, B, Onsman, A, and Brown, T. Exploratory factor analysis: a five-step guide for novices. Australas J Paramed . (2010) 8:1–13. doi: 10.33151/ajp.8.3.93

23. Organization WH. Process of translation and adaptation of instruments. 2009 . World Health Organization. (2010).

24. Polit, DF, and Beck, CT. The content validity index: are you sure you know what’s being reported? Critique and recommendations. Res Nurs Health . (2006) 29:489–97. doi: 10.1002/nur.20147

25. Shi, J, Mo, X, and Sun, Z. Content validity index in scale development. Journal of Central South University Medical sciences . (2012) 37:152–5. doi: 10.3969/j.issn.1672-7347.2012.02.007

26. Lawshe, CH. A quantitative approach to content validity 1. Pers Psychol . (1975) 28:563–75. doi: 10.1111/j.1744-6570.1975.tb01393.x

27. Henry, JW, and Stone, RW. A structural equation model of end-user satisfaction with a computer-based medical information system. Inf Resours Manag J . (1994) 7:21–33. doi: 10.4018/irmj.1994070102

28. Lomax, RG, and Schumacker, RE. A beginner’s guide to structural equation modeling . New York: Psychology Press (2004).

29. Kline, R. Details of path analysis. Principles and practice of structural equation modeling . New York: Guilford (2005).

30. Schreiber, JB, Nora, A, Stage, FK, Barlow, EA, and King, J. Reporting structural equation modeling and confirmatory factor analysis results: a review. J Educ Res . (2006) 99:323–38. doi: 10.3200/JOER.99.6.323-338

31. Hair, JF, Black, WC, Babin, BJ, Anderson, RE, Black, W, and Anderson, R. Multivariate data analysis (eighth) . United Kingdom: Cengage Learning EMEA (2019).

32. Nunnally, JC. Psychometric theory 3E . New York: Tata McGraw-Hill Education (1994).

33. Bland, JM, and Altman, DG. Statistics notes: Cronbach’s alpha. BMJ . (1997) 314:572. doi: 10.1136/bmj.314.7080.572

34. Koo, TK, and Li, MY. A guideline of selecting and reporting Intraclass correlation coefficients for reliability research. J Chiropr Med . (2016) 15:155–63. doi: 10.1016/j.jcm.2016.02.012

35. Soleimani-Sefat, E, Parandeh, A, Rahmati, F, and Kamalikhah, T. Development and psychometric properties of the mental health literacy questionnaire (MHLQ) among young Iranian soldiers. Int J Prev Med . (2022) 13:101. doi: 10.4103/ijpvm.IJPVM_468_20

36. Tehrani, H, Nejatian, M, Moshki, M, and Jafari, A. Psychometric properties of Persian version of depression literacy (D-lit) questionnaire among general population. Int J Ment Heal Syst . (2022) 16:1–11. doi: 10.1186/s13033-022-00550-x

37. Griffiths, KM, Walker, J, and Batterham, PJ. Help seeking for social anxiety: a pilot randomised controlled trial. Digital Health . (2017) 3:205520761771204. doi: 10.1177/2055207617712047

38. Calear, AL, Batterham, PJ, Torok, M, and McCallum, S. Help-seeking attitudes and intentions for generalised anxiety disorder in adolescents: the role of anxiety literacy and stigma. Eur Child Adolesc Psychiatry . (2021) 30:243–51. doi: 10.1007/s00787-020-01512-9

39. Hanlon, HR. Perceptions of clinical anxiety disorders among adolescents in Ireland: An exploration of anxiety literacy, stigma and help-giving intentions . Ireland: Trinity College Dublin, School of Psychology, Discipline of Psychology (2021).

40. Tay, JL, Goh, YSS, Sim, K, and Klainin-Yobas, P. Impact of the HOPE intervention on mental health literacy, psychological well-being and stress levels amongst university undergraduates: a randomised controlled trial. Int J Environ Res Public Health . (2022) 19:9772. doi: 10.3390/ijerph19159772

41. Kantaş Yılmaz, F, and Ünkür, P. Determination of mental health literacy, help-seeking Behaviours and psychological distress levels of health science students. J Health Lit . (2023) 7:9–20. doi: 10.22038/jhl.2022.66098.1308

42. Hanlon, HR, and Swords, L. Overthinkers, attention-seekers and wallflowers: peer perceptions of clinical anxiety disorders in adolescence. J Public Ment Health . (2019) 18:4–13. doi: 10.1108/JPMH-07-2018-0049

43. Mahmoodi, SMH, Ahmadzad-Asl, M, Eslami, M, Abdi, M, Kahnamoui, YH, and Rasoulian, M. Mental health literacy and mental health information-seeking behavior in Iranian university students. Front Psychiatry . (2022) 13:893534. doi: 10.3389/fpsyt.2022.893534

44. Organization WH. Cardiovascular diseases 2024. Available at: https://www.who.int/news-room/fact-sheets/detail/cardiovascular-diseases-(cvds) .

45. Sarrafzadegan, N, and Mohammmadifard, N. Cardiovascular disease in Iran in the last 40 years: prevalence, mortality, morbidity, challenges and strategies for cardiovascular prevention. Arch Iran Med . (2019) 22:204–10.

PubMed Abstract | Google Scholar

46. Celano, CM, Daunis, DJ, Lokko, HN, Campbell, KA, and Huffman, JC. Anxiety disorders and cardiovascular disease. Curr Psychiatry Rep . (2016) 18:1–11. doi: 10.1007/s11920-016-0739-5

47. Anderson, RJ, Grigsby, AB, Freedland, KE, De Groot, M, McGill, JB, Clouse, RE, et al. Anxiety and poor glycemic control: a meta-analytic review of the literature. Int J Psychiatry Med . (2002) 32:235–47. doi: 10.2190/KLGD-4H8D-4RYL-TWQ8

48. Yan, J, Pan, Y, Cai, W, Cheng, Q, Dong, W, and An, T. Association between anxiety and hypertension: a systematic review and meta-analysis of epidemiological studies. Neuropsychiatr Dis Treat . (2015) 11:1121–230. doi: 10.2147/NDT.S77710

49. Lambiase, MJ, Kubzansky, LD, and Thurston, RC. Prospective study of anxiety and incident stroke. Stroke . (2014) 45:438–43. doi: 10.1161/STROKEAHA.113.003741

50. Mal, K, Awan, ID, Ram, J, Shaukat, F, and Awan, I. Depression and anxiety as a risk factor for myocardial infarction. Cureus . (2019) 11:e6064. doi: 10.7759/cureus.6064

Keywords: validity, reliability, health literacy, mental health, psychometric

Citation: Jafari A, Moshki M, Mokhtari AM, Naddafi F and Nejatian M (2024) Validity and reliability of anxiety literacy (A-Lit) and its relationship with demographic variables in the Iranian general population. Front. Public Health . 12:1359146. doi: 10.3389/fpubh.2024.1359146

Received: 20 December 2023; Accepted: 25 March 2024; Published: 17 April 2024.

Reviewed by:

Copyright © 2024 Jafari, Moshki, Mokhtari, Naddafi and Nejatian. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Mahbobeh Nejatian, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

This paper is in the following e-collection/theme issue:

Published on 17.4.2024 in Vol 26 (2024)

This is a member publication of National University of Singapore

Comparing Open-Access Database and Traditional Intensive Care Studies Using Machine Learning: Bibliometric Analysis Study

Authors of this article:

Original Paper

• Yuhe Ke 1 * , MBBS   ; 
• Rui Yang 2 * , MSc   ; 
• Nan Liu 2 , PhD  

1 Division of Anesthesiology and Perioperative Medicine, Singapore General Hospital, Singapore, Singapore

2 Centre for Quantitative Medicine, Duke-NUS Medical School, National University of Singapore, Singapore, Singapore

*these authors contributed equally

Corresponding Author:

Nan Liu, PhD

Centre for Quantitative Medicine

Duke-NUS Medical School

National University of Singapore

8 College Road

Singapore, 169857

Phone: 65 66016503

Email: [email protected]

Background: Intensive care research has predominantly relied on conventional methods like randomized controlled trials. However, the increasing popularity of open-access, free databases in the past decade has opened new avenues for research, offering fresh insights. Leveraging machine learning (ML) techniques enables the analysis of trends in a vast number of studies.

Objective: This study aims to conduct a comprehensive bibliometric analysis using ML to compare trends and research topics in traditional intensive care unit (ICU) studies and those done with open-access databases (OADs).

Methods: We used ML for the analysis of publications in the Web of Science database in this study. Articles were categorized into “OAD” and “traditional intensive care” (TIC) studies. OAD studies were included in the Medical Information Mart for Intensive Care (MIMIC), eICU Collaborative Research Database (eICU-CRD), Amsterdam University Medical Centers Database (AmsterdamUMCdb), High Time Resolution ICU Dataset (HiRID), and Pediatric Intensive Care database. TIC studies included all other intensive care studies. Uniform manifold approximation and projection was used to visualize the corpus distribution. The BERTopic technique was used to generate 30 topic-unique identification numbers and to categorize topics into 22 topic families.

Results: A total of 227,893 records were extracted. After exclusions, 145,426 articles were identified as TIC and 1301 articles as OAD studies. TIC studies experienced exponential growth over the last 2 decades, culminating in a peak of 16,378 articles in 2021, while OAD studies demonstrated a consistent upsurge since 2018. Sepsis, ventilation-related research, and pediatric intensive care were the most frequently discussed topics. TIC studies exhibited broader coverage than OAD studies, suggesting a more extensive research scope.

Conclusions: This study analyzed ICU research, providing valuable insights from a large number of publications. OAD studies complement TIC studies, focusing on predictive modeling, while TIC studies capture essential qualitative information. Integrating both approaches in a complementary manner is the future direction for ICU research. Additionally, natural language processing techniques offer a transformative alternative for literature review and bibliometric analysis.

Introduction

The start of critical care as a medical subspecialty can be traced back to a polio epidemic during which a substantial number of patients needed prolonged mechanical ventilation [ 1 ]. Over time, the field of critical care has experienced significant growth and continual evolution. Research in this field has played a pivotal role in unraveling the complexities of numerous diseases and treatment modalities, driving substantial advancements in clinical practice over the past decades [ 2 ]. Groundbreaking studies have investigated critical areas such as sepsis, mechanical ventilation, acute lung and kidney injuries, intensive care unit (ICU) delirium, and sedation in critically ill patients [ 3 ].

These research studies have often been conducted in traditional ways such as prospective and randomized controlled trials [ 4 ], cohort and observational studies, clinical trials [ 5 ], and clinical and translational research [ 6 ]. These traditional methods have revolutionized patient care and improved outcomes significantly. For instance, the implementation of protocol-driven, goal-directed management of sepsis and appropriate fluid therapy has led to remarkable reductions in mortality rates [ 7 , 8 ], and these findings have been integral in developing evidence-based practice guidelines that are now the gold standard [ 9 , 10 ].

Despite their undeniable merits, traditional research methods in intensive care also come with several limitations [ 11 ]. Clinical trials are known for their high costs [ 12 ], stringent standardization requirements, and ethical oversight [ 13 ]. Data collection can be laborious, prone to human errors, and constrained in terms of quantity and granularity [ 14 ]. Moreover, obtaining patient consent for most randomized controlled trials in the ICU poses challenges [ 15 ], necessitating alternative consent models. These limitations have become increasingly apparent as medical complexity continues to grow exponentially [ 16 ].

The advent of electronic health records (EHRs) has heralded a new era in clinical research by facilitating the digitization of health care systems [ 17 ]. In this era of data science, a more integrated approach can be adopted, using machine learning (ML) algorithms to tackle the complexity of critical illness [ 18 ]. Open-access databases (OADs), such as the Medical Information Mart for Intensive Care (MIMIC) database [ 19 ] and the Philips eICU Collaborative Research Database (eICU-CRD) [ 20 ], have played a transformative role by enabling free data sharing.

The concept of free and open databases plays a pivotal role in promoting data sharing and advancing medical knowledge in accordance with the findable, accessible, interoperable, and reusable (FAIR) guiding principle. The FAIR principles, which emphasize that data should be findable, accessible, interoperable, and reusable, are essential for fostering a collaborative and transparent scientific research environment [ 21 , 22 ]. By removing barriers to access, free, and open databases allow researchers, regardless of their affiliations or resources, to contribute to and benefit from the collective pool of information. Accessibility fosters inclusivity and diversity in research, promoting a broader range of perspectives and approaches to medical challenges. This democratization of knowledge leads to a more equitable distribution of information. Researchers can now leverage these vast repositories of information for ML and artificial intelligence studies, marking a departure from traditional intensive care (TIC) research approaches.

Conducting a literature review [ 23 ] to investigate the disparities between traditional ICU research and studies based on open-access data sets holds significant importance as it provides a comprehensive understanding of the strengths and limitations of the latter. However, conventional methods of literature reviews and bibliometric analysis have their limitations, especially when dealing with large-scale literature due to computational complexity and the labor-intensive nature of manual interpretations [ 24 - 26 ]. To address these challenges, natural language processing (NLP) offers a promising avenue, while topic modeling techniques can be used to extract various topic themes from extensive data sets [ 27 , 28 ].

Built on the foundations of bidirectional encoder representations from transformers (BERT), BERTopic introduces a novel approach to topic modeling [ 29 , 30 ]. Unlike traditional unsupervised models like latent Dirichlet allocation, which rely on “bag-of-words” model [ 31 ], BERTopic overcomes the problem of semantic information loss, significantly enhancing the accuracy of generated topics, and providing more interpretable compositions for each topic, which greatly facilitates the classification of topics.

With the aid of BERTopic, this study aims to shed light on the disparities and commonalities between studies conducted through OADs and TIC research. By analyzing the overall trends and patterns in these 2 groups, we seek to identify knowledge gaps and explore avenues for complementary contributions between these research approaches.

Data Filtering

We performed an ML-based analysis of research abstracts in the Web of Science (WoS) database to automatically categorize the research papers to conduct this literature mapping analysis. There was no limit to the year of publication of the articles. The search query consisted of the following keyword to identify all the studies that were published under the umbrella of intensive care: (“ICU” OR “intensive care”). The search terms were deliberately left to be broad to cover broad spectrums of journals in the field.

The inclusion criteria were as follows: (1) written in English, (2) articles that had keywords related to intensive care, (3) articles that had the article type of “article” or “review.” We excluded articles with incomplete data fields (eg, title, abstract, publication year, and paper citation). The articles included were then further processed to identify if they were studies using OADs. These articles were labeled as “open-access database,” while the rest of the articles extracted were labeled as “traditional intensive care.”

The search used for this study was performed on January 18, 2023, from WoS. This generated 227,893 search results, which were subsequently reselected using Python. An advanced search from PubMed was done based on the broad search terms of ICU studies used from previous Cochrane ICU literature review [ 32 ] to ensure the accuracy of the results. The numbers corroborated with a discrepancy of 4.9% (227,893 WoS keyword search vs 239,748 PubMed ICU keyword search).

Selection Criteria for OADs

A title search using keywords from all currently existing OADs was conducted to identify OAD studies. These include (1) MIMIC [ 19 ], (2) eICU-CRD [ 20 ], (3) Amsterdam University Medical Centers Database (AmsterdamUMCdb) [ 33 ], (4) High Time Resolution ICU Dataset (HiRID) [ 34 ], and (5) Pediatric Intensive Care database [ 35 ]. We avoided including only keywords in the search and restricted the search years by the year that the OAD was made publicly available to reduce the inadvertent inclusion of incorrect articles due to keywords. For instance, the search term for OADs published with the MIMIC database included title keyword search with the terms (“MIMIC-IV” OR “MIMIC-III” OR “MIMIC-II” OR “MIMIC Dataset” OR “medical information mart for intensive care” OR “MIMIC IV” OR “MIMIC III” OR “MIMIC II”) in studies that were published after 2003. The title keyword search for the searches and the year of cutoff for each OAD are presented in Multimedia Appendix 1 .

Furthermore, to ensure the accuracy of the supervised keyword classification, a manual review of the classification by 2 critical care physicians was done for 100 articles from each category that were randomly selected. The review was done independently with the physicians labeling the extract publications into OAD and TICs. An accuracy of 99% was achieved on independent reviews, and full agreement was achieved after discussion on the discrepancy. The final results were matched with the supervised keyword classification.

We performed a bibliometric analysis by directly extracting publication details from the WoS database using Python (Python Software Foundation). The analysis involved assessing the number of articles published per year, calculating total citation counts, and identifying the top journals that published intensive care-related articles. Comprehensive results are presented in Multimedia Appendix 2 .

Data Analysis

Uniform manifold approximation and projection.

Uniform manifold approximation and projection (UMAP) is a manifold learning technique for dimension reduction, which can identify key structures in high-dimensional data space and map them to low-dimensional space to accomplish dimensionality reduction. Compared to other dimensionality reduction algorithms, such as principal component analysis [ 36 ], UMAP can retain more global features [ 37 ]. In this paper, we constructed a corpus consisting of abstract words from all studies. However, due to the massive size of the corpus, visualizing and analyzing the high-dimensional data to explore the differences in the vocabulary patterns between the OAD and TIC studies is a challenge. The UMAP package in Python, which implements the UMAP algorithm, was used to project the high-dimensional corpus to 4 dimensions. By cross plotting each dimension, we were able to investigate underlying differences in corpus distribution between OAD and TIC studies.

Topic modeling can help us explore the similarities and differences between research topics in OAD and TIC studies. Unlike conventional topic modeling models, BERTopic uses the BERT framework for embeddings, enabling a deeper understanding of semantic relationships [ 30 ]. The BERTopic model was implemented by the BERTopic package in Python and divided 146,727 studies into 30 topic IDs. We also performed latent Dirichlet allocation topic modeling through Python’s LdaModel package for comparison. Through the review of topic keywords by 2 critical care physicians, BERTopic exhibited superior accuracy and sophistication in topic identification, with enhanced interpretability and scientific rigor.

Consequently, the BERTopic model was used for the final analysis. Each of these topics was given a corresponding clinical research category. The overlapping categories were merged into topic families for easier comparisons. By using these advanced techniques, we were able to uncover hidden patterns and relationships within the literature and provide insights into the current state of intensive care research.

A total of 227,893 records were identified from the WoS database on January 18, 2023, of which 195,463 full records were subsequently processed. Records were excluded if they are not “article” or “review” or if they do not contain keywords related to intensive care. After exclusions, 145,426 articles were identified as TIC studies and 1301 articles were categorized as OAD ( Figure 1 ).

We examined the number of articles published per year to analyze the trends in TIC and OAD studies ( Figure 2 ). Over the past 2 decades, TIC studies have experienced exponential growth, culminating in a peak of 16,378 articles in 2021. A subsequent decline in the number of publications occurred in 2022, likely attributable to delayed indexing within the WoS database and a reduction in COVID-19–related studies as the pandemic stabilized [ 38 ]. In contrast, the first OAD study emerged in 2003, with its popularity experiencing a consistent upsurge since 2018. Nonetheless, the number of OAD publications remains markedly lower in comparison to TIC publications.

The OAD studies were published most frequently in new open-access journals such as Frontiers in Medicine , Frontiers in Cardiovascular Medicine , and Scientific Reports while the TIC studies were published most frequently in established journals like Critical Care Medicine , Intensive Care Medicine , and Critical Care ( Multimedia Appendix 2 ). Further analysis of keywords from the abstracts showed 2.4% (3492/145,426) TIC studies were meta-analyses or systematic reviews, while only 0.08% (1/1301) OAD study was in this category. There were 5.61% (73/1301) OAD studies, and 7.43% (10,799/145,426) TIC studies that had the keyword of “cost.” Examples of the data fields that are available within OADs such as MIMIC and eICU-CRD are listed in Textbox 1 . Some information fields such as end-of-life goals and values and health care provider psychology are not available within the current EHRs extracted for OADs.

Examples of information that is available in current OADs

• Patient information: demographics and social set-up
• Hospital context: admission time and discharge time, intensive care unit (ICU) and hospital admissions, and pre-ICU admission
• Diagnosis: physician-curated ICU diagnosis and data-driven phenotypes
• Intervention: medications, procedures, and organ support
• Diagnostics: blood test, microbiology, and scans
• Clinical texts: clinical notes and diagnostic reports
• Physiological monitoring: basic monitoring and waveforms

Examples of information that is not readily available in current OADs

• Patient information: family set up and visiting, financial information, and special populations
• Hospital context: post-ICU discharge details, delayed admission or discharge, and health personnel psychology
• Diagnosis: pre-ICU history and diagnosis requiring clinical symptoms
• Intervention: indications for interventions, complications, and intraoperative and postoperative
• Diagnostics: pathology photographs, imaging, and molecular or genetic studies
• Clinical texts: patient narratives, end-of-life goals and patient value, and health personnel behavior
• Physiological monitoring: advanced monitoring

The UMAP algorithm was used to project the high-dimension corpus to 4 dimensions and allowed exploration of the vocabulary patterns between the OAD and TIC studies ( Figure 3 ). The projection values are represented by the x-axis, while the densities are represented by the y-axis. The presence of considerable overlap between TIC studies and OAD studies suggests that they share a substantial number of common terminologies, which may correspond to similar research topics. Nonetheless, TIC studies exhibit a more extensive coverage than OAD studies, which may stem from broader research scope and extended research duration.

Subsequently, the BERTopic model was then used to generate 30 topic IDs ( Figure 4 ). The internal commonalities of each topic ID were reviewed by critical care physicians and assigned a specific subtopic in intensive care research. The model was able to automatically classify the topics with high interpretability and the topic components were interpreted with ease. For instance, components in topic ID 5 consist of, in decreasing order of weightage: “learning,” “model,” “machine,” “machine learning,” “models,” “data,” “prediction,” and “performance.” This topic was consequently labeled “predictive model” (topic ID 5 in Multimedia Appendix 3 ).

The overall topic distribution in TIC studies was more uniform, while the OAD studies tended to be concentrated on several topics including topic ID 2 (kidney injury), 5 (predictive model), and 13 (sepsis). Some topics that were missing in OAD studies included 6 (pediatrics care), 21 (viral infections), 23 (health personnel and psychology), and 28 (nutrition and rehabilitation).

The similarity matrix shows that there was little overlap between the topics ( Multimedia Appendix 4 ). To facilitate the interpretability of the categories, the overlapping topic IDs were merged to form the final 22 topic families ( Multimedia Appendix 3 ).

Topics such as “healthcare associated infection,” “thoracic surgeries,” and “pregnancy related” research were among the more frequently discussed 15 topics in TIC studies but have limited publications in OAD studies. The topics of “predictive model,” “obesity,” and “fungal infections” were popular in OAD studies but not the TIC studies. Overall, the topic distributions of the TIC studies were distributed more evenly with the topic family of sepsis accounting for a quarter of the studies, while publications in the OAD studies were heavily skewed toward the predictive model (>40%) and sepsis (>30%; Figure 5 ).

Principal Results

This study conducted a comprehensive review and bibliometric analysis of OAD and TIC studies. NLP was used to facilitate this large-scale literature review. Studies using OADs mainly concentrated on a few topics, such as predictive modeling, while TIC studies covered a wider range of topics with a more balanced distribution.

Advantages of OAD Studies

OAD studies offer several advantages that have contributed to their increasing popularity in intensive care research. The granularity of data and easy access to large-cohort databases, such as MIMIC [ 39 ], has enabled researchers to perform predictive modeling and conduct various secondary analyses efficiently [ 40 , 41 ]. This accessibility has provided valuable opportunities for exploring specific aspects of patient care, evident in studies investigating phenomena like “weekend effects” and circadian rhythms in ICU patients before discharge [ 42 - 46 ]. The vast amount of longitudinal and time series data available in OADs has also facilitated the implementation of complex ML and deep learning methods [ 47 ].

Limitations of OAD Studies

However, it is crucial to acknowledge the retrospective nature of OAD data, which inherently limits the assessment of confounding factors and the ability to draw strong causal conclusions. The observational design of OAD studies may result in lower-quality evidence according to the GRADE (Grading of Recommendations, Assessment, Development, and Evaluations) framework [ 48 , 49 ], and thus, the research from OAD studies has yet to be fully integrated into existing evidence-based guidelines, as exemplified by the omission of OAD studies in the 2021 sepsis guidelines [ 50 ]. Nevertheless, OADs remain a valuable resource for supplementing and complementing TIC studies, providing unique insights and enhanced predictive scores for intensive care settings.

Furthermore, approximately 50% of the studies using OADs published focused on predictive modeling. The increased usage of ML methods in predictive modeling has not been without critique. Some medical prediction problems inherently possess linear characteristics, and the selection of features may predominantly focus on already known strong predictors, leading to limited improvements in prediction accuracy with ML [ 51 ]. Additionally, interstudy heterogeneity poses a challenge in comparing results obtained from different ML models applied to the same data sets [ 52 ]. The ethical implications of relying solely on ML models to make high-risk health care decisions instead of involving clinical expertise are also relevant considerations [ 51 , 53 ].

While OADs provide comprehensive patient data, there are certain limitations in their ability to capture specific information essential for certain critical care research areas. Notably, data fields related to qualitative aspects such as ethics and end-of-life care [ 54 , 55 ], and health care personnel psychology [ 56 ] may be challenging, if not impossible, to obtain through OADs generated from EHRs. Consequently, TIC studies have played a pivotal role in addressing these limitations by capturing critical information that is integral to understanding ethical considerations, patient experiences, and health care provider psychology in intensive care [ 57 , 58 ].

Synergy Between OAD and TIC Studies

The synergy between OAD and TIC studies is a promising approach to enhance the comprehensiveness and robustness of intensive care research. OADs, with their large cohort sizes, can serve as external validation cohorts for ML models developed from TIC studies, potentially reducing the sample sizes required for prospective research. Furthermore, OAD studies can corroborate the results of TIC studies, benefiting from larger sample sizes and real-world data, thus providing more practical insights for implementation in intensive care settings [ 43 ]. The integration of OAD and TIC studies presents an opportunity to bridge the gaps in data availability and research methodologies, ultimately enriching the understanding and practice of critical care medicine.

Potential Impact of NLP

The usage of large language models such as BERTopic has proven to be a valuable tool for large-scale literature review and topic extraction [ 58 ]. This approach has enabled accurate, reliable, and granular topic generation, offering clinicians a more effective means of interpreting data compared to traditional bag-of-words models [ 59 ]. The potential of NLP to analyze scientific articles and identify trends and knowledge gaps holds promise for shaping the future of research in critical care medicine. As the volume of publications in critical care continues to grow and large language modeling continues to advance in health care [ 60 ], AI technology will be crucial in efficiently identifying and predicting emerging trends.

Future Directions

Future research in the field of critical care can explore novel applications of ML beyond predictive modeling. For instance, using ML to study patterns in how papers are cited, shared, and discussed on the web could help predict their potential impact on the scientific community. This analysis can aid in identifying highly influential papers and understanding the factors that contribute to their recognition. Additionally, investigations into methods for enhancing the interpretability and transparency of ML algorithms in critical care research would further facilitate the ethical and responsible use of AI technologies.

Strengths and Limitations

The study’s application of NLP-driven in analyzing scientific articles and identifying trends highlights the potential impact of AI technologies in streamlining literature reviews and identifying emerging trends more efficiently.

Another notable strength of this study is the usage of the WoS database, the world’s oldest and most extensively used repository of research publications and citations, encompassing approximately 34,000 journals [ 61 ]. The comprehensiveness of this database provides a robust representation of the literature in the field of intensive care research. Nevertheless, it is essential to acknowledge that some articles published in nonindexed journals might not have been captured, and future studies could benefit from considering additional databases to supplement our findings.

One other limitation lies in the classification of OAD and TIC studies, which may be subject to variations in the interpretation of keywords. However, we optimized the keyword combinations during the search process in the WoS database and implemented Python filtering techniques, resulting in a relatively high level of accuracy in our classifications. The number of studies was further corroborated with a manual search on PubMed and a review of the classifications of the studies was done by critical care physicians.

Although there were no specific language restrictions, the nature of the search term being in English inadvertently excluded valuable contributions from non-English research. This may potentially limit the generalizability of our findings to a broader international audience. In future investigations, the inclusion of articles from various languages could offer a more comprehensive and diverse perspective on intensive care research.

Conclusions

This study has provided valuable insights into the expanding landscape of intensive care research through a comprehensive bibliometric analysis of a large number of publications by leveraging NLP technologies. While OAD studies have demonstrated significant promise, it is essential to view them as a complementary approach rather than a replacement for TIC studies. The unique strength of TIC studies lies in their ability to capture crucial qualitative information, which is essential for comprehensive and ethical decision-making. The integration of both OAD and TIC studies offers a synergistic approach to enriching our understanding of critical care medicine and advancing patient care outcomes. As NLP technology continues to advance, it holds the potential to offer a feasible and transformative alternative for literature review and bibliometric analysis.

Acknowledgments

We thank Dr Nicholas Brian Shannon for assistance with the manual review of the supervised keyword classification. This work was supported by the Duke-NUS Signature Research Programme, funded by the Ministry of Health, Singapore.

Data Availability

The data sets generated during and/or analyzed during this study are available from the corresponding author on reasonable request. The complete set of code used in this study is readily available for download on GitHub [ 62 ].

Authors' Contributions

YK and NL played key roles in the conceptualization of the project. RY was responsible for formalizing the methodology and conducting data curation with the advisory of YK. YK contributed to the validation of the data, ensuring its relevance to the research objectives. RY took the lead in visualizing the data. Both YK and RY drafted the original manuscript. NL served as the project supervisor, overseeing the implementation, and providing valuable input in the writing, review, and editing phases.

Conflicts of Interest

None declared.

Search terms for open-access database (OAD) studies with the cutoff by the years of publications.

Top 20 journals ranked by total citation in which the open-access database and traditional intensive care studies were published. The average citation per article was obtained with the total citation/total number of articles. The citation counts were obtained from Web of Science.

Topic ID and topic family and the components and weightage in each of the categories.

Similarity matrix of 30 topics.

• Kelly FE, Fong K, Hirsch N, Nolan JP. Intensive care medicine is 60 years old: the history and future of the intensive care unit. Clin Med (Lond). 2014;14(4):376-379. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cook D, Brower R, Cooper J, Brochard L, Vincent JL. Multicenter clinical research in adult critical care. Crit Care Med. 2002;30(7):1636-1643. [ CrossRef ] [ Medline ]
• Rosenberg AL, Tripathi RS, Blum J. The most influential articles in critical care medicine. J Crit Care. 2010;25(1):157-170. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Granholm A, Alhazzani W, Derde LPG, Angus DC, Zampieri FG, Hammond NE, et al. Randomised clinical trials in critical care: past, present and future. Intensive Care Med. 2022;48(2):164-178. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Markey KA, Ottridge R, Mitchell JL, Rick C, Woolley R, Ives N, et al. Assessing the efficacy and safety of an 11β-hydroxysteroid dehydrogenase type 1 inhibitor (AZD4017) in the idiopathic intracranial hypertension drug trial, IIH:DT: clinical methods and design for a phase II randomized controlled trial. JMIR Res Protoc. 2017;6(9):e181. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Verdonk F, Feyaerts D, Badenes R, Bastarache JA, Bouglé A, Ely W, et al. Upcoming and urgent challenges in critical care research based on COVID-19 pandemic experience. Anaesth Crit Care Pain Med. 2022;41(5):101121. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gurnani PK, Patel GP, Crank CW, Vais D, Lateef O, Akimov S, et al. Impact of the implementation of a sepsis protocol for the management of fluid-refractory septic shock: a single-center, before-and-after study. Clin Ther. 2010;32(7):1285-1293. [ CrossRef ] [ Medline ]
• Wang JL, Chin CS, Chang MC, Yi CY, Shih SJ, Hsu JY, et al. Key process indicators of mortality in the implementation of protocol-driven therapy for severe sepsis. J Formos Med Assoc. 2009;108(10):778-787. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Levy MM, Evans LE, Rhodes A. The surviving sepsis campaign bundle: 2018 update. Crit Care Med. 2018;46(6):997-1000. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fan E, Del Sorbo L, Goligher EC, Hodgson CL, Munshi L, Walkey AJ, et al. An official American Thoracic Society/European Society of Intensive Care Medicine/Society of Critical Care Medicine clinical practice guideline: mechanical ventilation in adult patients with acute respiratory distress syndrome. Am J Respir Crit Care Med. 2017;195(9):1253-1263. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Goldfrad C, Vella K, Bion JF, Rowan KM, Black NA. Research priorities in critical care medicine in the UK. Intensive Care Med. 2000;26(10):1480-1488. [ CrossRef ] [ Medline ]
• Moore TJ, Heyward J, Anderson G, Alexander GC. Variation in the estimated costs of pivotal clinical benefit trials supporting the US approval of new therapeutic agents, 2015-2017: a cross-sectional study. BMJ Open. 2020;10(6):e038863. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Umscheid CA, Margolis DJ, Grossman CE. Key concepts of clinical trials: a narrative review. Postgrad Med. 2011;123(5):194-204. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Maré IA, Kramer B, Hazelhurst S, Nhlapho MD, Zent R, Harris PA, et al. Electronic data capture system (REDCap) for health care research and training in a resource-constrained environment: technology adoption case study. JMIR Med Inform. 2022;10(8):e33402. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O'Hearn K, Gibson J, Krewulak K, Porteous R, Saigle V, Sampson M, et al. Consent models in Canadian critical care randomized controlled trials: a scoping review. Can J Anaesth. 2022;69(4):513-526. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ghassemi M, Celi LA, Stone DJ. State of the art review: the data revolution in critical care. Crit Care. 2015;19(1):118. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood). 2014;33(7):1123-1131. [ CrossRef ] [ Medline ]
• Mlodzinski E, Wardi G, Viglione C, Nemati S, Crotty Alexander L, Malhotra A. Assessing barriers to implementation of machine learning and artificial intelligence-based tools in critical care: web-based survey study. JMIR Perioper Med. 2023;6:e41056. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Johnson AEW, Bulgarelli L, Shen L, Gayles A, Shammout A, Horng S, et al. MIMIC-IV, a freely accessible electronic health record dataset. Sci Data. 2023;10(1):1. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pollard TJ, Johnson AEW, Raffa JD, Celi LA, Mark RG, Badawi O. The eICU collaborative research database, a freely available multi-center database for critical care research. Sci Data. 2018;5:180178. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Inau ET, Sack J, Waltemath D, Zeleke AA. Initiatives, concepts, and implementation practices of FAIR (findable, accessible, interoperable, and reusable) data principles in health data stewardship practice: protocol for a scoping review. JMIR Res Protoc. 2021;10(2):e22505. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wilkinson MD, Dumontier M, Aalbersberg IJJ, Appleton G, Axton M, Baak A, et al. The FAIR guiding principles for scientific data management and stewardship. Sci Data. 2016;3:160018. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bahadoran Z, Mirmiran P, Kashfi K, Ghasemi A. Importance of systematic reviews and meta-analyses of animal studies: challenges for animal-to-human translation. J Am Assoc Lab Anim Sci. 2020;59(5):469-477. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Haidich AB. Meta-analysis in medical research. Hippokratia. 2010;14(Suppl 1):29-37. [ FREE Full text ] [ Medline ]
• Thompson DF, Walker CK. A descriptive and historical review of bibliometrics with applications to medical sciences. Pharmacotherapy. 2015;35(6):551-559. [ CrossRef ] [ Medline ]
• Donthu N, Kumar S, Mukherjee D, Pandey N, Lim WM. How to conduct a bibliometric analysis: an overview and guidelines. J Bus Res. 2021;133:285-296. [ FREE Full text ] [ CrossRef ]
• Zhao W, Chen JJ, Perkins R, Liu Z, Ge W, Ding Y, et al. A heuristic approach to determine an appropriate number of topics in topic modeling. BMC Bioinformatics. 2015;16(Suppl 13):S8. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Doanvo A, Qian X, Ramjee D, Piontkivska H, Desai A, Majumder M. Machine learning maps research needs in COVID-19 literature. Patterns (N Y). 2020;1(9):100123. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Devlin J, Chang MW, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. ArXiv. Preprint posted online on May 24 2019. 2018. [ CrossRef ]
• Grootendorst M. BERTopic: neural topic modeling with a class-based TF-IDF procedure. ArXiv. Preprint posted online on March 11 2022. 2022. [ CrossRef ]
• Blei DM, Ng AY, Jordan MI. Latent Dirichlet allocation. J Mach Learn Res. 2003;3:993-1022. [ FREE Full text ]
• Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, Doig CJ. A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care. 2005;9(5):R575-R582. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Thoral PJ, Peppink JM, Driessen RH, Sijbrands EJG, Kompanje EJO, Kaplan L, et al. Sharing ICU patient data responsibly under the Society of Critical Care Medicine/European Society of Intensive Care Medicine joint data science collaboration: the Amsterdam university medical centers database (AmsterdamUMCdb) example. Crit Care Med. 2021;49(6):e563-e577. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Faltys M, Zimmermann M, Lyu X, Hüser M, Hyland S, Rätsch G, et al. HiRID, a high time-resolution ICU dataset. PhysioNet. 2021. URL: https://physionet.org/content/hirid/1.1.1/ [accessed 2024-04-02]
• Zeng X, Yu G, Lu Y, Tan L, Wu X, Shi S, et al. PIC, a paediatric-specific intensive care database. Sci Data. 2020;7(1):14. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Maćkiewicz A, Ratajczak W. Principal components analysis (PCA). Comput Geosci. 1993;19(3):303-342. [ CrossRef ]
• McInnes L, Healy J, Melville J. UMAP: uniform manifold approximation and projection for dimension reduction. ArXiv. Preprint posted online on September 18 2020. 2018. [ CrossRef ]
• Murray CJL. COVID-19 will continue but the end of the pandemic is near. Lancet. 2022;399(10323):417-419. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mark R. The story of MIMIC. In: Secondary Analysis of Electronic Health Records. Cham, Switzerland. Springer International Publishing; 2016;43-49.
• Alghatani K, Ammar N, Rezgui A, Shaban-Nejad A. Predicting intensive care unit length of stay and mortality using patient vital signs: machine learning model development and validation. JMIR Med Inform. 2021;9(5):e21347. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Liu D, Zheng M, Sepulveda NA. Using artificial neural network condensation to facilitate adaptation of machine learning in medical settings by reducing computational burden: model design and evaluation study. JMIR Form Res. 2021;5(12):e20767. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhang Z, Ho KM, Hong Y. Machine learning for the prediction of volume responsiveness in patients with oliguric acute kidney injury in critical care. Crit Care. 2019;23(1):112. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Meyer A, Zverinski D, Pfahringer B, Kempfert J, Kuehne T, Sündermann SH, et al. Machine learning for real-time prediction of complications in critical care: a retrospective study. Lancet Respir Med. 2018;6(12):905-914. [ CrossRef ] [ Medline ]
• Chen H, Zhu Z, Zhao C, Guo Y, Chen D, Wei Y, et al. Central venous pressure measurement is associated with improved outcomes in septic patients: an analysis of the MIMIC-III database. Crit Care. 2020;24(1):433. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Faust L, Feldman K, Chawla NV. Examining the weekend effect across ICU performance metrics. Crit Care. 2019;23(1):207. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Davidson S, Villarroel M, Harford M, Finnegan E, Jorge J, Young D, et al. Vital-sign circadian rhythms in patients prior to discharge from an ICU: a retrospective observational analysis of routinely recorded physiological data. Crit Care. 2020;24(1):181. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Xie F, Yuan H, Ning Y, Ong MEH, Feng M, Hsu W, et al. Deep learning for temporal data representation in electronic health records: a systematic review of challenges and methodologies. J Biomed Inform. 2022;126:103980. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schünemann HJ, et al. GRADE Working Group. What is "quality of evidence" and why is it important to clinicians? BMJ. 2008;336(7651):995-998. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-926. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Evans L, Rhodes A, Alhazzani W, Antonelli M, Coopersmith CM, French C, et al. Surviving sepsis campaign: international guidelines for management of sepsis and septic shock 2021. Crit Care Med. 2021;49(11):e1063-e1143. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Volovici V, Syn NL, Ercole A, Zhao JJ, Liu N. Steps to avoid overuse and misuse of machine learning in clinical research. Nat Med. 2022;28(10):1996-1999. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fleuren LM, Klausch TLT, Zwager CL, Schoonmade LJ, Guo T, Roggeveen LF, et al. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med. 2020;46(3):383-400. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yoon CH, Torrance R, Scheinerman N. Machine learning in medicine: should the pursuit of enhanced interpretability be abandoned? J Med Ethics. 2022;48(9):581-585. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gillett GR. Intensive care unit research ethics and trials on unconscious patients. Anaesth Intensive Care. 2015;43(3):309-312. [ CrossRef ] [ Medline ]
• Aulisio MP, Chaitin E, Arnold RM. Ethics and palliative care consultation in the intensive care unit. Crit Care Clin. 2004;20(3):505-523, x-xi. [ CrossRef ] [ Medline ]
• Raudenská J, Steinerová V, Javůrková A, Urits I, Kaye AD, Viswanath O, et al. Occupational burnout syndrome and post-traumatic stress among healthcare professionals during the novel coronavirus disease 2019 (COVID-19) pandemic. Best Pract Res Clin Anaesthesiol. 2020;34(3):553-560. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Davidson JE, Jones C, Bienvenu OJ. Family response to critical illness: postintensive care syndrome-family. Crit Care Med. 2012;40(2):618-624. [ CrossRef ] [ Medline ]
• White DB, Angus DC, Shields AM, Buddadhumaruk P, Pidro C, Paner C, et al. A randomized trial of a family-support intervention in intensive care units. N Engl J Med. 2018;378(25):2365-2375. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Popoff B, Occhiali É, Grangé S, Bergis A, Carpentier D, Tamion F, et al. Trends in major intensive care medicine journals: a machine learning approach. J Crit Care. 2022;72:154163. [ CrossRef ] [ Medline ]
• Yang R, Tan TF, Lu W, Thirunavukarasu AJ, Ting DSW, Liu N. Large language models in health care: development, applications, and challenges. Health Care Sci. 2023;2(4):255-263. [ FREE Full text ] [ CrossRef ]
• Birkle C, Pendlebury DA, Schnell J, Adams J. Web of science as a data source for research on scientific and scholarly activity. Quant Sci Stud. 2020;1(1):363-376. [ FREE Full text ] [ CrossRef ]
• GitHub. URL: https://github.com/YangRui525/Comparing-OAD-and-TIC-Studies

Abbreviations

Edited by A Mavragani; submitted 19.04.23; peer-reviewed by D Chrimes, S Pesälä; comments to author 14.07.23; revised version received 01.08.23; accepted 14.01.24; published 17.04.24.

©Yuhe Ke, Rui Yang, Nan Liu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.