quantitative research about medical technology

• Research article
• Open access
• Published: 06 January 2021

Effects of the COVID-19 pandemic on medical students: a multicenter quantitative study

• Aaron J. Harries   ORCID: orcid.org/0000-0001-7107-0995 1 ,
• Carmen Lee 1 ,
• Lee Jones 2 ,
• Robert M. Rodriguez 1 ,
• John A. Davis 2 ,
• Megan Boysen-Osborn 3 ,
• Kathleen J. Kashima 4 ,
• N. Kevin Krane 5 ,
• Guenevere Rae 6 ,
• Nicholas Kman 7 ,
• Jodi M. Langsfeld 8 &
• Marianne Juarez 1  

BMC Medical Education volume  21 , Article number:  14 ( 2021 ) Cite this article

137k Accesses

155 Citations

37 Altmetric

Metrics details

The COVID-19 pandemic disrupted the United States (US) medical education system with the necessary, yet unprecedented Association of American Medical Colleges (AAMC) national recommendation to pause all student clinical rotations with in-person patient care. This study is a quantitative analysis investigating the educational and psychological effects of the pandemic on US medical students and their reactions to the AAMC recommendation in order to inform medical education policy.

The authors sent a cross-sectional survey via email to medical students in their clinical training years at six medical schools during the initial peak phase of the COVID-19 pandemic. Survey questions aimed to evaluate students’ perceptions of COVID-19’s impact on medical education; ethical obligations during a pandemic; infection risk; anxiety and burnout; willingness and needed preparations to return to clinical rotations.

Seven hundred forty-one (29.5%) students responded. Nearly all students (93.7%) were not involved in clinical rotations with in-person patient contact at the time the study was conducted. Reactions to being removed were mixed, with 75.8% feeling this was appropriate, 34.7% guilty, 33.5% disappointed, and 27.0% relieved.

Most students (74.7%) agreed the pandemic had significantly disrupted their medical education, and believed they should continue with normal clinical rotations during this pandemic (61.3%). When asked if they would accept the risk of infection with COVID-19 if they returned to the clinical setting, 83.4% agreed.

Students reported the pandemic had moderate effects on their stress and anxiety levels with 84.1% of respondents feeling at least somewhat anxious. Adequate personal protective equipment (PPE) (53.5%) was the most important factor to feel safe returning to clinical rotations, followed by adequate testing for infection (19.3%) and antibody testing (16.2%).

Conclusions

The COVID-19 pandemic disrupted the education of US medical students in their clinical training years. The majority of students wanted to return to clinical rotations and were willing to accept the risk of COVID-19 infection. Students were most concerned with having enough PPE if allowed to return to clinical activities.

Peer Review reports

The COVID-19 pandemic has tested the limits of healthcare systems and challenged conventional practices in medical education. The rapid evolution of the pandemic dictated that critical decisions regarding the training of medical students in the United States (US) be made expeditiously, without significant input or guidance from the students themselves. On March 17, 2020, for the first time in modern US history, the Association of American Medical Colleges (AAMC), the largest national governing body of US medical schools, released guidance recommending that medical students immediately pause all clinical rotations to allow time to obtain additional information about the risks of COVID-19 and prepare for safe participation in the future. This decisive action would also conserve scarce resources such as personal protective equipment (PPE) and testing kits; minimize exposure of healthcare workers (HCWs) and the general population; and protect students’ education and wellbeing [ 1 ].

A similar precedent was set outside of the US during the SARS-CoV1 epidemic in 2003, where an initial cluster of infection in medical students in Hong Kong resulted in students being removed from hospital systems where SARS surfaced, including Hong Kong, Singapore and Toronto [ 2 , 3 ]. Later, studies demonstrated that the exclusion of Canadian students from those clinical environments resulted in frustration at lost learning opportunities and students’ inability to help [ 3 ]. International evidence also suggests that medical students perceive an ethical obligation to participate in pandemic response, and are willing to participate in scenarios similar to the current COVID-19 crisis, even when they believe the risk of infection to themselves to be high [ 4 , 5 , 6 ].

The sudden removal of some US medical students from educational settings has occurred previously in the wake of local disasters, with significant academic and personal impacts. In 2005, it was estimated that one-third of medical students experienced some degree of depression or post-traumatic stress disorder (PTSD) after Hurricane Katrina resulted in the closure of Tulane University School of Medicine [ 7 ].

Prior to the current COVID-19 pandemic, we found no studies investigating the effects of pandemics on the US medical education system or its students. The limited pool of evidence on medical student perceptions comes from two earlier global coronavirus surges, SARS and MERS, and studies of student anxiety related to pandemics are also limited to non-US populations [ 3 , 8 , 9 ]. Given the unprecedented nature of the current COVID-19 pandemic, there is concern that students may be missing out on meaningful educational experiences and months of clinical training with unknown effects on their current well-being or professional trajectory [ 10 ].

Our study, conducted during the initial peak phase of the COVID-19 pandemic, reports students’ perceptions of COVID-19’s impact on: medical student education; ethical obligations during a pandemic; perceptions of infection risk; anxiety and burnout; willingness to return to clinical rotations; and needed preparations to return safely. This data may help inform policies regarding the roles of medical students in clinical training during the current pandemic and prepare for the possibility of future pandemics.

We conducted a cross-sectional survey during the initial peak phase of the COVID-19 pandemic in the United States, from 4/20/20 to 5/25/20, via email sent to all clinically rotating medical students at six US medical schools: University of California San Francisco School of Medicine (San Francisco, CA), University of California Irvine School of Medicine (Irvine, CA), Tulane University School of Medicine (New Orleans, LA), University of Illinois College of Medicine (Chicago, Peoria, Rockford, and Urbana, IL), Ohio State University College of Medicine (Columbus, OH), and Zucker School of Medicine at Hofstra/Northwell (Hempstead, NY). Traditional undergraduate medical education in the US comprises 4 years of medical school with 2 years of primarily pre-clinical classroom learning followed by 2 years of clinical training involving direct patient care. Study participants were defined as medical students involved in their clinical training years at whom the AAMC guidance statement was directed. Depending on the curricular schedule of each medical school, this included intended graduation class years of 2020 (graduating 4th year student), 2021 (rising 4th year student), and 2022 (rising 3rd year student), exclusive of planned time off. Participating schools were specifically chosen to represent a broad spectrum of students from different regions of the country (West, South, Midwest, East) with variable COVID-19 prevalence. We excluded medical students not yet involved in clinical rotations. This study was deemed exempt by the respective Institutional Review Boards.

We developed a survey instrument modeled after a survey used in a previously published peer reviewed study evaluating the effects of the COVID-19 pandemic on Emergency Physicians, which incorporated items from validated stress scales [ 11 ]. The survey was modified for use in medical students to assess perceptions of the following domains: perceived impact on medical student education; ethical beliefs surrounding obligations to participate clinically during the pandemic; perceptions of personal infection risk; anxiety and burnout related to the pandemic; willingness to return to clinical rotations; and preparation needed for students to feel safe in the clinical environment. Once created, the survey underwent an iterative process of input and review from our team of authors with experience in survey methodology and psychometric measures to allow for optimization of content and validity. We tested a pilot of our preliminary instrument on five medical students to ensure question clarity, and confirm completion of the survey in approximately 10 min. The final survey consisted of 29 Likert, yes/no, multiple choice, and free response questions. Both medical school deans and student class representatives distributed the survey via email, with three follow-up emails to increase response rates. Data was collected anonymously.

For example, to assess the impact on students’ anxiety, participants were asked, “How much has the COVID-19 pandemic affected your stress or anxiety levels?” using a unipolar 7-point scale (1 = not at all, 4 = somewhat, 7 = extremely). To assess willingness to return to clinical rotations, participants were asked to rate on a bipolar scale (1 = strongly disagree, 2 = disagree, 3 = somewhat disagree, 4 = neither disagree nor agree, 5 = somewhat agree, 6 = agree, and 7 = strongly agree) their agreement with the statement: “to the extent possible, medical students should continue with normal clinical rotations during this pandemic.” (Survey Instrument, Supplemental Table  1 ).

Survey data was managed using Qualtrics hosted by the University of California, San Francisco. For data analysis we used STATA v15.1 (Stata Corp, College Station, TX). We summarized respondent characteristics and key responses as raw counts, frequency percent, medians and interquartile ranges (IQR). For responses to bipolar questions, we combined positive responses (somewhat agree, agree, or strongly agree) into an agreement percentage. To compare differences in medians we used a signed rank test with p value < 0.05 to show statistical difference. In a secondary analysis we stratified data to compare questions within key domains amongst the following sub-groups: female versus male, graduation year, local community COVID-19 prevalence (high, medium, low), and students on clinical rotations with in-person patient care. This secondary analysis used a chi square test with p value < 0.05 to show statistical difference between sub-group agreement percentages.

Of 2511 students contacted, we received 741 responses (29.5% response rate). Of these, 63.9% of respondents were female and 35.1% were male, with 1.0% reporting a different gender identity; 27.7% of responses came from the class of 2020, 53.5% from the class of 2021, and 18.7% from the class of 2022. (Demographics, Table 1 ).

Most student respondents (74.9%) had a clinical rotation that was cut short or canceled due to COVID-19 and 93.7% reported not being involved in clinical rotations with in-person patient contact at the time of the study. Regarding students’ perceptions of cancelled rotations (allowing for multiple reactions), 75.8% felt this was appropriate, 34.7% felt guilty for not being able to help patients and colleagues, 33.5% felt disappointed, and 27.0% felt relieved.

Most students (74.7%) agreed that their medical education had been significantly disrupted by the pandemic. Students also felt they were able to find meaningful learning experiences during the pandemic (72.1%). Free response examples included: taking a novel COVID-19 pandemic elective course, telehealth patient care, clinical rotations transitioned to virtual online courses, research or education electives, clinical and non-clinical COVID-19-related volunteering, and self-guided independent study electives. Students felt their medical schools were doing everything they could to help students adjust (72.7%). Overall, respondents felt the pandemic had interfered with their ability to develop skills needed to prepare for residency (61.4%), though fewer (45.7%) felt it had interfered with their ability to apply to residency. (Educational Impact, Fig.  1 ).

Perceived educational impacts of the COVID-19 pandemic on medical students

A majority of medical students agreed they should be allowed to continue with normal clinical rotations during this pandemic (61.3%). Most students agreed (83.4%) that they accepted the risk of being infected with COVID-19, if they returned. When asked if students should be allowed to volunteer in clinical settings even if there is not a healthcare worker (HCW) shortage, 63.5% agreed; however, in the case of a HCW shortage only 19.5% believed students should be required to volunteer clinically. (Willingness to Participate Clinically, Fig.  2 ).

Willingness to participate clinically during the COVID-19 pandemic

When asked if they perceived a moral, ethical, or professional obligation for medical students to help, 37.8% agreed that medical students have such an obligation during the current pandemic. This is in contrast to their perceptions of physicians: 87.1% of students agreed with a physician obligation to help during the COVID-19 pandemic. For both groups, students were asked if this obligation persisted without adequate PPE: only 10.9% of students believed medical students had this obligation, while 34.0% agreed physicians had this obligation. (Ethical Obligation, Fig.  3 ).

Ethical obligation to volunteer during the COVID-19 pandemic

Given the assumption that there will not be a COVID-19 vaccine until 2021, students felt the single most important factor in a safe return to clinical rotations was having access to adequate PPE (53.3%), followed by adequate testing for infection (19.3%) and antibody testing for possible immunity (16.2%). Few students (5%) stated that nothing would make them feel comfortable until a vaccine is available. On a 1–7 scale (1 = not at all, 4 = somewhat, 7 = extremely), students felt somewhat prepared to use PPE during this pandemic in the clinical setting, median = 4 (IQR 4,6), and somewhat confident identifying symptoms most concerning for COVID-19, median = 4 (IQR 4,5). Students preferred to learn about PPE via video demonstration (76.7%), online modules (47.7%), and in-person or Zoom style conferences (44.7%).

Students believed they were likely to contract COVID-19 in general (75.6%), independent of a return to the clinical environment. Most respondents believed that missing some school or work would be a likely outcome (90.5%), and only a minority of students believed that hospitalization (22.1%) or death (4.3%) was slightly, moderately, or extremely likely.

On a 1–7 scale (1 = not at all, 4 = somewhat, and 7 = extremely), the median (IQR) reported effect of the COVID-19 pandemic on students’ stress or anxiety level was 5 (4, 6) with 84.1% of respondents feeling at least somewhat anxious due to the pandemic. Students’ perceived emotional exhaustion and burnout before the pandemic was a median = 2 (IQR 2,4) and since the pandemic started a median = 4 (IQR 2,5) with a median difference Δ = 2, p value < 0.001.

Secondary analysis of key questions revealed statistical differences between sub-groups. Women were significantly more likely than men to agree that the pandemic had affected their anxiety. Several significant differences existed for the class of 2020 when compared to the classes of 2021 and 2022: they were less likely to report disruptions to their education, to prefer to return to rotations, and to report an effect on anxiety. There were no significant differences with students who were still involved with in-person patient care compared with those who were not. In comparing areas with high COVID-19 prevalence at the time of the survey (New York and Louisiana) with medium (Illinois and Ohio) and low prevalence (California), students were less likely to report that the pandemic had disrupted their education. Students in low prevalence areas were most likely to agree that medical students should return to rotations. There were no differences between prevalence groups in accepting the risk of infection to return, or subjective anxiety effects. (Stratification, Table  2 ).

The COVID-19 pandemic has fundamentally transformed education at all levels - from preschool to postgraduate. Although changes to K-12 and college education have been well documented [ 12 , 13 ], there have been very few studies to date investigating the effects of COVID-19 on undergraduate medical education [ 14 ]. To maintain the delicate balance between student safety and wellbeing, and the time-sensitive need to train future physicians, student input must guide decisions regarding their roles in the clinical arena. Student concerns related to the pandemic, paired with their desire to return to rotations despite the risks, suggest that medical students may take on emotional burdens as members of the patient care team even when not present in the clinical environment. This study offers insight into how best to support medical students as they return to clinical rotations, how to prepare them for successful careers ahead, and how to plan for their potential roles in future pandemics.

Previous international studies of medical student attitudes towards hypothetical influenza-like pandemics demonstrated a willingness (80%) [ 4 ] and a perceived ethical obligation to volunteer (77 and 70%), despite 40% of Canadian students in one study perceiving a high likelihood of becoming infected [ 5 , 6 ]. Amidst the current COVID-19 pandemic, our participants reported less agreement with a medical student ethical obligation to volunteer in the clinical setting at 37.8%, but believed in a higher likelihood of becoming infected at 75.6%. Their willingness to be allowed to volunteer freely (63.5%) may suggest that the stresses of an ongoing pandemic alter students’ perceptions of the ethical requirement more than their willingness to help. Students overwhelmingly agreed that physicians had an ethical obligation to provide care during the COVID-19 pandemic (87.1%), possibly reflecting how they view the ethical transition from student to physician, or differences between paid professionals and paying for an education.

At the time our study was conducted, there were widespread concerns for possible HCW shortages. It was unclear whether medical students would be called to volunteer when residents became ill, or even graduate early to start residency training immediately (as occurred at half of schools surveyed). This timing allowed us to capture a truly unique perspective amongst medical students, a majority of whom reported increased anxiety and burnout due to the pandemic. At the same time, students felt that their medical schools were doing everything possible to support them, perhaps driven by virtual town halls and daily communication updates.

Trends in secondary analysis show important differences in the impacts of the pandemic. Women were more likely to report increased anxiety as compared to men, which may reflect broader gender differences in medical student anxiety [ 15 ] but requires more study to rule out different pandemic stresses by gender. Graduating medical students (class of 2020) overall described less impact on medical education and anxiety, a decreased desire to return to rotations, but equal acceptance of the risk of infection in clinical settings, possibly reflecting a focus on their upcoming intern year rather than the remaining months of undergraduate medical education. Since this class’s responses decreased overall agreement on these questions, educational impacts and anxiety effects may have been even greater had they been assessed further from graduation. Interestingly, students from areas with high local COVID-19 prevalence (New York and Louisiana) reported a less significant effect of the pandemic on their education, a paradoxical result that may indicate that medical student tolerance for the disruptions was greater in high-prevalence areas, as these students were removed at the same, if not higher, rates as their peers. Our results suggest that in future waves of the current pandemic or other disasters, students may be more patient with educational impacts when they have more immediate awareness of strains on the healthcare system.

A limitation of our study was the survey response rate, which was anticipated given the challenges students were facing. Some may not have been living near campus; others may have stopped reading emails due to early graduation or limited access to email; and some would likely be dealing with additional personal challenges related to the pandemic. We attempted to increase response rates by having the study sent directly from medical school deans and leadership, as well as respective class representatives, and by sending reminders for completion. The survey was not incentivized, and a higher response rate in the class of 2021 across all schools may indicate that students who felt their education was most affected were most likely to respond. We addressed this potential source of bias in the secondary analysis, which showed no differences between 2021 and 2022 respondents. Another limitation was the inherent issue with survey data collection of missing responses for some questions that occurred in a small number of surveys. This resulted in slight variability in the total responses received for certain questions, which were not statistically significant. To be transparent about this limitation, we presented our data by stating each total response and denominator in the Tables.

This initial study lays the groundwork for future investigations and next steps. With 72.1% of students agreeing that they were able to find meaningful learning in spite of the pandemic, future research should investigate novel learning modalities that were successful during this time. Educators should consider additional training on PPE use, given only moderate levels of student comfort in this area, which may be best received via video. It is also important to study the long-term effects of missing several months of essential clinical training and identifying competencies that may not have been achieved, since students perceived a significant disruption to their ability to prepare skills for residency. Next steps could be to study curriculum interventions, such as capstone boot camps and targeted didactic skills training, to help students feel more comfortable as they transition into residency. Educators must also acknowledge that some students may not feel comfortable returning to the clinical environment until a vaccine becomes available (5%) and ensure they are equally supported. Lastly, it is vital to further investigate the mental health effects of the pandemic on medical students, identifying subgroups with additional stressors, needs related to anxiety or possible PTSD, and ways to minimize these negative effects.

In this cross-sectional survey, conducted during the initial peak phase of the COVID-19 pandemic, we capture a snapshot of the effects of the pandemic on US medical students and gain insight into their reactions to the unprecedented AAMC national recommendation for removal from clinical rotations. Student respondents from across the US similarly recognized a significant disruption to their medical education, shared a desire to continue with in-person rotations, and were willing to accept the risk of infection with COVID-19. Our novel results provide a solid foundation to help shape medical student roles in the clinical environment during this pandemic and future outbreaks.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Association of American Medical Colleges. Interim Guidance on Medical Students’ Participation in Direct Patient Contact Activities: Principles and Guidelines. https://www.aamc.org/news-insights/press-releases/important-guidance-medical-students-clinical-rotations-during-coronavirus-covid-19-outbreak . Published March 17, 2020. Accessed April 1, 2020.

Clark J. Fear of SARS thwarts medical education in Toronto. BMJ. 2003;326(7393):784. https://doi.org/10.1136/bmj.326.7393.784/c .

Article   Google Scholar  

Loh LC, Ali AM, Ang TH, Chelliah A. Impact of a spreading epidemic on medical students. Malays J Med Sci. 2006;13(2):30–6.

Google Scholar  

Mortelmans LJ, Bouman SJ, Gaakeer MI, Dieltiens G, Anseeuw K, Sabbe MB. Dutch senior medical students and disaster medicine: a national survey. Int J Emerg Med. 2015;8(1):77. https://doi.org/10.1186/s12245-015-0077-0 .

Huapaya JA, Maquera-Afaray J, García PJ, Cárcamo C, Cieza JA. Conocimientos, prácticas y actitudes hacia el voluntariado ante una influenza pandémica: estudio transversal con estudiantes de medicina en Perú [Knowledge, practices and attitudes toward volunteer work in an influenza pandemic: cross-sectional study with Peruvian medical students]. Medwave. 2015;15(4):e6136Published 2015 May 8. https://doi.org/10.5867/medwave.2015.04.6136 .

Herman B, Rosychuk RJ, Bailey T, Lake R, Yonge O, Marrie TJ. Medical students and pandemic influenza. Emerg Infect Dis. 2007;13(11):1781–3. https://doi.org/10.3201/eid1311.070279 .

Kahn MJ, Markert RJ, Johnson JE, Owens D, Krane NK. Psychiatric issues and answers following hurricane Katrina. Acad Psychiatry. 2007;31(3):200–4. https://doi.org/10.1176/appi.ap.31.3.200 .

Al-Rabiaah A, Temsah MH, Al-Eyadhy AA, et al. Middle East respiratory syndrome-Corona virus (MERS-CoV) associated stress among medical students at a university teaching hospital in Saudi Arabia. J Infect Public Health. 2020;13(5):687–91. https://doi.org/10.1016/j.jiph.2020.01.005 .

Wong JG, Cheung EP, Cheung V, et al. Psychological responses to the SARS outbreak in healthcare students in Hong Kong. Med Teach. 2004;26(7):657–9. https://doi.org/10.1080/01421590400006572 .

Stokes DC. Senior medical students in the COVID-19 response: an opportunity to be proactive. Acad Emerg Med. 2020;27(4):343–5. https://doi.org/10.1111/acem.13972 .

Rodriguez RM, Medak AJ, Baumann BM, et al. Academic emergency medicine physicians’ anxiety levels, stressors, and potential mitigation measures during the acceleration phase of the COVID-19 pandemic. Acad Emerg Med. 2020;27(8):700–7. https://doi.org/10.1111/acem.14065 .

Sahu P. Closure of universities due to coronavirus disease 2019 (COVID-19): impact on education and mental health of students and academic staff. Cureus. 2020;12(4):e7541Published 2020 Apr 4. https://doi.org/10.7759/cureus.7541 .

Reimers FM, Schleicher A. A framework to guide an education response to the COVID-19 pandemic of 2020: OECD. https://www.hm.ee/sites/default/files/framework_guide_v1_002_harward.pdf .

Choi B, Jegatheeswaran L, Minocha A, Alhilani M, Nakhoul M, Mutengesa E. The impact of the COVID-19 pandemic on final year medical students in the United Kingdom: a national survey. BMC Med Educ. 2020;20:206–16. https://doi.org/10.1186/s12909-020-02117-1 .

Dyrbye LN, Thomas MR, Shanafelt TD. Systematic review of depression, anxiety, and other indicators of psychological distress among U.S. and Canadian medical students. Acad Med. 2006;81(4):354–73. https://doi.org/10.1097/00001888-200604000-00009 .

Download references

Acknowledgments

The authors wish to thank Newton Addo, UCSF Statistician.

Author information

Authors and affiliations.

Department of Emergency Medicine, University of California San Francisco School of Medicine, San Francisco General Hospital, 1001 Potrero Avenue, Building 5, Room #6A4, San Francisco, California, 94110, USA

Aaron J. Harries, Carmen Lee, Robert M. Rodriguez & Marianne Juarez

University of California San Francisco School of Medicine, San Francisco, California, USA

Lee Jones & John A. Davis

Clinical Emergency Medicine, University of California Irvine School of Medicine, Irvine, CA, USA

Megan Boysen-Osborn

University of Illinois College of Medicine, Chicago, IL, USA

Kathleen J. Kashima

Deming Department of Medicine, Tulane University School of Medicine, New Orleans, Louisiana, USA

N. Kevin Krane

Basic Science Education, Tulane University School of Medicine, New Orleans, Louisiana, USA

Guenevere Rae

Emergency Medicine, Ohio State College of Medicine, Columbus, OH, USA

Nicholas Kman

Department of Science Education, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York, USA

Jodi M. Langsfeld

You can also search for this author in PubMed   Google Scholar

Contributions

All authors made substantial contributions to the study and met the specific conditions listed in the BMC Medical Education editorial policy for authorship. All authors have read and approved the manuscript. AH as principal investigator contributed to study design, survey instrument creation, IRB submission for his respective medical school, acquisition of data and recruitment of other participating medical schools, data analysis, writing and editing the manuscript. CL contributed to background literature review, study design, survey instrument creation, acquisition of data, data analysis, writing and editing the manuscript. LJ contributed to study design, survey instrument creation, acquisition of data from his respective medical school and recruitment of other participating medical schools, data analysis, and editing the manuscript. RR contributed to study design, survey instrument creation, data analysis, writing and editing the manuscript. JD contributed to study design, survey instrument creation, recruitment of other participating medical schools, data analysis, and editing the manuscript. MBO contributed as individual site principal investigator obtaining IRB exemption acceptance and acquisition of data from her respective medical school along with editing the manuscript. KK contributed as individual site principal investigator obtaining IRB exemption acceptance and acquisition of data from her respective medical school along with editing the manuscript. NKK contributed as individual site co-principal investigator obtaining IRB exemption acceptance and acquisition of data from his respective medical school along with editing the manuscript. GR contributed as individual site co-principal investigator obtaining IRB exemption acceptance and acquisition of data from her respective medical school along with editing the manuscript. NK contributed as individual site principal investigator obtaining IRB exemption acceptance and acquisition of data from his respective medical school along with editing the manuscript. JL contributed as individual site principal investigator obtaining IRB exemption acceptance and acquisition of data from her respective medical school along with editing the manuscript. MJ contributed to study design, survey instrument creation, data analysis, writing and editing the manuscript.

Corresponding authors

Correspondence to Aaron J. Harries or Marianne Juarez .

Ethics declarations

Ethics approval and consent to participate.

This study was reviewed and deemed exempt by each participating medical school’s Institutional Review Board (IRB): University of California San Francisco School of Medicine, IRB# 20–30712, Reference# 280106, Tulane University School of Medicine, Reference # 2020–331, University of Illinois College of Medicine), IRB Protocol # 2012–0783, Ohio State University College of Medicine, Study ID# 2020E0463, Zucker School of Medicine at Hofstra/Northwell, Reference # 20200527-SOM-LAN-1, University of California Irvine School of Medicine, submitted self-exemption IRB form. In accordance with the IRB exemption approval, each survey participant received an email consent describing the study and their optional participation.

Consent for publication

This manuscript does not contain any individualized person’s data, therefore consent for publication was not necessary according to the IRB exemption approval.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: table s1..

Survey Instrument

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Harries, A.J., Lee, C., Jones, L. et al. Effects of the COVID-19 pandemic on medical students: a multicenter quantitative study. BMC Med Educ 21 , 14 (2021). https://doi.org/10.1186/s12909-020-02462-1

Download citation

Received : 29 July 2020

Accepted : 16 December 2020

Published : 06 January 2021

DOI : https://doi.org/10.1186/s12909-020-02462-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Undergraduate medical education
• COVID-19 pandemic
• Medical student anxiety

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Research article
• Open access
• Published: 26 February 2018

The use of advanced medical technologies at home: a systematic review of the literature

• Ingrid ten Haken 1 ,
• Somaya Ben Allouch 1 &
• Wim H. van Harten 2 , 3  

BMC Public Health volume  18 , Article number:  284 ( 2018 ) Cite this article

23k Accesses

41 Citations

7 Altmetric

Metrics details

The number of medical technologies used in home settings has increased substantially over the last 10–15 years. In order to manage their use and to guarantee quality and safety, data on usage trends and practical experiences are important. This paper presents a literature review on types, trends and experiences with the use of advanced medical technologies at home.

The study focused on advanced medical technologies that are part of the technical nursing process and ‘hands on’ processes by nurses, excluding information technology such as domotica. The systematic review of literature was performed by searching the databases MEDLINE, Scopus and Cinahl. We included papers from 2000 to 2015 and selected articles containing empirical material.

The review identified 87 relevant articles, 62% was published in the period 2011–2015. Of the included studies, 45% considered devices for respiratory support, 39% devices for dialysis and 29% devices for oxygen therapy. Most research has been conducted on the topic ‘user experiences’ (36%), mainly regarding patients or informal caregivers. Results show that nurses have a key role in supporting patients and family caregivers in the process of homecare with advanced medical technologies and in providing information for, and as a member of multi-disciplinary teams. However, relatively low numbers of articles were found studying nurses perspective.

Conclusions

Research on medical technologies used at home has increased considerably until 2015. Much is already known on topics, such as user experiences; safety, risks, incidents and complications; and design and technological development. We also identified a lack of research exploring the views of nurses with regard to medical technologies for homecare, such as user experiences of nurses with different technologies, training, instruction and education of nurses and human factors by nurses in risk management and patient safety.

Peer Review reports

As a result of demographic changes and the rapidly increasing number of older patients, there is a need for cost savings and health reforms, which include an increased move from inpatient to outpatient care in most industrialized countries over the last 10–15 years [ 1 , 2 ]. As a consequence, the transfer of advanced medical devices into home settings was considerable and it is expected that there will be a further increase in the near future [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ].

When ‘an increase’ in the number of medical technologies used at home is mentioned, it is not clear which and how many technologies are involved. Today, there are an estimated 500,000 different kinds and types of medical devices available on the world market [ 8 , 9 ]. The European Commission (EC) publishes data regarding legislation and regulations for medical devices, but the actual figures for medical technologies in outpatient practice are not available [ 10 ]. The U.S. National Center for Health Statistics (NCHS) stated that technologies have shifted from hospitals into the home, but it too does not illustrate its findings with statistics [ 11 ]. We searched for data with regard to the actual number of medical technologies used in home settings and it proved difficult to find any systematic data sets available throughout the international landscape.

An important condition for the application of medical technology in the home setting is that quality of care and patient safety must be guaranteed [ 6 ]. From a historical perspective medical technologies were designed for hospital settings [ 12 , 13 ]. This means that specific factors regarding the implementation and use at home now need to be taken into account [ 7 , 14 , 15 ]. In general, risks with medical technologies can be classified regarding (a) environmental factors; (b) human factors and (c) technological factors [ 16 ]. Human factors, however, are very important in patient safety in both hospital and in home settings [ 1 , 6 , 12 ]. For example, a major risk factor is the number of users and handovers in the chain of care. In home settings, a sometimes impressive number of different users of medical technology, often with various levels of training, instruction or education, are involved. Although patient empowerment moves control to the patient and/or relatives, an important user group is that of professional nurses. Understanding user experiences and information about adverse events and near incidents are important aspects for developing knowledge regarding implementation and use in home care setting. Sharing this knowledge can support patients and caregivers, and especially nurses in their professional work and will also contribute to patient safety and quality of care.

Therefore, there is a need to address the question first, which types of technologies are used at home; second, how frequently are they used and third, what trends can be distinguished. Additional research questions are whether there are any scientific data regarding particular user experiences; training, instruction and education; safety and risks, and finally, what can be concluded about the role of nurses in using medical technologies in the home environment. The objective of this paper therefore is to present a systematic literature search on the international state of art concerning various aspects of the use of advanced medical technologies at home.

Definitions

First, we want to clarify some definitions. In general, ‘health technology’ refers to the application of organized knowledge and skills in the form of devices, medicines, vaccines, procedures and systems developed to solve a health problem and improve quality of life [ 17 ]. The World Health Organization [ 8 ] uses the definition of ‘medical device’ as ‘An article, instrument, apparatus or machine that is used in the prevention, diagnosis or treatment of illness or disease, or for detecting, measuring, restoring, correcting or modifying the structure or function of the body for some health purpose …….’. A specification for a home use medical device is: ‘A medical device intended for users in any environment outside of a professional healthcare facility. This includes devices intended for use in both professional healthcare facilities and homes’ [ 18 ].

The landscape of medical devices is diverse with technologies varying from relatively simple to very complex devices. Wagner et al. [ 19 ] stated that ‘high-tech dependency’ (for children) matches with ‘technology-dependency’ if it concerns ‘a medical device to compensate for the loss of a vital bodily function and substantial and ongoing nursing care to avert death or further disability’. ‘The needs of these patients may vary from the continuous assistance of a device and highly trained caretaker to less frequent treatment and intermittent nursing care’ [ 20 ]. Although patients dependent of advanced medical technologies at home are often medically stable, they sometimes have high technical needs and may be expected to need long-term recovery. They also require skilled nursing [ 21 ] and a considerable degree of advanced decision making, planning, training and oversight [ 22 ]. An overall definition of ‘advanced medical technology’ is: ‘Medical devices and software systems that are complex, provide critical patient data, or that directly implement pharmacologic or life-support processes whereby inadvertent misuse or use error could present a known probability of patient harm’ [ 23 ]. Examples of advanced medical technologies used at home include ventilators for respiratory support, systems for haemo- or peritoneal dialysis and infusion pumps to provide nutrition or medication.

In the Netherlands, the National Institute for Public Health and the Environment (RIVM) [ 24 ] uses the following definition:

Advanced medical technology or high-tech technology in the home setting is defined as technology that is part of the technical skills in nursing and meets the following conditions:

technology that is advanced or high-tech, for example equipment with a plug, an on/off switch, an alarm button and a pause button;

technology that had been applied formerly only in hospital care, but that is now also often applied in home settings;

technology that can be categorized as ‘supporting physiological functions’, ‘administration’ or ‘monitoring’.

Within the Dutch classification of advanced medical technologies 19 different devices are identified (see Table  1 ), which will be used in this review as a basis to categorize the technologies. It is a classification format in which specific advanced technologies are defined. Terms as ‘advanced medical technology’ (from now on abbreviated as AMT) will be used consistently as synonyms for ‘complex medical technology’ and ‘high-tech medical technology’. The term ‘technology’ will be used in the meaning of ‘device’ or ‘equipment’. The target is on technologies that are instrumental and ‘hands on’ use by nurses in the care for patients. This means that information technology (IT) based technologies as domotica (automation for a home) are not part of the study.

Eligibility and search strategy

The systematic review of the literature was conducted early 2016. Key concepts for the review were ‘medical technologies’ or ‘medical devices’, and ‘home settings’. The concept of ‘home settings’ is related to the terms ‘home nursing’ and ‘home care service’, of which the stem is ‘home’. Combining the key concepts provided the search string: (‘medical technology’ OR ‘medical device’). As domotica is not part of the study, the search string was extended with: AND NOT (eHealth OR telecare OR telemedicine). The exact search string is (“medical technology” OR “medical devices”) AND home AND NOT (ehealth OR telecare OR telemedicine). Online databases MEDLINE, Scopus and Cinahl were searched electronically using the search string to obtain data.

Inclusion and exclusion criteria

Criteria for selection were defined prior to the search process. General criteria for inclusion were:

Year of publication: 2000–2015.

An abstract or an article (with or without abstract) has to be available, containing reference to AMT information.

The article is published in English, German, French or Dutch/Flemish language.

If medical technology is cited, it has to conform to the definition of ‘advanced medical technology’ [ 24 ].

The abstract or the article has to contain empirical material. For the purpose of this review, ‘empirical material’ has been defined as: AMT which is designed for the home setting, or where the design or choices took into account the setting of the home, or where the medical technology has been tested for the home or if the medical technology is already on the market and being used in the home setting.

For further selection, inclusion criteria related to the key concepts for title and abstract were applied, such as ‘advanced medical technology’, ‘high-tech medical technology’, ‘home-centred health-enabling technology’ and ‘care at home’. The classification of the RIVM (see Table 1 ) has been taken as a basis to categorize technologies in this review. Domotica and telemonitoring technologies scored under ‘monitoring’, such as fetal cardiotocography, and respiratory and circulatory monitoring, were left out. If the abstract or article was about electronic health records, ‘smart home’, ambient intelligence, pervasive computing, software of devices, smartphone or surgical robots, the article was also removed from selection. Technologies as ‘VAD (ventricular assist device)’, ‘dental devices’ and ‘AED (automatic external defibrillator)’ were not seen as part of the technical nursing process and these records were left out as well. Studies conducted in the hospital, hospice or nursing home settings were also excluded. An overview of all inclusion and exclusion criteria can be found in Table  2 .

Screening process

The search in the online databases using the search string, identified a total of 1287 references. After checking for duplicates, 1070 articles remained. Those articles were reviewed by a reviewer for titles and abstracts on basis of the inclusion and exclusion criteria. A double check was performed by two reviewers, who independently screened random samples of 20% of the articles. There was an initial agreement of 88%. In case of disagreement about the inclusion of an article, the decision was based on a joint discussion by all three reviewers to an agreement of 100% and the resulting screening policy was applied to the rest of the abstracts. Based on the selected titles and/or abstracts, articles were retrieved or requested in full text and assessed for eligibility. Some articles were excluded from further study, for reasons of ‘full text not available’ or the article contained no empirical material. Finally, 87 studies remained which were included in the analysis (see Table  3 ). A graphical representation of the screening process has been included in Fig.  1 .

PRISMA flowchart

Appraisal of selected studies

To conduct the systematic literature search on the international state of art concerning various aspects of the use of advanced medical technologies at home, several sources are consulted. To guarantee a scientific standard, only articles were retrieved from academic databases. MEDLINE refers to journals for biomedical literature from around the world; Cinahl contains an index of nursing and research journals covering nursing, biomedicine, health sciences librarianship, alternative medicine, allied health and more. These databases related to discipline have been supplemented with Scopus, which is considered to be the largest abstract and citation database of peer-reviewed literature. Grey literature, such as national and international reports on regulations and safety of medical technologies, is also used to illustrate the background of the problem statement and describe definitions. The Classification of advanced medical technologies in the Netherlands according to the National Institute for Public Health and the Environment (RIVM) has been used as a framework to categorise the medical technologies in the selected articles. No methodological conditions of selected studies were applied in advance and the quality criterion we applied was that of the article had to contain empirical material, as we wanted to obtain an comprehensive overview of published studies of any design and to get insight in a variety of contents.

Categorization of included articles

The characteristics of the included articles are outlined in Table  3 . All included articles were categorized by year of publication and the type of research, like the designs, methods and used instruments in the studies. Research features were synthesized where possible into overarching categories. For example, ‘systematic review’ and ‘narrative review’ were scored as ‘review’ and instruments as ‘semi-structured interview’ and ‘in-depth individual interview’ were both assigned to the category ‘interview’.

For each study, the medical technology or technologies on which the study was based was scored. The categorization was in accordance with the classification of AMTs (see Table 1 ). For example, the devices ‘continuous positive airway pressure (CPAP)’ and ‘negative pressure ventilation (NPV) have both been categorized as ‘respiratory support’; and the devices ‘jejeunostomy tube’ and ‘gastronomy tube’ as ‘enteral nutrition’. With regard to the category ‘dialysis’, further subdivision was made by using ‘haemo dialysis’ and ‘peritoneal dialysis’. If in an article a medical technology was mentioned as an example, but was no subject of study, then the technology was not scored.

‘Medical diagnosis (or diagnoses)’ as mentioned in the studies, was included in the analysis only if it was related to the medical technology as the subject of study, not if it has been mentioned as an example. In some cases, an underlying cause of diagnosis was indicated. For example, ‘chronic respiratory failure due to congenital myopathy’, in itself a neurological disorder, has been scored as ‘neurological disorder’. Diseases or disorders have been classified as much as possible under the overarching name. For example ‘pneumonia’ and ‘cystic fibrosis’ are categorized under ‘respiratory failure’, and ‘gastroparesis’ and ‘Crohns disease’ under ‘gastrointestinal disorder’. The category ‘other’ contains diagnoses which occur only once, such as ‘chromosomal anomaly’, or which are not yet determined, like ‘chronic diseases’ or ‘congenital abnormalities’.

In relation to the research questions, articles were classified regarding one of the following categories and, where appropriate, into subcategories:

User experiences

Training, instruction and education, safety, risks, incidents and complications.

From an analysis of the articles, additional categories of content emerged:

Design and technological development

Application with regard to certain diseases or disorders, indication for and extent of use

Policy and management

Types of medical technologies used, frequency of use and trends.

In four of the 87 articles (5%) there were no specific medical technologies mentioned as a subject of study (see Table  4 ). Almost half of the studies (45%) considered medical technologies for respiratory support and 39% devices for dialysis, either haemo- ( n  = 18), peritoneal- ( n  = 15) or dialysis not specified ( n  = 1). Of the studies, 29% reported on devices for oxygen therapy. In addition, there has been relatively more research conducted on equipment for ‘infusion therapy’ ( n  = 19; 22%), parenteral nutrition and enteral nutrition with a score of 20% each ( n  = 17). Relatively little research has been carried out on suction devices (8%), external electrostimulation (5%), nebulizer (5%), insulin pump therapy (3%), sleep apnea treatment (2%), patient lifting hoists (2%), vacuum assisted wound closure (1%) and continuous passive motion (1%). None of de studies considered medical technologies with regard to decubitus treatment, skeletal traction or UV (ultraviolet) therapy.

Table 4 shows that on the years 2000 and 2001 no relevant articles on the subject were found. Over the period 2000–2005, 17 articles were published, the same number over 2006–2010, and there has been a substantial increase in the number of publications to 54 over the years 2011–2015. In general, it can be concluded that more frequent investigated technologies show a fairly even distribution of publications over the years 2000–2015. Technologies, on which little research had been done, except for nebulizers, have been mainly investigated since 2010. An increase of published articles over the years 2000–2015 is apparent particularly for haemo dialysis and to a lesser extent, for devices for enteral- and parenteral nutrition. As mentioned before, several studies reported on the increase of the number of medical technologies used in home settings, but concrete data are not available. However, the number of studies and the visible trends may be indicative of the frequency of use.

In 63% of the cases ( n  = 55), a medical diagnosis (or diagnoses) was mentioned in the article. Where a diagnosis has been mentioned, in almost half of the studies ( n  = 26; 47%) it concerned diagnoses in the field of respiratory failure (see Fig.  2 ). This is not surprising, since ‘respiratory support’ is the medical technology most commonly found in the articles, similarly ‘oxygen therapy’ has also been considered relatively often. Diagnoses with regard to neurological disorders occurred in 42% of the studies ( n  = 23). Just over a quarter of the studies (27%) considered diagnoses ‘other’, such as ‘sepsis’, ‘chromosomal anomaly’ or other not specified medical disorders, nearly a quarter (24%) considered ‘cancer’ and 22% kidney disorders ( n =  12).

Number of medical diagnoses mentioned in articles on AMTs ( n  = 87, multiple answers possible)

An analysis of the used research designs identified that 64% ( n  = 56) of the studies used an observational (non-experimental) design and only a small part of the studies ( n  = 5; 6%) used an experimental design, such as a Randomized Control Trial (RCT). Of the included studies 19 were reviews and 8 were essays. A quantitative design ( n  = 37) was used more frequently than a qualitative design ( n  = 25); and only one study applied ‘mixed methods’ (quantitative and qualitative). Just over one-third of the studies (35%) used a descriptive design, and a similar number used a cross-sectional study (36%). Case series were used in 12% of the articles and a cohort-study in 9%. A phenomenological approach was applied in 16% of the records. Research instruments most frequently used were interviews (33%) and survey/questionnaires (21%). In 10% of the cases other instruments were used, including different types of assessments or tests.

With regard to the categories of content, most research has been carried out on ‘user experiences’ (see Fig.  3 ): just over one-third of the articles ( n  = 31; 36%) focused on this topic. Of these articles almost all studies focused on experiences of patients or informal caregivers ( n  = 29) and only a small number ( n  = 2) considered the user experiences of nurses or other professionals (see Table  5 ). More than half of the studies ( n  = 19) used a qualitative research design; of these 13 used a phenomenological approach. The goal of these studies was to elicit the essence of human phenomena as experienced by the users. Seven studies used a quantitative design and one an integrated mixed method. Three of the studies applied a grounded theory approach and two an experimental design (randomized controlled trial). The research instruments in this content category to collect data were interviews, either semi-structured or in-depth, and a survey. About two-thirds of the articles regarding ‘user experiences’ were published in the period 2011–2015, with an accent on the psychosocial impact of patients or informal caregivers.

Number of articles on AMTs with main content categories ( n  = 87)

Relatively little research was found on ‘training, instruction, education’ ( n  = 7), for the use of AMTs in home settings. It was remarkable that all the studies identified as focusing on this topic, concentrated on one category of AMT. Respiratory support was the subject of study in four instances and in the other three, the focus was on technologies for enteral nutrition, haemo dialysis and external electro-stimulation. Four of the seven articles utilized quantitative methods, among which three of them used an observational non-experimental design and one was an experimental randomized double-blind clinical trial. Another study within the initial seven articles used a qualitative observational non-experimental design, one was a review and another was in essay format.

In total, 22% of the articles discussed topics on safety, risks, incidents and complications ( n  = 19). In the majority of cases ( n  = 13) general aspects about the subject, for instance safe use, factors affecting safety, a safe transfer of the equipment and monitoring of assessing safety were considered. One article described technological factors with regard to safety, three articles reported on environmental factors and two explored human factors. Safety aspects were explored over a wide range of medical technologies. Five articles were reviews and one an essay. Quantitative methods were used in ten of the cases, particularly for monitoring, evaluating and assessing safety, technological and environmental factors. Only three studies used a qualitative design. Retrospective chart reviews or case series were used to collect data in some cases of unforeseen events. Table 5 shows about a doubling of published articles in the period 2011–2015 regarding this content category, compared to the previous period 2000–2010.

Approximately 20% of the selected articles considered the content category ‘design and technological development of the medical device’ ( n  = 17). The studies each focused on only one type of AMT and treated a relative wide range of eight different categories, such as ‘respiratory support’, ‘oxygen therapy’, ‘haemo dialysis’, ‘infusion therapy’, ‘insulin pump therapy’ and ‘enteral nutrition’, but also ‘external electrostimulation’ and ‘patient lifting hoists’. Interestingly, in this group of articles, relatively often ( n  = 6) no medical diagnosis was mentioned. Around half of the studies ( n  = 8) referring to this topic were in review or essay format. All other studies used a quantitative research design and throughout the search no application of qualitative designs were found. Two studies used an experimental study design (randomized crossover trial) to obtain data and two described a prospective cohort study. The majority of papers ( n  = 11) were published in the period 2011–2015 and six in the preceding period up to and including 2010.

Seven articles concerned the application of AMTs, all of them devices with regard to at least respiratory support and/or nutritional support. Five studies used a non-experimental quantitative design including the analysis of clinical data, such as record reviews or cohort studies, and two articles were reviews. Most articles on this subject ( n  = 5) were published in the period 2012–2015.

Six articles described policy or management systems in different countries regarding the use of AMTs at home. The majority of the articles ( n = 4 ) were in essay or review format. The other papers concerned a qualitative cross-sectional case study analysis and an observational quantitative study in which data are collected prospectively using a database. The categories of content will now be discussed in greater detail.

Content description and trends to secondary research questions

In this category, 22 articles described the psychosocial impact on patients or informal caregivers from the use of medical technologies at home. Living at home with the assistance of medical technology needs a range of adjustments. Fex et al. [ 25 , 26 ] state that self-care is more than mastering the technology, in terms of the health-illness transition, it requires ‘…. an active learning process of accepting, managing, adjusting and improving technology’. When it comes to children, they have to learn to incorporate disability, illness and technology actively within their process of growing up [ 27 ]. It seems that the use of medical technologies in the home can have both a positive and a negative psychosocial impact on patients and their families, which in turn causes ambivalence in experiences [ 27 , 28 ]. On the one hand, patients in general gain more independence, an enhanced overall health and a better quality of life [ 29 , 30 , 31 , 32 , 33 , 34 ]. On the other hand, for some patients the experience is one of dependency on others for executing daily activities, and these circumstances, to some extent, a social restricted live and perceived stigmatization [ 29 , 30 ]. The situation in which patients need to use medical technology at home also affects family functioning and requires next of kin responsibilities [ 35 , 36 , 37 ]. As a result, next of kin caregivers are frequently faced with poor sleep quality and quantity, and/−or other significant psychosocial effects [ 38 , 39 , 40 , 41 ]. Nevertheless, family members had a positive attitude to the concept of bringing the technology into the home [ 42 ]. Knowledge of how to use the technology and permanent access to support from healthcare professionals and significant others, enabled next of kin caregivers to take responsibility for providing necessary care and to facilitate patients learning to provide self-care [ 25 , 36 , 42 , 43 , 44 ]. Bezruczko et al. [ 45 , 46 ] developed a measure of mothers’ confidence to care for children assisted with medical technologies in their homes. To provide high quality sustainable care, nurses have to recognize and understand the psychosocial dimensions for both patients and family members which arise as a result of changing role and providing care for the patients. The need to provide emotional support and support with appropriate coping strategies is a key professional role [ 25 , 26 , 47 ]. Insight into the psychosocial effects on those involved can be used to assist designers of medical devices to find strategies to better facilitate the integration of these technologies into the home [ 28 ].

Seven articles reported on the usability, barriers and accessibility experienced by patients or informal caregivers. Findings in these studies showed that several technologies were rarely perceived as user-friendly and that home medical devices inadequately met the needs of individuals with physical or sensory deficits [ 48 , 49 ]. An accessible design which meets the diversity of individual user needs, characteristics and features would be better able to help patients manage their own treatment and so could contribute to the quality of care and safety of patients and lay users [ 50 , 51 ]. Munck et al. [ 52 ] stated that restricted patients were reminded daily of the medical technology and were more dependent on assistance from healthcare professionals than masterful patients.

In contrast to the group of patients or informal caregivers, only two papers in this content category focused on the user experiences of nurses or other professional caregivers. The review demonstrates that to maintain patient safety, more education on application of medical devices for users is needed together with improved awareness and understanding of how to use the medical technology correctly in a patient-safe way [ 53 , 54 ]. More collaboration between all involved ‘actors’ in the process of care is also requisite. Continuity among carers, trust between patient and carers and supportive communication between informal and professional caregivers are important factors for the successful implementation of medical technologies in the home environment while maintaining patient safety [ 44 , 51 , 53 , 54 , 55 ].

Three articles regarding this topic focused on nurses or other professionals and four on the patients or informal caregivers. The results showed that successful use of advanced medical technologies at home requires adequate staff education and training programmes. Although many topics in educational programmes are suitable for different types of professionals in care provision, the focus for the level and application of information can vary for Registered Nurses and unregistered care staff. In addition, for overall learning experiences to be of maximum benefit there is a need for a clear focus on the specific client groups [ 56 ]. According to Sunwoo et al. [ 57 ], in the case of home non-invasive ventilation the degree of clinical support needed is extremely variable given the mixed indications for this respiratory support. A relatively simple procedure, such as the replacement of a feeding tube, can be performed by nurses, the patient and informal caregivers, provided they are trained well [ 58 ]. However, several studies revealed the complexity of the education needed by patients and informal caregivers for the use of advanced medical technologies at home [ 59 , 60 ]. Nevertheless, the studies revealed that a structured education programme, specific training, or the support of a dedicated discharge coordinator has several advantages [ 59 , 61 , 62 ]. It was evident that good preparation by patients or informal caregivers may result in a shorter length of stay in hospital, a better performance with regard to the use of the equipment or less requests by patients and/or families for assistance.

Most articles regarding this topic ( n  = 13) reported on safety in general, like aspects of safe use, factors affecting safety, complications and prevention of incidents in the home. Some identified the risk factors and the complications that may arise [ 63 , 64 , 65 ], where Stieglitz et al. [ 66 ] also emphasize that human error is the main reason for critical incidents and that regular instruction for medical staff and patients is necessary. To prevent untoward and adverse events, evidence based guidelines, recommendations on the preferred methods for managing the equipment, troubleshooting techniques for potential complications and monitoring activities are necessary [ 67 , 68 ]. Faratro et al. [ 68 ] added that key performance and quality indicators are important mechanisms to ensure patient safety when using a medical device in the home. Methods to address or evaluate patient safety issues are for example, a home visit audit tool, a nationwide adverse event reporting system, programs such as the Medical Product Safety Network HomeNet, or, in the case of peripherally inserted central catheters (PICCs) a central catheter stabilization system [ 69 , 70 , 71 , 72 ]. However, a study conducted by Pourrat and Neuville [ 73 ] in France found that there are very few internal medical devices vigilance reports found within organizations that deliver devices for home parenteral nutrition and that safety management could be improved. The safe transfer of medical devices from a hospital setting to the home and vice versa, comes with several challenges regarding technological, environmental and human factors [ 14 ]. While many hospitals have developed policies to control the pathways of home-used devices in the hospitals, in case patients take them into the hospital when they are admitted for treatment [ 74 ]. Improvement of the safety of devices intended for use in home settings, implies also improvement of safety when their transfer to the hospital settings is urgently needed.

One article considered the technological factors, three the environmental and two the human factors. An example of research on the technological factors of safety related aspects of medical technologies used in home settings by Hilbers et al. [ 75 ] found that manufacturers pay insufficient attention to safety-related items in technical documentation for the use in the home setting. For instance, the environmental factor of electricity blackout leads to electrically powered medical devices failing. Studies show that this type of event causes a dramatic increase in appeal for access to emergency or hospital facilities, and that disaster preparation needs to include the specific needs of patients reliant on electrically driven devices [ 76 , 77 , 78 ]. Regarding human factors impacting on safety aspects, one article assessed the suitability of a particular theoretical framework for understanding safety-critical interactions of patients using medical devices in the home [ 79 ], while Tennankore et al. [ 80 ] described adverse events in home haemodialysis by the use of patients. It was remarkable that none of the articles focused on human factors with regard to the use of medical technologies at home by nurses or other professional caregivers.

Of those articles that focused on this topic, ten reported on the comparison between different types of medical technologies, or their advantages and disadvantages. The comparison of different devices for oxygen therapy was made by two articles [ 81 , 82 ] and one reported on the comparison of two types of enteral nutrition tubes [ 83 ]. Some studies regarding respiratory support considered the process of making a choice between different types of devices [ 84 , 85 , 86 ] while one paper considered the conditions for home-based haemo dialysis [ 87 ]. A minority, explored the individual characteristics and the clinical applications of several devices for respiratory support [ 88 , 89 ] and one considered devices for insulin pump therapy [ 90 ]. Seven papers discussed the technological development or effectiveness of medical technologies. The testing of devices for external electro-stimulation was described in two papers [ 91 , 92 ], with the testing of a new design patient lift was subject of one study [ 93 ]. Hanada and Kudou [ 94 ] explored the current status of electromagnetic interference with medical devices in the home setting, an issue of importance as more devices are considered for home use. The technological development of respiratory support for home use was part of one study [ 95 ], as were the possibilities of solar-assisted home haemo dialysis [ 96 ]. While the study by Pourtier [ 97 ] describes the advantages of analgesia pumps that can be read remotely by nurses, but also emphasizes the central position of a professional nurse in the transfer of information within a multi-disciplinary team.

Application with regard to certain diseases or disorders, indications for and extent of use

All articles described several aspects that need to be considered for use, such as clinical characteristics of the patients, indications for the use in the home setting, the technical availability of devices, the extent of their use at home or eventual complications and morbidity. It was important to note that all but one article ( n  = 6) were about children or related to adults with what are usually regarded as paediatric diseases. Results show that the use of AMTs at home among children after hospital discharge is common (in 20%–60% of cases), or is standard for patients with some disorders [ 98 , 99 , 100 , 101 ]. The timely application of advanced home medical technology benefits patients and can help to reduce respiratory morbidity [ 102 ]. Nevertheless, the rate of death of patients with Möbius syndrome using the devices at home was high (30%) [ 98 ], as was that of patients with intestinal failure dependent on home parental nutrition therapy in Brazil (75% for 5 years) [ 103 ]. The average cumulative survival of children needing home ventilation was found to be between 75 and 90%, depending on the medical diagnosis [ 104 ].

Three of the papers were concerned with costs and/or reimbursement. The application of medical technologies in the home environment can be cost-effective when compared to institutionalized care [ 22 , 105 , 106 ]. Nevertheless, successful employment of medical technologies in the home necessitates medical guidelines for the indicators for use, careful identification of patients as well as careful planning and attention to details [ 105 , 106 , 107 ]. Two studies concerned the dilemma’s for implementation of the technologies in home healthcare and emphasized the importance of cooperation in the chain of key stakeholders to maximize efficiency of high-tech healthcare at home, one with regard to the purchasing policy of medical technologies [ 108 ] and one with regard to the interventions of local community service centres and hospitals supporting optimal use of these technologies in the home setting [ 5 ].

The use of medical technologies in the home setting has drawn increased attention in health care over the last 15 years, as the feasibility of this type of medical support has rapidly grown. This article systematically reviewed the international literature with regard to the state of the art on this subject, in order to provide a comprehensive overview.

Trend analysis over the period 2000–2015 shows that most research has been conducted about respiratory support, dialysis and oxygen therapy; relatively little about vacuum assisted wound closure and continuous passive motion, and no about decubitus treatment, skeletal traction and UV therapy. A substantial increase in publications was found in the period 2011–2015. Although the number of studies on technologies is indicative of the extent to which they are used in home settings, however, no firm conclusions can be drawn about this.

This review also identified that most research is conducted with regard to ‘user experiences’ of medical technologies in the home, ‘safety, risks, incidents and complications’, and ‘design and technological development of medical technologies’. There have been relatively few studies which have explored the topic of training, instruction and education. Content analysis showed that the use of AMTs in the home setting can have both a positive and a negative psychosocial impact on the patients and their families, and that it has become part of self-management and patient empowerment. Successful use of advanced equipment requires adequate education and training programmes for both patients, informal caregivers and nurses or other professionals. When trying to maximize or assure safety, technological, environmental and human factors have to be taken into account, and it is evident that human factors are the main reason for critical incidents. Studies on the design and technological development of medical technologies emphasize that research is necessary to improve its possibilities and effectiveness. The research found on the application of the technologies focused predominantly on children and the results indicate that the rate of the use of home medical devices among children after hospital discharge is common. Also that when compared to institutionalized care, the application of medical technologies in the home environment can be cost-effective. Much is known, but information on several key issues is limited or lacking.

An important finding was that in almost all the reviewed articles, the study subjects were patients or informal caregivers with very few studies focused on the role and activities of nurses or other professionals as users. This was unexpected as nurses are the main group of users of AMTs at home and they have to transfer knowledge and skills on how to use the devices to patients and other caregivers. Nurses also have a key role in setting up and maintaining collaboration between all actors involved in the process of care with regard to the use of home medical technologies and in giving support to patients and family members in this respect. There is need to initiate further in depth research on AMTs use at home focusing on the role of specifically nurses.

Another interesting result was that, despite the fact that most adverse events with AMTs at home are caused by human factors, hardly any studies conducted on this subject were found. None of the articles focused on related human factors regarding the use by nurses or other professional caregivers, although this is the main user group. Research on this area could contribute to improved patient safety and quality of care. The results also revealed the tension between the advantages and disadvantages of medical technologies as experienced by patients at home. Important aspects needed to promote the benefits include improving the user-friendliness of the devices and attuning their designs for the use in home settings. This emphasizes the importance of professionals (and patient groups) working together with the designers with regard to sharing knowledge and user experiences of the use of AMTs at home in order to improve quality of care and patient safety. This collaboration emerged as of key importance in the successful use of AMTs in the home as well.

Although all included articles were retrieved from academic databases and served our purpose, there was considerable heterogeneity of quality of the studies. Most of the studies have explicitly described their research design, albeit to a greater or lesser extent. On the other hand, there were a few studies that did not even mention their methodological approach, though it could be derived from the description. Most included reviews are of moderate quality. Although findings are almost always described clearly, the search strategy and selection criteria used are often lacking. The quantitative studies are generally well described in different methodological aspects, such as selection of respondents, research design, data collection methods and analyses. Studies of qualitative nature show more variation in the depth with which the design is described. However, almost all qualitative studies have described the research instruments very well, such as semi-structured interviews or questionnaires. Despite the varying quality of the studies, we believe that the whole of different methodological approaches and the relatively large number of included studies ( n  = 87) has yielded a fairly reliable overview on the international state of art concerning various aspects of the use of advanced medical technologies at home. For future research, we recommend to emphasize the development of a more detailed methodological design, zooming in on specific technologies, using large databases or conducting large surveys, and focusing on specific groups of respondents. Both in quantitative and in qualitative studies, a good definition of the research question(s), selection of respondents, development of instruments and analysis of findings, contributes to validity, consistency and neutrality.

Some limitations do have to be taken into account with this review. Although we used the RIVM-definition of ‘advanced medical technology’, not all devices are considered as ‘complex devices’ by nurses in practice. For example, the use of an anti-decubitus mattress in the context of ‘decubitus treatment’ and ‘patient lifting hoists’ are considered by nurses as being of less or lower complexity. However, overall the RIVM-classification was found to be a good starting point, and provided a practical and useful framework from which to work to gain an insight and overview of available medical technologies. Of some of the chosen technologies defined using the RIVM-classification of AMTs, questions do have to be asked as to whether they really are part of the technical skills in nursing process. For example, ‘external electrostimulation’ and ‘continuous passive motion’ are mainly applied by physiotherapists, although with appropriate training nurses can apply them. Then too, devices regarded as only ‘monitoring’ were excluded from the review.

This systematic review study was designed to fill a gap in the current research by investigating what is known about different aspects of medical technologies used in the home. From the results it is obvious that a wide and growing range of medical technologies are used at home. Different types of technologies have been subject of study, increasingly –also in scope- over the period 2011–2015.

Professional nurses have a central role in the process of homecare which has to be recognized when considering use of AMTs at home. Nurses have to support patients and family caregivers and in consequence have a key role in providing information for, and as a member of multi-disciplinary teams. Closer collaboration by all actors involved in the process of care and feedback of user experiences to the designers is essential for the provision of high quality of care and patient safety.

This review also identified a lack of research exploring the perspectives of nurses in the processes involved in introducing and maintaining technology in homecare. Most of the research has been conducted regarding the experiences of patient experience and how informal caregivers perceive their role in using medical technologies at home. The few studies that were found, demonstrate the need for more research focused on the experiences of nurses working with advanced technologies in the home. The same applies to research on training, instruction and education to use medical technologies, as in these areas too, there was limited available research so here again there is need for further research. Despite the fact that most adverse events with medical technologies in home settings are caused by human factors, our findings also identified a lack of research in this area for nurses.

This study demonstrates that, although there is increasing attention on and recognition of the need for the use of medical technologies in the environment of the home, the research has not kept pace with the advances in care. Subjects such as user experiences of nurses with different technologies, training, instruction and education of nurses and human factors by nurses in risk management and patient safety urgently need to be investigated by further research.

Abbreviations

Automatic external defibrillator

Advanced medical technology

Continuous positive airway pressure

European Commission

Information technology

National Center for Health Statistics

Negative pressure ventilation

Peripherally inserted central catheters

Randomized Control Trial

National Institute for Public Health and the Environment

Ultraviolet

Ventricular assist device

World Health Organization

Hollestelle M, Hilbers E, van Tienhoven E, Geertsma R. Geavanceerde medische technologie in de thuissituatie: inventarisatie, gebruikersaantallen en risico’s. RIVM rapport 265011004/2005. Rijksinstituut voor Volksgezondheid en Milieu RIVM, Bilthoven; 2005.

WHO. Primary Health Care. Now more than ever. World Health Organization, Geneva; 2008.

Arno P, Bonuck K, Padgung R. The economic impact of high-technology home care. Spec. Suppl. Hast Cent Rep. 1994;24:S15–9.

Article   CAS   Google Scholar  

Arras J, Neveloff Dubler N. Bringing the hospital home: ethical and social implications of high tech home care. Spec. Suppl. Hast Cent Rep. 1994;24:S19–28.

Lehoux P, Charland C, Richard L, Pineault R, St-Arnaud J. Technologies de pointe et soins à domicile. Où en sont les CLSC? Can Med Assoc J. 2002;166:1277–8.

Google Scholar  

IGZ. Risico’s van medische technologie onderschat. Staatstoezicht op de volksgezondheid, Den Haag; 2008.

Hilbers E, de Vries C, Geertsma R. Veiligheid van medische technologie in de thuissituatie. Een analyse op basis van onderzoek van technische dossiers van fabrikanten. RIVM Rapport 360050022/2009. Rijksinstituut voor Volksgezondheid en Milieu RIVM: Bilthoven; 2009.

WHO. WHO Global model regulatory framework for medical devices including IVDS. Draft for comment. World Health Organization, Geneva; 2016.

EC. Commission welcomes new agreement for safer use of medical devices. 2016 [cited 2016 June 29]: Available from: http://ec.europa.eu/growth/tools-databases/newsroom/cf/itemdetail.cfm?item_id=8863&lang=en . 2016.

Council of the European Union. Proposal for a regulation of the European Parliament and of the council on medical devices, and amending directive 2001/83/EC, regulation (EC) no 178/2002 and regulation (EC) no 1223/2009. 2016.

NCHS. Health, United States, 2009. In Brief—Medical Technology [Internet]. National Center for Health Statistics, Hyattsville, M.D.; 2010. Available from: http://www.cdc.gov/nchs/data/hus/hus09_InBrief_MedicalTech.pdf

Weick-Brady MD, Lazerow RN. Medical devices: promoting a save migration into the home. Home Healthc Nurse. 2006;24:298–304.

Article   PubMed   Google Scholar  

Kaufman-Rivi D, Collins-Mitchell J, Jetley R. Design considerations for medical devices in the home environment. Biomed. Instrum. Technol. 2010;Home Health:21–6.

Fu M, Weick-Brady M, Tanno E. Medical devices in the home: a unique challenge for the FDA. Work. 2012;41:361–5.

PubMed   Google Scholar  

Brady P. Creating home medical devices. When it comes to designing products for home healthcare, challenges are universal. Health Manag. Technol. 2011;May:32(5):17.

Verkerke GJ, ten Cate Hoedemaker HO, Drost R, Pot JWGA, Lansbergen MDI, HM BCA, et al. Medische Technologie at risk? In: Onderzoek naar risico’s bij medische technologie en mogelijkheden om deze te voorkomen of te reduceren. Onderzoeksrapportage in opdracht van het Ministerie voor Volksgezondheid, Welzijn en Sport. Onderz. Opdr. ven het Minist. voor Volksgezond. Welz. en Sport; 2011.

Sixtieth World Health Assembly. Health technologies [internet]. 2007. Available from: http://www.who.int/medical_devices/resolution_wha60_29-en1.pdf

FDA. What is a home use device? [cited 2016 June 29]: Available from: https://www.fda.gov/MedicalDevices/ProductsandMedicalProcedures/HomeHealthandConsumer/HomeUseDevices/default.htm#1 .

Wagner J, Power E, Fox H. Technology dependent children: hospital versus home care. Philadelphia, PA: Office of Technology Assessment Task Force. JP; 1988.

OTA. Technology Dependent Children: Hospital v. Home Care. A Technical Memorandum (Report No. OTA-TM-H-38). US Government Printing Office Washington, DC; 1987. http://ota.fas.org/reports/8728.pdf

Hochstadt N, Yost D. In: Hochstadt N, Yost D, editors. The medically complex child: the transition to home care. Ann Arbor, Michigan: Harwood Academic Publishers, Bell & Howell; 1991.

Lewarski JS, Current GPC. Issues in home mechanical ventilation. Chest. 2007;132:671–6.

APSF. Definition and examples of advanced medical technology (AMT). [Internet]. 2016 [cited 2016 Jun 29]. Available from: http://www.apsf.org/initiatives.php

Hessels J, de Vries C, Hilbers E, de Bruijn A, Geertsma R. Risico’s bij gebruik van complexe medische technologie in de thuissituatie. Stand van zaken 2013. RIVM Rapport 2014–0061. Rijksinstituut voor Volksgezondheid en Milieu RIVM, Bilthoven; 2014.

Fex A, Ek A-C, Söderhamn O. Self-care among persons using advanced medical technology at home. J Clin Nurs. 2009;18:2809–17.

Fex A, Flensner G, Ek A-C, Söderhamn O. Health-illness transition among persons using advanced medical technology at home. Scand J Caring Sci. 2011;25:253–61.

Kirk S. How children and young people construct and negotiate living with medical technology. Soc Sci Med Elsevier Ltd. 2010;71:1796–803.

Thomson R, Martin JL, Sharples S. The psychosocial impact of home use medical devices on the lives of older people: a qualitative study. BMC Health Serv Res. 2013;13:467–75.

Article   PubMed   PubMed Central   Google Scholar  

Brooks D, King A, Tonack M, Simson H, Gould M, Goldstein R. User perspectives on issues that influence the quality of daily life of ventilator-assisted individuals with neuromuscular disorders. Can Respir J. 2004;11:547–54.

Lehoux P, Saint-Arnaud J, Richard L. The use of technology at home: what patient manuals say and sell vs. what patients face and fear. Sociol Health Illn. 2004;26:617–44.

Heaton J, Noyes J, Sloper P, Shah R. Families’ experiences of caring for technology-dependent children: a temporal perspective. Health Soc Care Community. 2005;13:441–50.

Alsaleh FM, Smith FJ, Thompson R, Al-Saleh MA, KMG T. Insulin pump therapy: impact on the lives of children/young people with diabetes mellitus and their parents. Int J Clin Pharm. 2014;36:1023–30.

Bortolussi R, Zotti P, Conte M, Marson R, Polesel J, Colussi A, et al. Quality of life, pain perception, and distress correlated to ultrasound-guided peripherally inserted central venous catheters in palliative care patients in a home or hospice setting. J Pain Symptom Manag. 2015;50:118–23.

Article   Google Scholar  

Gavish L, Barzilay Y, Koren C, Stern A, Weinrauch L, Novel FDJ. Continuous passive motion device for self-treatment of chronic lower back pain: a randomised controlled study. Physiotherapy. 2015;101:75–81.

Article   CAS   PubMed   Google Scholar  

Wang K-WK, Barnard A. Technology-dependent children and their families: a review. J Adv Nurs. 2004;45:36–46.

Munck B, Sandgren A, Fridlund B, Mårtensson J. Next-of-kin’s conceptions of medical technology in palliative homecare. J Clin Nurs. 2012;21:1868–77.

Toly VB, Musil CM, Carl JC. Families with children who are technology dependent: normalization and family functioning. West J Nurs Res. 2012;34:52–71.

Glader LJ, Palfrey JS. Care of the child assisted by technology. Pediatr Rev. 2009;30:439–45.

Keilty K, Cohen E, Ho M, Spalding K, Stremler R. Sleep disturbance in family caregivers of children who depend on medical technology: a systematic review. J Pediatr Rehabil Med. 2015;8:113–30.

Paddeu EM, Giganti F, Piumelli R, De Masi S, Filippi L, Viggiano MP, et al. Sleeping problems in mothers and fathers of patients suffering from congenital central hypoventilation syndrome. Sleep Breath. 2015;19:1057–64.

Toly VB, Musil CM, Zauszniewski JA. Resourcefulness training intervention: a promising approach to improve mental health of mothers with technology-dependent children. Appl Nurs Res. 2014;27:87–90.

Fex A, Flensner G, Ek A-C, Söderhamn O. Living with an adult family member using advanced medical technology at home. Nurs Inq. 2011;18:336–47.

Fex A, Flensner G, Ek A-C, Söderhamn O. Self-care agency and perceived health among people using advanced medical technology at home. J Adv Nurs. 2012;68:806–15.

Craig GM, Scambler G, Spitz L. Why parents of children with neurodevelopmental disabilities requiring gastrostomy feeding need more support. Dev Med Child Neurol. 2003;45:183–8.

Bezruczko N, Chen SPC, Hill C, Chesniak JM. Fundamental measurement of functional caregiving in rehabiliation medicine. In: 19th IMEKO world Congr; 2009. p. 2115–22.

Bezruczko N, Chen SPC, Hill CD, Chesniak JM. Measurement of mothers’ confidence to care for children assisted with tracheostomy technology in family homes. J Appl Meas. 2011;12:339–57.

Kirk S, Glendinning C, Callery P. Parent or nurse? The experience of being the parent of a technology-dependent child. J Adv Nurs. 2005;51:456–64.

Lehoux P. Patients’ perspectives on high-tech home care: a qualitative inquiry into the user-friendliness of four technologies. BMC Health Serv Res. 2004;4:1–9.

Fung CH, Igodan U, Alessi C, Martin JL, Dzierzewski JM, Josephson K, et al. Human factors/usability barriers to home medical devices among individuals with disabling conditions: in-depth interviews with positive airway pressure device users. Disabil Health J. 2015;8:86–92.

Lemke MR, Mendonca RJ. A question of accessibility: understanding lay users of medical devices. Biomed Instrum Technol 2013;Spring:201:20–5.

Rajkomar A, Farrington K, Mayer A, Walker D, Blandford A. Patients’ and carers’ experiences of interacting with home haemodialysis technology: implications for quality and safety. BMC Nephrol. 2014;15:195.

Munck B, Sandgren A, Fridlund B, Mårtensson J. Patients’ understanding of medical technology in palliative home care: a qualitative analysis. J Hosp Palliat Nurs. 2012;14:191–8.

Munck B, Fridlund B, Mårtensson J. District nurses’ conceptions of medical technology in palliative homecare. J Nurs Manag. 2011;19:845–54.

Siewers V, Holmøy T, Frich JC. Experiences with using mechanical in–exsufflation in amyotrophic lateral sclerosis. Eur J Phys. 2013;15:201–7.

Jayanti A, Wearden AJ, Morris J, Brenchley P, Abma I, Bayer S, et al. Barriers to successful implementation of care in home haemodialysis (BASIC-HHD):1. Study design, methods and rationale. BMC Nephrol. 2013;14:197.

Hewitt-Taylor J. Children who require long-term ventilation: staff education and training. Intensive Crit Care Nurs. 2004;20:93–102.

Sunwoo BY, Mulholland M, Rosen IM, Wolfe LF. The changing landscape of adult home noninvasive ventilation technology, use, and reimbursement in the united states. Chest. 2014;145:1134–40.

Fayemendy P, Sourisseau H, Jesus P, Desport JC. Changement de sonde de gastrostomie à ballonnet de l’adulte. Nutr Clin Metab. 2014;28:216–8.

Graf JM, Montagnino BA, Hueckel R, McPherson ML. Children with new tracheostomies: planning for family education and common impediments to discharge. Pediatr Pulmonol. 2008;43:788–94.

Wong J, Eakin J, Migram P, Cafazzo JA, Halifax NVD, Chan CT. Patients’ experiences with learning a complex medical device for the self-administration of nocturnal home hemodialysis. Nephrol Nurs J. 2009;36:27–32.

Tearl DK, Cox TJ, Hospital HJH. Discharge of respiratory-technology-dependent children: role of a dedicated respiratory care discharge coordinator. Respir Care. 2006;51:744–9.

dos Santos-Fontes RL, Ferreiro de Andrade KN, Sterr A. Conforto AB. Home-based nerve stimulation to enhance effects of motor training in patients in the chronic phase after stroke: a proof-of-principle study. Neurorehabil Neural Repair. 2013;27:483–90.

Buchman AL, Opilla M, Kwasny M, Diamantidis TG, Okamoto R. Risk factors for the development of catheter-related bloodstream infections in patients receiving home Parenteral nutrition. J Parenter Enter Nutr. 2014;38:744–9.

Davenport A. Complications of hemodialysis treatments due to dialysate contamination and composition errors. Hemodial Int. 2015;19(Suppl 3):S30–3.

Szeinbach SL, Pauline J, Villa KF, Commerford SR, Collins A, Seoane-Vazquez E. Evaluating catheter complications and outcomes in patients receiving home parenteral nutrition. J Eval Clin Pract. 2015;21:153–9.

Stieglitz S, George S, Priegnitz C, Hagmeyer L, Randerath W. Life-threatening events in respiratory medicine: misconnections of invasive and non-invasive ventilators and interfaces. Pneumologie. 2013;67:228–32.

McGoldrick M. Preventing infections in patients using respiratory therapy equipment in the home. Home Healthc Nurse. 2010;28:212–20.

Faratro R, Jeffries J, Nesrallah GE, MacRae JM. The care and keeping of vascular access for home hemodialysis patients. Hemodial Int. 2015;19(Suppl 1):S80–92.

François K, Faratro R, d’Gama C, Wong E, Fung S, Chan CT. Utility of baseline home visit audit in home Hemodialysis. ASAIO J. 2015;61:695–700.

Kaufman-Rivi D, Hazlett AC, Hardy MA, Smith JM, Seid HB. Provider experiences with negative-pressure wound therapy systems. Adv Skin Wound Care. 2013;26:311–8.

Kaufman D, Weick-Brady M. Homenet: ensuring patient safety with medical device use in the home. Home Healthc Nurse. 2009;27:300–7.

Egan GM, Siskin GP, Weinmann R, Galloway MM. A prospective Postmarket study to evaluate the safety and efficacy of a new peripherally inserted central catheter stabilization system. J Infus Nurs. 2013;36:181–8.

Pourrat M, Neuville S. Nutrition parentérale à domicile: prise en charge et suivi des incidents; mise en œuvre des vigilances. Enquête sur la France métropolitaine. Nutr Clin Metabolisme. 2007;21:109–14.

Tanno E. Policies and pathways of home-use medical devices through hospitals. J Clin Eng. 2010;35:215–22.

Hilbers ESM, de Vries CGJCA, Geertsma RE. Medical technology at home: safety-related items in technical documentation. Int J Technol Assess Health Care. 2013;29:20–6.

Nakayama T, Tanaka S, Uematsu M, Kikuchi A, Hino-Fukuyo N, Morimoto T, et al. Effect of a blackout in pediatric patients with home medical devices during the 2011 eastern Japan earthquake. Brain and Development. 2014;36:143–7.

Rand DA, Mener DJ, Lerner EB, DeRobertis N. The effect of an 18-hour electrical power outage on an urban emergency medical services system. Prehosp Emerg Care. 2005;9:391–7.

Greenwald PW, Rutherford AF, Green RA, Giglio J. Emergency department visits for home medical device failure during the 2003 North America blackout. Acad Emerg Med. 2004;11:786–9.

Rajkomar A, Mayer A, Blandford A. Understanding safety-critical interactions with a home medical device through distributed cognition. J Biomed Inform. 2015;56:179–94.

Tennankore KK, D’Gama C, Faratro R, Fung S, Wong E, Chan CT. Adverse technical events in home hemodialysis. Am J Kidney Dis. 2014;65(1):116–21.

Su C-L, Lee C-N, Chen H-C, Feng L-P, Lin H-W, Chiang L-L. Comparison of domiciliary oxygen using liquid oxygen and concentrator in northern Taiwan. J Formos Med Assoc. 2014;113:23–32.

Paul J, Otvos T. Comparison of nasal cannulas and the OxyArm in patients requiring chronic domiciliary oxygen therapy. Can Respir J. 2006;13:421–6.

Ao P, Sebastianski M, Selvarajah V, Gramlich L. Comparison of complication rates, types, and average tube patency between jejunostomy tubes and percutaneous gastrostomy tubes in a regional home enteral nutrition support program. Nutr Clin Pract. 2015;30:393–7.

Leger SS. Humidification des voies aériennes lors de la ventilation à domicile. Rev Mal Respir. 2005;22:1065–70.

Gregoretti C, Navalesi P, Ghannadian S, Carlucci A, Pelosi P. Choosing a ventilator for home mechanical ventilation. Breathe. 2013;9:394–409.

Chatburn RL. Which ventilators and modes can be used to deliver noninvasive ventilation? Respir Care. 2009;54:85–101.

Farrington K, Greenwood R. Home haemodialysis: trends in technology. NDT Plus. 2011;4:iii23–4.

PubMed   PubMed Central   Google Scholar  

Scala R. Bi-level home ventilators for non invasive positive pressure ventilation. Monaldi Arch Chest Dis – Pulm Ser. 2004:213–21.

Khirani S, Louis B, Leroux K, Delord V, Fauroux B, Lofaso F. Harms of unintentional leaks during volume targeted pressure support ventilation. Respir Med. 2013;107:1021–9.

Kropff J, Del Favero S, Place J, Toffanin C, Visentin R, Monaro M, et al. 2 month evening and night closed-loop glucose control in patients with type 1 diabetes under free-living conditions: a randomised crossover trial. Lancet Diabetes Endocrinol. 2015;3:939–47.

Yik YI, Ismail KA, Hutson JM, Southwell BR. Home transcutaneous electrical stimulation to treat children with slow-transit constipation. J Pediatr Surg. 2012;47:1285–90.

Prenton S, Kenney LP, Stapleton C, Cooper G, Reeves ML, Heller BW, et al. Feasibility study of a take-home Array-based functional electrical stimulation system with automated setup for current functional electrical stimulation users with foot-drop. Arch Phys Med Rehabil. 2014;95:1870–7.

Bostelman R, Ryu J-C, Chang T, Johnson J, Agrawal SK. An advanced patient lift and transfer device for the home. J Med Device. 2010;4:11004.

Hanada E, Kudou T. Ensuring electromagnetic compatibility between medical devices and wireless communications and measures against electromagnetic noise. Trans Jpn Soc Med Biol Eng. 2014;52:OS-43–4.

Dubois P, Bérenger E. La ventilation à domicile. IRBM. 2009;30:258–61.

Agar JWM, Perkins A, Tjipto A. Solar-assisted hemodialysis. Clin J Am Soc Nephrol. 2012;7:310–4.

Pourtier J. La technologie au service d’une meilleure prise en charge de la douleur au domicile. La Rev l’infirmière. 2013;190:36–8.

Matsui K, Kataoka A, Yamamoto A, Tanoue K, Kurosawa K, Shibasaki J, et al. Clinical characteristics and outcomes of Möbius syndrome in a children’s hospital. Pediatr Neurol. 2014;51:781–9.

Feudtner C, Villareale NL, Morray B, Sharp V, Hays RM, Neff JM. Technology-dependency among patients discharged from a children’s hospital: a retrospective cohort study. BMC Pediatr. 2005;May 9;5(1):8.

Brown KA, Bertolizio G, Leone M, Dain SL. Home noninvasive ventilation: what does the anesthesiologist need to know? Anesthesiology. 2012;117:657–68.

Michihata N, Matsui H, Fushimi K, Yasunaga H. Clinical features of adult patients admitted to pediatric wards in Japan. J Adolesc Health. 2015;57:421–4.

Han YJ, Park JD, Lee B, Choi YH, Suh DI, Lim BC, et al. Home mechanical ventilation in childhood-onset hereditary neuromuscular diseases: 13 years’ experience at a single center in Korea. PLoS One. 2015;10:e0122346.

Lee ADW, Galvao FHF, Dias MCG, Cruz ME, Marin M, Pedrol CN, et al. Home parenteral nutrition program and referral of potential candidates for intestinal and multivisceral transplantation in a single Brazilian center. Transplant Proc. 2014;46:1839–41.

Amin RS, Fitton CM. Tracheostomy and home ventilation in children. Semin Neonatol. 2003;8:127–35.

Southey D, Pullinger D, Loggos S, Kumari N, Lengyel E, Morgan I, et al. Discharge of thoracic patients on portable digital suction: is it cost-effective? Asian Cardiovasc Thorac Ann. 2015;23:832–8.

Hendrickson E, Corrigan ML. Navigating reimbursement for home parenteral nutrition. Nutr Clin Pract. 2013;28:566–71.

Weiler-Ravell D. The quandary of home-care respiratory management. Isr Med Assoc J. 2002;4:459–60.

Short D, Norwood J. Why is high-tech healthcare at home purchasing underdeveloped and what could be done to improve it? Health Serv Manag Res. 2003;16:127–35.

Download references

Acknowledgements

The authors thank Ronnie van de Riet, head of the Medical Technical Care Team of the hospital ZiekenhuisGroep Twente, for his time and commitment to this project.

Not applicable.

Availability of data and materials

The dataset generated and/or analyzed during the current study are available in Table  3 .

Author information

Authors and affiliations.

Saxion University of Applied Sciences, Research Group Technology, Health & Care (TH&C), P.O. Box 70.000, 7500, KB, Enschede, The Netherlands

Ingrid ten Haken & Somaya Ben Allouch

Department Health Technology & Services Research (HTSR), University of Twente, Faculty Behavioural, Management and Social Sciences (BMS), Ravelijn 5246, P.O. Box 217, 7500, AE, Enschede, The Netherlands

Wim H. van Harten

Rijnstate General Hospital, Arnhem, The Netherlands

You can also search for this author in PubMed   Google Scholar

Contributions

All authors meet the criteria for authorship and all those entitled to authorship are listed as authors. ITH made the conception and design of the study; acquisition, analysis and interpretation of data; and drafting the article. SBA and WVH have made substantial contributions to the conception and design of the study; the analysis and interpretation of data; and revising the article critically for important intellectual content. All authors have approved the final article, this submission and its publication.

Corresponding author

Correspondence to Ingrid ten Haken .

Ethics declarations

Author’s information.

Ingrid ten Haken is researcher in the research group Technology, Health & Care at Saxion University of Applied Sciences, Enschede, The Netherlands. Somaya Ben Allouch is head of the research group. Wim van Harten is professor at the University of Twente, Faculty Behavioural, Management and Social Sciences, department Health Technology & Services Research and CEO of Rijnstate general hospital, Arnhem, The Netherlands.

Ethics approval and consent to participate

Consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

ten Haken, I., Ben Allouch, S. & van Harten, W.H. The use of advanced medical technologies at home: a systematic review of the literature. BMC Public Health 18 , 284 (2018). https://doi.org/10.1186/s12889-018-5123-4

Download citation

Received : 03 October 2017

Accepted : 26 January 2018

Published : 26 February 2018

DOI : https://doi.org/10.1186/s12889-018-5123-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Home health nursing
• Medical technologies
• Patient safety
• Quality of health care
• Systematic review

BMC Public Health

ISSN: 1471-2458

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Quantitative Research Methods in Medical Education

Affiliation.

• 1 From the Division of Hospital Internal Medicine (J.T.R.) Division of General Internal Medicine (A.P.S., T.J.B.), Mayo Clinic College of Medicine and Science, Department of Medicine, Mayo Clinic, Rochester, Minnesota.
• PMID: 31045900
• DOI: 10.1097/ALN.0000000000002727

There has been a dramatic growth of scholarly articles in medical education in recent years. Evaluating medical education research requires specific orientation to issues related to format and content. Our goal is to review the quantitative aspects of research in medical education so that clinicians may understand these articles with respect to framing the study, recognizing methodologic issues, and utilizing instruments for evaluating the quality of medical education research. This review can be used both as a tool when appraising medical education research articles and as a primer for clinicians interested in pursuing scholarship in medical education.

PubMed Disclaimer

Similar articles

• Qualitative Research Methods in Medical Education. Sawatsky AP, Ratelle JT, Beckman TJ. Sawatsky AP, et al. Anesthesiology. 2019 Jul;131(1):14-22. doi: 10.1097/ALN.0000000000002728. Anesthesiology. 2019. PMID: 31045898 Review.
• A Toolkit for Medical Education Scholarship. Sullivan GM. Sullivan GM. J Grad Med Educ. 2018 Feb;10(1):1-5. doi: 10.4300/JGME-D-17-00974.1. J Grad Med Educ. 2018. PMID: 29467965 Free PMC article. No abstract available.
• A writer's guide to education scholarship: Quantitative methodologies for medical education research (part 1). Thoma B, Camorlinga P, Chan TM, Hall AK, Murnaghan A, Sherbino J. Thoma B, et al. CJEM. 2018 Jan;20(1):125-131. doi: 10.1017/cem.2017.17. Epub 2017 Apr 26. CJEM. 2018. PMID: 28443532 Review.
• Is medical education research 'hard' or 'soft' research? Gruppen LD. Gruppen LD. Adv Health Sci Educ Theory Pract. 2008 Mar;13(1):1-2. doi: 10.1007/s10459-007-9092-0. Epub 2007 Dec 4. Adv Health Sci Educ Theory Pract. 2008. PMID: 18060572 No abstract available.
• Developing scholarly projects in education: a primer for medical teachers. Beckman TJ, Cook DA. Beckman TJ, et al. Med Teach. 2007 Mar;29(2-3):210-8. doi: 10.1080/01421590701291469. Med Teach. 2007. PMID: 17701635
• General practitioner residents' experiences and perceptions of outpatient training in primary care settings in China: a qualitative study. Wu L, Tong Y, Yu Y, Yu X, Zhou Y, Xu M, Guo Y, Song Z, Xu Z. Wu L, et al. BMJ Open. 2023 Sep 15;13(9):e076821. doi: 10.1136/bmjopen-2023-076821. BMJ Open. 2023. PMID: 37714679 Free PMC article.
• Dental Teacher Feedback and Student Learning: A Qualitative Study. Fine P, Leung A, Tonni I, Louca C. Fine P, et al. Dent J (Basel). 2023 Jun 30;11(7):164. doi: 10.3390/dj11070164. Dent J (Basel). 2023. PMID: 37504230 Free PMC article.

Publication types

• Search in MeSH

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.
• Silverchair Information Systems

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Volume 18 Supplement 5

Proceedings from the 2018 Sino-US Conference on Health Informatics

• Open access
• Published: 07 December 2018

A comparative quantitative study of utilizing artificial intelligence on electronic health records in the USA and China during 2008–2017

• Xieling Chen 1 ,
• Ziqing Liu 2 ,
• Jun Yan 4 ,
• Tianyong Hao 5 &
• Ruoyao Ding 6  

BMC Medical Informatics and Decision Making volume  18 , Article number:  117 ( 2018 ) Cite this article

10k Accesses

30 Citations

Metrics details

The application of artificial intelligence techniques for processing electronic health records data plays increasingly significant role in advancing clinical decision support. This study conducts a quantitative comparison on the research of utilizing artificial intelligence on electronic health records between the USA and China to discovery their research similarities and differences.

Publications from both Web of Science and PubMed are retrieved to explore the research status and academic performances of the two countries quantitatively. Bibliometrics, geographic visualization, collaboration degree calculation, social network analysis, latent dirichlet allocation, and affinity propagation clustering are applied to analyze research quantity, collaboration relations, and hot research topics.

There are 1031 publications from the USA and 173 publications from China during 2008–2017 period. The annual numbers of publications from the USA and China increase polynomially. JAMIA with 135 publications and JBI with 13 publications are the top prolific journals for the USA and China, respectively. Harvard University with 101 publications and Zhejiang University with 12 publications are the top prolific affiliations for the USA and China, respectively. Massachusetts is the most prolific region with 211 publications for the USA, while for China, Taiwan is the top 1 with 47 publications. China has relatively higher institutional and international collaborations. Nine main research areas for the USA are identified, differentiating 7 for China.

Conclusions

There is a steadily growing presence and increasing visibility of utilizing artificial intelligence on electronic health records for the USA and China over the years. The results of the study demonstrate the research similarities and differences, as well as strengths and weaknesses of the two countries.

With the expanding use and increasing possibility of including information relating to patient outcomes and functionality such as clinical decision support, Electronic Health Records (EHRs) becomes increasingly valuable information about patient health conditions and responses to treatment over time [ 1 ]. The field of utilizing artificial intelligence techniques on EHRs data processing has attracted increasing interests from scientific community, reflected by the increasing of publications from major scientific literature databases such as Web of Science (WoS) and PubMed. The USA and China are top 2 largest economies in the world. According to literature retrieval in WoS, the two countries have the most publications the field in the last decade. Therefore, it is meaningful to conduct a quantitative analysis of the research publications from the two countries to compare their research similarities and differences, as well as strengths and weaknesses.

Research publication plays an important role in providing key linkage between knowledge generation, uptake and use in the scientific process [ 2 ]. Bibliometrics involves statistical analysis of written publications. It has been the method of choice for quantitative assessments of academic research to comprehensively explore the research advances in the past and identify future research trends in a specific field [ 3 ]. Bibliographic data from citation indexes, e.g., titles, journal, abstracts, author addresses, and etc., are analyzed statistically to recognize the popularity and impact of specific publications, authors, affiliations, or an entire field. Bibliometrics has been widely performed in the evaluation of various research areas [ 4 , 5 ]. Especially, it has also been adopted to the evolution of interdisciplinary research field, e.g., natural language processing in medical research [ 6 ], natural language processing empowered mobile computing research [ 7 ], technology enhanced language learning research [ 8 ], and text mining in medical research [ 9 ].

To that end, relevant publications in the field were retrieved from both WoS and PubMed to quantitatively explore the academic performances of the two countries in terms of current research status, research intellectual structures, and research focuses. Analyzing techniques include bibliometrics, geographic visualization, collaboration degree calculation, social network analysis, latent dirichlet allocation, and affinity propagation clustering.

Specifically, the following comparisons are conducted: 1) studying the quantitative distributions and growth characteristics of the publications, 2) identifying prolific publication sources, authors, and affiliations, 3) exploring publication geographical distributions, 4) investigating collaboration degrees and collaboration patterns, 5) visualizing scientific collaboration relations, and 6) discovering hot research topics and topic evolutions.

Data sources

The publications in the research field during 2008–2017 from WoS and PubMed databases were preferred. With a list of search keywords determined by domain experts, as shown in Table  1 , publications with “Article” type were retrieved and downloaded as plain texts. After manual review, 1031 records from the USA and 173 records from China were obtained for comparison analysis. Key elements including title, publication year, keywords, abstract, author address were extracted. In addition, corresponding affiliations and regions were automatically extracted from author address information. Key words from author keywords, Keywords Plus/PubMed MeSH, title, and abstract, were extracted by our developed natural language processing module.

In addition to basic bibliometric analysis, the techniques used in this paper include: geographic visualization, co-authorship index and collaboration degree calculation, social network analysis, and topic modelling analysis.

Geographic visualization analysis

Geographic visualization [ 10 ] refers to a set of visualization technologies for supporting geospatial data analysis. It provides ways to explore both the information display and the data behind the information itself to more readily view complex relations in images [ 11 , 12 ]. Geographic visualization works essentially by helping people see the unseen more effectively in a visual environment than when using textual or numerical description. In this study, we apply geographic visualization analysis to explore publication geographical distributions in the USA and China, respectively.

Co-authorship index and collaboration degree

Co-authorship index shown as Eq. ( 1 ), was firstly elaborated by Schubert and Braun [ 13 ]. It is obtained by calculating proportionally the publications co-authored by single, two, multi- and mega-authors for different countries. Here, the publications have been firstly divided into four categories according to author count, i.e., single-author, two-author, multiple-author publications with three to four authors, and mega-author publications with five or more authors.

In the equation, N ij is the publication count co-authored by j authors in the i th country, N io is the publication count in the i th country, N oj is the publication count co-authored by j authors in all countries, N oo is the publication count in all countries. CAI  = 100 represents the average level. CAI  > 100 indicates higher than the average, while  CAI  < 100 reflects lower than the average.

As a measure of scientific research’s connective relation to the level of author, affiliation, or country, the collaboration degree can be calculated as Eq. ( 2 ) [ 14 , 15 ].

In the equation, C Ai indicates the collaboration degree of the i year in the author, affiliation or country level. α j donates the count of author, affiliation or country for each publication. N  is the annual publication count.

In this study, co-authorship index is used to study collaboration patterns of authors, and collaboration degree is applied to measure the scientific research’s connective relation to the three levels.

Social network analysis

Social network analysis (SNA) focuses on the structure of ties within, e.g., persons, organizations, or the products of human activity or cognition such as web sites [ 16 ]. SNA works based mainly on networks and graph theory [ 17 ], and it provides both a visual and a mathematical analysis of human relations. In this study, the collaboration relations for authors, affiliations and countries are explored using social network analysis. In the network, the nodes are specific authors, affiliations or countries, and the lines are the collaboration relations. The size of node indicates the publication count of a specific author, affiliation or country. The width of link indicates the collaboration frequency between the two authors, affiliations or countries.

Topic modelling analysis

Topic modeling extracts semantic information from a collection of texts using statistical algorithms. Latent Dirichlet Allocation (LDA) is an improved three-layer Bayesian model developed by Blei et al. [ 18 ]. In LDA, each document in the text corpus is modeled as a set of draws from a mixture distribution over a set of hidden topics, where topics are assumed to be uncorrelated and each is characterized by a distribution over words. In LDA, a word is defined as an item from a vocabulary indexed by {1, …,  V }, a document is a sequence of N words denoted by d  = ( w 1 , …,  w N ), and a corpus is a collection of M documents denoted by D  = { d 1 , …,  d M }. The generation process is as follows: 1) The term distribution β indicating the probability of a word occurring in a given topic is as β ~ Dirichlet ( δ ); 2) The proportions θ of the topic distribution for a document d are determined by θ ~ Dirichlet ( α ); 3) A topic is chosen by the distribution z i ~ Multinomial ( θ ) for each word w i in the document d , and a word is chosen from a multinomial probability distribution conditioned on the topic z i  :  p ( w i |  z i ,  β ). As for variational expectation-maximization, the log-likelihood for one document d   ∈   D is given by Eq. ( 3 ), and the likelihood for Gibbs sampling estimation with k topics is as Eq. ( 4 ).

Further, Affinity Propagation (AP) clustering is used for the cluster analysis of the topics identified by LDA. AP was proposed by Frey and Dueck [ 19 ] with a basis of message passing. It does not require users to set cluster count in advance, but considers all data points to be potential exemplars and transmits real-valued messages recursively until a set exemplars of high-quality emerges [ 20 ]. AP was found to identify clusters with lower error rate and less time [ 21 ].

AP calculates the “responsibility” r ( i ,  k ) and the “availability” a ( i ,  k ), shown as Eqs. ( 5 ) and ( 6 ) for each node i and each candidate exemplar k . r ( i ,  k ) is the suitableness of k as an exemplar for i , while a ( i ,  k ) is the evidence that i should choose k as an exemplar.

In the equations, s ( i ,  k ) is the similarity between two nodes i and k . When a good set of exemplars emerges, Eqs. ( 5 ) and ( 6 ) will stop iterating. Each node i can then be assigned to the exemplar k that maximizes a ( i ,  k ) +  r ( i ,  k ). If i  =  k , then i is an exemplar. Numerical oscillations is controlled using a damping factor between 0 and 1.

In this study, words from author keywords and Keywords Plus/PubMed MeSH, publication title, as well as abstract with weights 0.4, 0.4, and 0.2 determined by our former study [ 6 ] are used as analysis units in topic modelling analysis. Term Frequency-Inverse Document Frequencies (TF-IDF) is used to filter out unimportant terms.

Growth of publications

The distributions of total publications by year for the USA and China are shown in Fig.  1 . The publication counts for both two countries are overall showing increasing trends in fluctuation. The average publications during the study period are 103.1 and 17.3 articles per year. The highest productivity is observed in 2017 with a total of 205 (19.88%) articles for the USA and 44 (25.43%) articles for China. The annual growth rates reach 26.18 and 40.54% on average for the USA and China, respectively. The trend of publications for the USA is similar with the polynomial curve ( p  < 0.05, R 2  = 95.07%) expressed as y  = 1.113636 x 2 − 4463.762 x  + 4473014, while the publication trend for China is similar with the polynomial curve ( p  < 0.05, R 2  = 84.86%) expressed as z  = 0.3674242 x 2 − 1475.01 x  + 1480346. With the simulation curves, the future productivity can be predicted. The predictive values for year 2018 for the USA and China are 230 and 47, respectively.

The distributions of total publications by year

Prolific publication sources

The 1031 records from the USA are published in 347 unique journal or conference proceeding sources, and 92 publication sources contribute to China’s 173 publications. The top 16 publication sources for the USA in Table  2 account for 49.08% of the total publications, and the 14 prolific ones for China contribute to 43.35% of the total publications. The top 3 publication sources for the USA are Journal of the American Medical Informatics Association , Journal of Biomedical Informatics , and AMIA Annual Symposium Proceedings . As for China, the top 3 prolific ones are Journal of Biomedical Informatics , Journal of Biomedical Engineering , and Studies in Health Technology and Informatics .

Prolific authors and affiliations

Three thousand three hundred fifty authors and 542 affiliations from the USA contribute to the 1031 publications, and 635 authors and 208 affiliations from China for the 173 publications. Table  3 shows prolific authors with Joshua C. Denny (53 publications), Hongfang Liu (36 publications), Guergana Savova (34 publications), Hua Xu (32 publications), and Christopher G. Chute (28 publications) as the top 5 for the USA. As for China, Buzhou Tang (7 publications) and Jianbo Lei (6 publications) are the top 2. Table  4 lists top prolific affiliations, where Harvard University with 101 publications is ranked at 1st for the USA. Other prolific affiliations include Vanderbilt University with 96 publications and Mayo Clinic with 93 publications. As for China, the top 3 are Zhejiang University , National Taiwan University , and China Academy of Chinese Medical Sciences .

Geographical distribution of publications

We study the concentration of researches in the USA and China at regional levels. The spatial characteristics of the publications from the two countries are explored. 46 states in the USA involve in the 1031 publications and 25 regions in China contribute to the 173 publications. The geographical distributions are shown as Figs.  2 and 3 , respectively. The figures display that the USA and China’ publications vary widely across the whole country. As for the USA, the top 5 prolific states are Massachusetts (211 publications), New York (173 publications), California (161 publications), Minnesota (122 publications), and Tennessee (102 publications). As for China, the top 5 regions are Taiwan (47 publications), Beijing (46 publications), Guangdong (22 publications), Shanghai (17 publications), and Zhejiang (16 publications). The publications authored by Chinese and the USA’s scholars are shown in Table  5 by top regions. For exploring the structures and dynamics of the publications, we split the whole period into two 5-year phases: 2008–2012 and 2013–2017. In the two different phases, Massachusetts, New York, California, and Minnesota always appear among the top 5 for the USA. As for China, Taiwan and Beijing are always at the top 2 places.

Geographical distributions of the publications in the USA

Geographical distributions of the publications in China

Authorship pattern and collaboration

The profiles of CAI for the USA and China have been illustrated in Fig.  4 . It is clearly indicated that CAIs of multi- and mega-author publications in the research filed in China are slightly higher than the average. However, the CAIs of multi- and mega-author publications in the USA are lower than the average. Figure  5 shows the collaboration degrees at the country, affiliation and author levels in the two countries. On the whole, the international collaboration degree is growing relatively slowly than the author and affiliation collaboration degrees. On average, 5.83 authors, 2.63 affiliations and 1.18 countries participate in each publication from the USA. As for China, on average each publication has 5.79 authors, 2.84 affiliations and 1.39 countries. The average degrees of affiliation and country for China’s publications are higher than that for the USA’s publications, while the average degrees of author is on the contrary.

Sketch map of collaboration patterns reflected by CAI

Annual collaboration degree distributions

The collaboration among countries/regions for the USA’s publications is then visualized as Fig.  6 (access via the link [ 22 ]). From the figure, the USA (the largest node in blue color) in the center of the network has the most collaborations with other countries/regions. The USA-China collaboration (the thickest line) is ranked at 1st. The collaboration networks among affiliations with publications > = 15 (access via the link [ = 15 for the USA. http://www.zhukun.org/haoty/resources.asp?id=JBMC2_US_affiliation . Accessed 10 July 2018." href="/articles/10.1186/s12911-018-0692-9#ref-CR23" id="ref-link-section-d192445341e5502">23 ]) and among authors with publications > = 12 (access via the link [ = 12 for the USA. http://www.zhukun.org/haoty/resources.asp?id=JBMC2_US_author . Accessed 10 July 2018." href="/articles/10.1186/s12911-018-0692-9#ref-CR24" id="ref-link-section-d192445341e5505">24 ]) are also visualized. Furthermore, we also visualize the collaborations for China’s publications including country/region collaboration (access via the link [ 25 ]), collaboration among affiliations with publications > = 3 (access via the link [ = 3 for China. http://www.zhukun.org/haoty/resources.asp?id=JBMC2_CN_affiliation . Accessed 10 July 2018." href="/articles/10.1186/s12911-018-0692-9#ref-CR26" id="ref-link-section-d192445341e5512">26 ]), and collaboration among authors with publications > = 3 (access via the link [ = 3 for China. http://www.zhukun.org/haoty/resources.asp?id=JBMC2_CN_author . Accessed 10 July 2018." href="/articles/10.1186/s12911-018-0692-9#ref-CR27" id="ref-link-section-d192445341e5515">27 ]). By accessing to the dynamic networks, through simply clicking the nodes, users can explore the collaboration relations for specific countries/regions, affiliations, or authors.

Collaboration network in country level for the USA’s publications

Topic generation and clustering

By setting TF-IDF value threshold as 0.1, top used terms in the author keywords, Keywords Plus/PubMed MeSH, title, and abstract of the publications are ranked by frequency. The top 5 terms and their frequencies for the USA are Drug (483), Medication (411), Cancer (370), Adverse (362), and Phenotype (275), while the top terms for China are Risk (195), Medicine (125), Drug (107), Cancer (76), and Diabetes (71). Figures  7 and 8 present the perplexities of models fitted using Gibbs sampling with different topic counts. The results suggest that the optimal topic count can be set to 35 for both the USA and China. The α is then set to 0.01339416 for the USA and 0.008163102 for China. We estimate the LDA models using Gibbs sampling with the parameters. Potential themes are assigned to each topic through semantics analysis of representative terms and text intention reviewing. Table  6 displays the top 5 best matching topics for the USA including Drug adverse event , Vaccine , Diabetes mellitus , Health data confidentiality , and Health data analysis technique , while the top 5 for China are Named entity recognition , Drug adverse event , Smoking , Prescription & drug , and Risk event . The AP clustering results based on term-topic posterior probability matrix are shown in Figs.  9 and 10 , where the 35 topics for the USA are categorized into 9 groups, and the 35 topics for China are categorized into 7 groups. For identifying emerging research topics, we firstly assign each publication to the topic with the highest posterior probability. We then explore the trends of research topics shown in Figs.  11 and 12 . We also conduct Mann–Kendall test [ 28 ] to examine whether topics present increasing or decreasing trends.

Left: estimated α value for the models fitted using VEM. Right: perplexities of the test data for the models fitted by using Gibbs sampling. Each line corresponded to one of the folds in the 10-fold cross-validation for the USA’s publications

Left: estimated α value for the models fitted using VEM. Right: perplexities of the test data for the models fitted by using Gibbs sampling. Each line corresponded to one of the folds in the 10-fold cross-validation for China’s publications

AP clustering result of the identified clusters for the USA’s publications

AP clustering result of the identified clusters for China’s publications

The trends of research topics for the USA’s publications

The trends of research topics for China’s publications

In this study, a comparative quantitative analysis of literature of utilizing artificial intelligence on electronic health records in the USA and China are conducted. This study identifies 1031 publications from the USA and 173 publications from China for the comparative analysis. Significant and polynomial increases in publication counts for both two countries can be found. This reflects a growing interest in the research field. However, the publication count of China is not at par with that of the USA, this can also be reflected by Tables 3 and 4 , where the top prolific authors and affiliations of the USA own relatively more publications than that of China. Most prolific publication sources are journals, while only some are conferences such as AMIA Annual Symposium Proceedings , indicating a wide influence of journal in the research field. From the publication distributions in region levels, it is obvious that for both the USA and China, most top prolific regions are also of economic prosperity.

From the authorship pattern analysis, it is found that publications published by scientists in the research field in China prefer to work in larger collaboration groups. This is consistent with the finding of Guan and Ma [ 29 ] that researchers have becoming more and more aware of the importance of collaboration. Comparatively, researchers in the USA prefer working with less collaboration. The collaboration degree analysis shows that authors or affiliations tend to collaborate more with those within the same country. Also, there are relatively more affiliations and countries participating in one publication on average for China than that for the USA. The USA and China are closest collaborators for each other.

Through topic modelling and clustering analysis, the 35 identified topics for the USA’s research are categorized into 9 areas including Thrombosis , Health data privacy & confidentiality , Drug adverse event & vaccine , Imaging , Disease , Audio-visual function , Application of Bayesian , Clinical data analysis technique , and Nursing . Meanwhile, the 35 identified topics for China’s research are classified into 7 areas including Cancer , Imaging , Clinical decision support , Drug & risk event , Chinese medicine , Gestational diabetes mellitus , and Clinical data analysis techniques . The results demonstrate the similarities and differences of the research between the two countries. From Figs.  11 and 12 , as well as Mann–Kendall test, 20 topics for the USA including Diabetes mellitus , Heart failure , Health data privacy & confidentiality , and etc., present statistically significant increasing trends at the two-sided p  = 0.05 level. The same is for 6 topics for China, including Named entity recognition , Risk event , Chinese medicine , Brain imaging , Drug adverse event , and Cancer . As an emerging focus in drug and cancer research topics, drug resistance has currently been one of the biggest obstacles in the treatment of cancers in clinical practice [ 30 ]. Some existing examples of cancer drug resistance research are as follows. Sun et al. [ 31 ] proposed a novel stochastic model connecting cellular mechanisms underlying cancer drug resistance to population-level patient survival for the examination of therapy-induced drug resistance and cancer metastasis. Sun and Hu [ 30 ] conducted a systematic review on the literature of mathematical modeling approaches and computational prediction methods for cancer drug resistance.

In this study, there are some limitations that are inherent to the database used and to search query developed by the authors. Such limitations were also encountered in the existing bibliometric studies, e.g., [ 32 , 33 ]. Firstly, despite the fact that WoS is a widely applied repository for bibliometric analysis and PubMed is an important data source on life sciences and biomedical topics, there are still unindexed conference proceedings and journal articles. Secondly, we treat publications of journal and conference types equally important in the analysis rather than bestowing weights for publications of different types. Furthermore, since no search query is 100% perfect, thus false positive and false negative results are always a possibility. In addition, the ranking of authors and affiliations in the study is based on data presented by WoS and PubMed. However, it is possible that some authors or affiliations might have different name spelling or more than one names, which might lead to an inaccuracy in the productivity of these authors or affiliations. Despite all these limitations, our study is the first to conduct a quantitative analysis of the research publications of utilizing artificial intelligence on electronic health records from the USA and China to compare their research similarities and differences, as well as strengths and weaknesses. The findings of our study can potentially help relevant researchers, especially newcomers, understand and compare the research performance and recent development in the USA and China, especially, as well as optimize research topic decision to keep abreast of current research hotspots.

Utilizing artificial intelligence techniques on EHRs research is an emerging and promising field. This research provides a most up-to-date quantitative analysis for exploring and comparing the research performance and development trends of the research field from the USA and China during the period 2008–2017. Results of this exploration present a comprehensive overview and an intellectual structure of the research, especially, research topics, for the two countries in the last decade.

Abbreviations

Affinity propagation

Electronic Health Records

Latent dirichlet allocation

Medical Subject Headings

Term Frequency-Inverse Document Frequencies

• United States

Web of Science

Patel VL, Shortliffe EH, Stefanelli M, Szolovits P, Berthold MR, Bellazzi R, Abu-Hanna A. The coming of age of artificial intelligence in medicine. Artif Intell Med. 2009;46(1):5–17.

Article   PubMed   Google Scholar  

Clements CJ, Watkins M, de Quadros C, Biellik R, Hadler J, McFarland D, et al. Researching routine immunization–do we know what we Don’t know? Vaccine. 2011;29(47):8477–82.

Wiysonge CS, Uthman OA, Ndumbe PM, Hussey GD. A bibliometric analysis of childhood immunization research productivity in Africa since the onset of the expanded program on immunization in 1974. BMC Med. 2013;11(1):66.

Article   PubMed Central   PubMed   Google Scholar  

Chen XL, Chen BY, Zhang CX, Hao TY. Discovering the recent research in natural language processing field based on a statistical approach. Lect Notes Comput Sci. 2017;10676:507–17.

Article   Google Scholar  

Chen XL, Weng H, Hao TY. A data-driven approach for discovering the recent research status of diabetes in China. Lect Notes Comput Sci. 2017;10594:89–101.

Chen XL, Xie HR, Wang FL, Liu ZQ, Xu J, Hao TY. A bibliometric analysis of natural language processing in medical research. BMC Med Inform Decis Mak. 2018;18(1):14.

PubMed Central   PubMed   Google Scholar  

Chen XL, Ding RY, Xu K, Wang S, Hao TY, Zhou Y. A bibliometric review of natural language processing empowered mobile computing. Wirel Commun Mob Comput. 2018;2018:1-21.

Chen XL, Hao JT, Hua SS, Hao TY. A bibliometric analysis of the research trends of technology enhanced language learning. Lect Notes Comput Sci. 2018;11284:169–79.

Hao TY, Chen XL, Li GZ, Yan J. A Bibliometric analysis of text Mining in Medical Research. Soft Comput. 2018; pp. 1–18.

Kraak MJ, Ormeling FJ. Cartography: visualization of spatial data. 3rd ed. New York: Guilford Publications; 2010.

Cartwright W, Miller S, Pettit C. Geographical visualization: past, present and future development. J Spat Sci. 2004;49(1):25–36.

Dodge M, McDerby M, Turner M. Geographic visualization: concepts, tools and applications. Chichester: Wiley; 2008.

Schubert A, Braun T. Relative indicators and relational charts for comparative assessment of publication output and citation impact. Scientometrics. 1986;9(5–6):281–91.

Zhang K, Wang Q, Liang QM, Chen H. A bibliometric analysis of research on carbon tax from 1989 to 2014. Renew Sust Energ Rev. 2016;58:297–310.

Wei YM, Mi ZF, Zhang H. Progress of integrated assessment models for climate policy. Syst Eng Theory Pract. 2013;33(8):1905–15.

Google Scholar  

McGloin JM, Kirk DS. Social network analysis. In Handbook of quantitative criminology. New York: Springer; 2010. p. 209–24.

Otte E, Rousseau R. Social network analysis: a powerful strategy, also for the information sciences. J Inf Sci. 2002;28(6):441–53.

Blei DM, Ng AY, Jordan MI. Latent Dirichlet allocation. J Mach Learn Res. 2003;3(Jan):993–1022.

Frey BJ, Dueck D. Clustering by passing messages between data points. Science. 2007;315(5814):972–6.

Article   CAS   PubMed   Google Scholar  

Frey BJ, Dueck D. Response to comment on “clustering by passing messages between data points ” . Science. 2008;319(5864):726.

Article   CAS   Google Scholar  

El-Samak AF, Ashour W. Optimization of traveling salesman problem using affinity propagation clustering and genetic algorithm. J Artif Intell Soft Comput Res. 2015;5(4):239–45.

The Network of Countries/Regions for the USA. http://www.zhukun.org/haoty/resources.asp?id=JBMC2_US_country . Accessed 10 July 2018.

The Network Affiliations with Publications >= 15 for the USA. http://www.zhukun.org/haoty/resources.asp?id=JBMC2_US_affiliation . Accessed 10 July 2018.

The Network of Authors with Publications >= 12 for the USA. http://www.zhukun.org/haoty/resources.asp?id=JBMC2_US_author . Accessed 10 July 2018.

The Network of Countries/Regions for China. http://www.zhukun.org/haoty/resources.asp?id=JBMC2_CN_country . Accessed 10 July 2018.

The Network of Affiliations with Publications >= 3 for China. http://www.zhukun.org/haoty/resources.asp?id=JBMC2_CN_affiliation . Accessed 10 July 2018.

The Network of Authors with Publications >= 3 for China. http://www.zhukun.org/haoty/resources.asp?id=JBMC2_CN_author . Accessed 10 July 2018.

Mann HB. Nonparametric tests against trend. Econometrica. 1945;13(3):245–59.

Guan J, Ma NA. Comparative study of research performance in computer science. Scientometrics. 2004;61(3):339–59.

Sun X, Hu B. Mathematical Modeling and Computational prediction of Cancer drug resistance. Brief Bioinform. 2017. pp. 1–18.

Sun X, Bao J, Shao Y. Mathematical modeling of therapy-induced Cancer drug resistance: connecting Cancer mechanisms to population survival rates. Sci Rep. 2016;6:22498.

Article   CAS   PubMed Central   PubMed   Google Scholar  

Sweileh M. Global research trends of World Health Organization’s top eight emerging pathogens. Glob Health. 2017;13(1):9.

Sweileh WM. Bibliometric analysis of literature on toxic epidermal necrolysis and Stevens-Johnson syndrome: 1940–2015. Orphanet J Rare Dis. 2017;12(1):14.

Download references

Acknowledgements

Not applicable.

Publication of the article is supported by grants from National Natural Science Foundation of China (61772146), Guangzhou Science Technology and Innovation Commission (201803010063), Natural Science Foundation of Guangdong Province (2018A030310051), and the Science and Technology Plan of Guangzhou (201804010296).

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author upon reasonable requests.

About this supplement

This article has been published as part of BMC Medical Informatics and Decision Making Volume 18 Supplement 5, 2018: Proceedings from the 2018 Sino-US Conference on Health Informatics. The full contents of the supplement are available online at https://bmcmedinformdecismak.biomedcentral.com/articles/supplements/volume-18-supplement-5 .

Author information

Authors and affiliations.

College of Economics, Jinan University, Guangzhou, China

Xieling Chen

The Second Clinical Medical College, Guangzhou University of Chinese Medicine, Guangzhou, China

The First Affiliate Hospital of Guangzhou Medical University, Guangzhou, China

AI Lab, Yidu Cloud (Beijing) Technology Co. Ltd., Beijing, China

School of Computer Science, South China Normal University, Guangzhou, China

Tianyong Hao

School of Information Science and Technology, Guangdong University of Foreign Studies, Guangzhou, China

Ruoyao Ding

You can also search for this author in PubMed   Google Scholar

Contributions

XLC leaded the method application, experiment conduction and the result analysis. ZQL participated in the design of the research and the revision of the manuscript. LW participated in the manuscript revision. JY participated in the manuscript revision. TYH provided theoretical guidance, the key term extraction program development and the revision of this paper. RYD participated in the manuscript revision. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Tianyong Hao or Ruoyao Ding .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Chen, X., Liu, Z., Wei, L. et al. A comparative quantitative study of utilizing artificial intelligence on electronic health records in the USA and China during 2008–2017. BMC Med Inform Decis Mak 18 (Suppl 5), 117 (2018). https://doi.org/10.1186/s12911-018-0692-9

Download citation

Published : 07 December 2018

DOI : https://doi.org/10.1186/s12911-018-0692-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Electronic health records
• Bibliometrics
• Topic modelling

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

• General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Expert Review
• Open access
• Published: 01 June 2024

Optimising the use of electronic medical records for large scale research in psychiatry

• Danielle Newby 1 ,
• Niall Taylor 2 ,
• Dan W. Joyce 3 &
• Laura M. Winchester   ORCID: orcid.org/0000-0003-3826-7694 2  

Translational Psychiatry volume  14 , Article number:  232 ( 2024 ) Cite this article

482 Accesses

1 Altmetric

Metrics details

• Bipolar disorder
• Psychiatric disorders

The explosion and abundance of digital data could facilitate large-scale research for psychiatry and mental health. Research using so-called “real world data”—such as electronic medical/health records—can be resource-efficient, facilitate rapid hypothesis generation and testing, complement existing evidence (e.g. from trials and evidence-synthesis) and may enable a route to translate evidence into clinically effective, outcomes-driven care for patient populations that may be under-represented. However, the interpretation and processing of real-world data sources is complex because the clinically important ‘signal’ is often contained in both structured and unstructured (narrative or “free-text”) data. Techniques for extracting meaningful information (signal) from unstructured text exist and have advanced the re-use of routinely collected clinical data, but these techniques require cautious evaluation. In this paper, we survey the opportunities, risks and progress made in the use of electronic medical record (real-world) data for psychiatric research.

Similar content being viewed by others

Opportunities and risks of large language models in psychiatry

Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety

Electronic health records and stratified psychiatry: bridge to precision treatment?

Introduction.

Psychiatry covers a vast heterogeneous group of mental disorders, manifesting as unusual mental or behavioural patterns that can impact an individual. Psychiatric research has increased rapidly to help in understanding the mechanisms of disease and treatments of multiple mental health and neurological disorders. With the growth of large-scale data, such as electronic medical records (EMR), research into psychiatric disorders can benefit from this and can provide multiple opportunities in psychiatric research that will produce evidence that could be incorporated into standards and guidelines. This, in turn, will directly impact clinical decision-making and, ultimately, the patient benefit.

Electronic medical (health) records (EMR) contain data describing clinical interactions, administrative, medico-legal, diagnostic, intervention, prescribing and investigations collected for the purposes of providing routine clinical care. In psychiatry (unlike other medical specialities), detailed clinical data is most often in unstructured, narrative “free text” and depending on the healthcare system, other clinical data (e.g. structured data recording the results of investigations and prescribing) will be available to varying degrees. Rather than representing unadulterated “real world” data, the potential for EMRs to provide relevant, reliable and rich data varies depending on the application; for example, reusing EMR data for predicting child and adolescent mental health problems after first contact with services [ 1 ] demonstrated limited utility. An often unrecognised problem with EMR data—particularly as a source of observational, retrospective cohort data—is that the content reflects treatment as usual (i.e. extracted prescribing data will likely display indication biases), the culture of the institution and its practitioners (e.g. unstructured narrative data might reflect the mixing of administrative, medico-legal and clinical data) and the institution’s implementation of an EMR platform [ 2 , 3 ] (for example, whether the pathology EMR system in use at the same hospital are linked meaningfully to the central EMR being used for research data extraction) [ 4 ].

There is still a common consensus that randomised control trials (RCTs) are the gold standard to provide causal evidence for the efficacy, effectiveness and benefits of interventions, and for inferential modelling of risk factors for mental illness. However, RCTs can be expensive, time-consuming, unethical to conduct and generally have short follow-up times compared to observational studies. Some argue that this delivers evidence lacking generalisability to patients and their presentations in routine clinical care and excludes those patients whose risk formulation excludes them from clinical trials. Therefore, evidence derived from EMR-based research has the potential to complement evidence from controlled trials, especially when considering health equity and reproducibility [ 5 , 6 , 7 ]. Furthermore, causal inference methods are being introduced to address some of the biases in observational research using EMR data.

This narrative review concentrates on how research on neuropsychiatric disorders (such as depression, bipolar, schizophrenia, anxiety, eating disorders and dementia) can utilise big data such as EMRs to generate evidence to inform clinical decision-making and, importantly, improve patient outcomes. Examples of types of data that can be utilised, examples of use, and the benefits and limitations of EMR will be discussed. Finally, a summary of how EMR can be further advanced, such as the use of genetics and data triangulation will be discussed to further help optimise EMR for psychiatry research.

Data sources for large-scale psychiatric research

There is a vast variety of data sources that can be used for large-scale research in psychiatry. Before designing a study, it is important to understand different data sources and their strengths and limitations to ensure a research question can firstly be answered and then without significant biases. Broadly speaking, large-scale data resources can be roughly grouped into three types (Fig. 1 ).

CPRD: Clinical Practice Research Datalink [ 174 ], QResearch ( https://www.qresearch.org/ ), THIN: The health improvement network ( https://www.the-health-improvement-network.com/ ), CRIS: Clinical Record Interactive Search, OPTUM ( https://www.optum.com/ ), NHS Digital ( https://digital.nhs.uk/ ), GLAD study: Genetic Links to Anxiety and Depression Study [ 175 ], SveDem: The Swedish Dementia Registry [ 176 ], UK Biobank [ 177 ], Our Future Health ( https://ourfuturehealth.org.uk/ ), All of Us ( https://allofus.nih.gov/ ), German National Cohort [ 178 ]. EMR and claims databases contain a variety of data formats which can be classified as structured or unstructured [ 69 ]. Structured data includes information such as age and gender, measurements such as blood pressure readings, height and also diagnosis codes, laboratory tests and medication prescribing. Whereas unstructured text includes narrative data such as clinical notes (e.g. biopsychosocial formulations, differential diagnoses, mental state examinations and risk formulations). Compared to narrative, unstructured data, structured data is easier to process with little pre-processing because it is stored in a standardised format. EHR and claims databases have vast patient numbers covering all diseases and disorders, giving the opportunity to look at psychiatric conditions and their comorbid diseases.

Disease registries contain patients with a specific condition and collect patient information longitudinally [ 8 ]. As the early and accurate diagnosis of psychiatric conditions is essential for better disease monitoring and management, registries represent a valuable tool for studying the known risk factors, as well as identifying new risk factors and markers that may help improve the accuracy of diagnostic procedures in psychiatry [ 8 ]. Disease registries also allow insights into medication use and their effectiveness and adverse effects in managing mental health conditions. Therefore, disease registries play an important part in improving health outcomes for patients and reducing healthcare costs [ 9 ].

Large population cohort studies contain large sample sizes and extensive phenotypic, imaging and biological measurements, including genetics [ 10 ]. Due to the large number of participants, this allows researchers to investigate psychiatric conditions with sufficient statistical power. With genotyping carried out for these large cohort studies this allows for the complex relationship of multiple small-effect genetic and environmental influences of psychiatric conditions to be studied [ 11 ]. One of the caveats of some large population studies is the potential lack of representativeness [ 12 ] and diversity, particularly for those with mental health conditions [ 13 ]. Other data sources that are potentially important for psychiatry research include data collected from wearables, mobile phones and social media platforms [ 14 , 15 , 16 , 17 , 18 ].

The current uses of EMR to enhance psychiatry research

EMR is used to generate a wide variety of evidence to inform and improve patient care ranging from using curated EMR data for epidemiology to identifying novel risk factors, opportunities for innovation in treatments and predictive analytics for those at risk and/or treatment response. The main uses related to the psychiatry field are discussed below.

Comparative effectiveness studies

Comparative effectiveness research using EMR can provide evidence to improve patient care and reduce healthcare costs. This is done by comparing the benefits and harms of alternative treatments or methods to prevent, diagnose, and treat a variety of health conditions [ 19 , 20 ]. There are a variety of study designs that can be implemented to understand the effects of different mental health disorders, such as anxiety and depression, on quality of life before and after diagnosis [ 21 ], as well as the effectiveness of different medications [ 22 , 23 , 24 ] and different treatment regimes [ 25 ] for a variety of mental illnesses [ 26 , 27 , 28 ]. For neurodegenerative conditions such as dementia, there is also a growing body of evidence using EMR to investigate the potential benefits and harms of licensed medications [ 29 , 30 , 31 , 32 , 33 , 34 ]. As there is evidence that common diseases such as diabetes and hypertension are probable risk factors [ 35 ], this suggests that treatments for these conditions may influence cognitive decline and potentially modify dementia risk. On the other hand other anticholinergic medications [ 36 , 37 ] and benzodiazepines [ 38 , 39 ], may accelerate decline or increase the risk of dementia.

Descriptive studies

Descriptive studies quantify features of the health of a population of interest. This leads to knowledge that could generate hypotheses for aetiologic research and inform action in the population it concerns [ 40 , 41 ]. The use of descriptive studies can be used to estimate the burden of disease in a population at a certain point in time or over time (e.g. incidence and prevalence). For psychiatry, descriptive studies can be used to ascertain if there have been changes in trends of mental health disorders such as depression [ 42 ] and anxiety [ 43 ] as they present to healthcare services or within certain populations of patients with chronic diseases, mental health conditions [ 44 ] and life-limiting diseases such as cancer [ 45 ]. This can help develop strategies that could mitigate and treat those with mental health conditions and descriptive epidemiology has been vital to understanding the impact of the COVID-19 pandemic on mental health [ 46 , 47 , 48 ]. Other types of descriptive studies entail describing drug utilisation and adverse drug reactions to medications [ 49 , 50 ]. These studies can provide information regarding potential over-, under- or mis-prescribing of medications leading to poorer patient outcomes, particularly in high-risk populations such as those with mental health or neurological conditions [ 49 , 51 , 52 ].

Prediction modelling

Predictive modelling attempts to complement evidence-based medical practice by providing methods for using clinical data to estimate an individual’s probability of, e.g. experiencing benefit or harm from a treatment, experiencing an outcome (prognosis) or having a diagnosis [ 53 ]. A critical stage in developing predictive models is external validation and calibration of a tentative model, ideally in a prospective evaluation. EMRs are often conceived as ideal data sources for predictive model development and, sometimes, validation; but currently, there is limited evidence for the robustness of predictive models in psychiatric applications more generally, for example, in a systematic review of risk prediction models [ 54 ], of 89 studies, only 29 had been subjected to external validation and 1 study was considered for implementation.

Common clinical domains for predictive modelling include suicide risk [ 55 ], diagnostic trajectories [ 56 ], treatment outcomes in depression [ 57 ] and identification of dementia cases [ 58 ]. Notably, many well-designed and implemented models (e.g. those with robust validation) have tended to use national registry data (rather than EMR-derived data). Whilst individual studies using EMR data have shown promise [ 59 , 60 , 61 ], there is little synthesised evidence demonstrating the value of EMRs for predictive modelling. Registry data is (importantly) different from EMR data (even if one federates a number of organisation’s individual EMRs) because registries are samples of the whole population, whereas EMRs are selection-biassed (i.e. only people who are unwell and require input from services will be visible in EMRs).

Challenges and opportunities with using electronic medical records for large-scale psychiatry research

One of the most important considerations when utilising EMR for research is that it is collected for healthcare and not for research purposes. It is important to understand this when using EMRs for research because they contain a vast amount of data that reflects medico-legal and administrative concerns, rather than being clinically relevant.

The Big Data Paradox

Big data can be characterised by its variety, volume, velocity, and veracity [ 62 , 63 ]. In context, EMR can be considered “big data” (due to its variety, volume and veracity) containing information in the order of thousands to millions of patients. The large number of patients and coverage of clinical conditions allow opportunities to study rare events or disorders (i.e. exploiting volume, variety and veracity) encountered in “real-world” clinical practice [ 64 ]. However, EMR is collected to support healthcare delivery and services, which gives rise to heterogeneity in the data collected. The volume of EMR datasets promises large sample sizes but this often leads to an assumption that derived error and uncertainty estimates will be necessarily more precise. However, this commonly received wisdom does not always hold; the “big data paradox” [ 65 , 66 ] describes how increasing the sample size alone does not guarantee a more precise estimate of e.g. sample averages. In studies of survey data, vaccine uptake and the prediction and tracking of flu [ 67 , 68 ], large sample sizes yielded misleadingly narrow uncertainty estimates leading to biased population inferences. We should be mindful of the quality, heterogeneity, and problem difficulty that are all functions of the data used, how it is collected, and the specific application or re-use of that data [ 65 ].

The dominance of unstructured text in electronic medical records in psychiatry

Unstructured data, such as free text, requires considerable pre-processing and, usually, domain expertise and human annotation. A major problem with clinical free text is the language used by clinicians is often idiosyncratic, with frequent abbreviations (sometimes, with parochial meaning such as the names of clinical services), and varied medical vocabularies [ 69 ]. Drug names, for example, often have different brand names in different national territories or “class” nomenclature (i.e. “antidepressant”) depending on the institution, requiring ontologies to be developed for mapping between synonymous terms (e.g. the Unified Medical Language System [ 70 ]) to assist pre-processing before being used in analyses or model development. Within psychiatry and mental health the number of clinical notes for any individual can be very large and written in a narrative but terminologically dense manner and often contain a high proportion of redundant text [ 71 ]. Further, unlike other medical and surgical specialities (that can utilise EMR-based sources of routinely collected structured data), psychiatry is far more reliant on clinical information such as symptoms, behaviour and clinical assessments within the unstructured notes. The major task is to represent this clinical text in a useful way for both algorithms and clinicians alike.

The computational processing and analysis of human language found in the unstructured text (clinical notes) falls under the broad field of natural language processing (NLP), which pertains to the statistical [ 72 , 73 ] and deterministic (e.g. rule-based) representation and processing of language. NLP seeks to represent words, sentences, paragraphs and sometimes, the entire text corpus in such a way that algorithms can be deployed to automate task-specific analyses of the text. Contemporary NLP usually combines rule-based methods with statistical (usually machine and deep learning methods) to represent written and spoken language. The current state of the art for NLP focuses on pre-trained language models (PLMs, very-large deep learning NLP networks trained using a language modelling objective) like BERT [ 74 ] and GPT-3 [ 75 ]. PLMs better capture semantic nuances contained in sequences of text and have seen state-of-the-art performance in a considerable number of domains e.g. finance, internet of things, biomedical [ 75 , 76 , 77 ]. Most impactful applications of PLMs to EMR-free text have focused on Information Extraction, e.g. named entity recognition. This has spawned a number of tools to create structured representations of that free text to aid clinical decision support, such as MedCAT [ 78 ], NeuroBlu [ 3 ], and Med-7 [ 79 ]. However, the research into the representation of clinical notes in psychiatry as a whole is still relatively limited, especially in relation to the latest trends in NLP.

A concrete example of a challenge in NLP applied to narrative EMR data in psychiatry concerns the vernacular use of diagnostic terminology; for example, a healthcare professional might summarily describe their impression that “the patient seems depressed”. In isolation, this statement might refer to signs (observations by the professional), symptoms (difficulties reported by the patient) or a summary diagnosis (the signs and symptoms observed in this clinical encounter meet diagnostic criteria for a depressive disorder or episode). Similarly, a recording of clinical state might read “Mood: normal” and could refer to the patient’s mood being normal for them (referencing a previously observed clinical state), a normative assessment representing a lack of pathology (where the clinician’s recording references their own experience of the population of people with “abnormal” mood) or could represent a change over time (i.e. that the patient’s mood has returned to some baseline). Resolving these different interpretations remains difficult using data-driven lexical or statistical analyses of language and necessarily, resource-intensive expert human annotation is required.

Resource challenges using machine learning-based NLP within psychiatry

Contemporary neural networks (NN) are computationally expensive when compared to other mature machine- and statistical-learning methods. Practical development of NN models requires parallel processing using Graphical Processing Units (GPU) that are costly. The last few years have seen neural networks reaching the size of hundreds of billions of parameters, and the amount of data used to train them is usually comparably vast. A prominent large language model, GPT-3 [ 75 ], has ~175 billion model parameters (by comparison, the human brain has ~86 billion neurons). Commercial interests often obfuscate accurate costing, but speculative estimates are of the order of several million US dollars to train models of this magnitude [ 80 , 81 ]. This trend of increasing performance through scaling of model size/complexity is problematic for resource-constrained environments such as publicly-funded hospitals (i.e. the UK’s National Health Service).

A crucial component of any AI/ML-driven algorithm or tool is that it is trusted and usable by human clinicians and patients; Critically, imbuing trust in a model requires that the algorithm deliver outputs that include justifications or reasons for reaching a given output or decision, sometimes referred to as XAI (eXplainable AI). Many ML methods (and especially deep learning neural networks) are opaque or “black-box” models, where the computational processes that intervene between input and output are too complex to be easily understood by any human user. There is an active research field dedicated to illuminating the machinery of such models, although the concept of what constitutes an explainable or interpretable model remains controversial [ 82 ]. If clinicians and patients are to trust an AI/ML model, they will likely favour model transparency and simplicity—often described as intrinsically interpretable models [ 83 ]—over the often modest performance gains given by complex DL models [ 84 ]. Free text data in sensitive (and, for psychiatry, often stigmatising) settings carries serious privacy risks due to the difficulty in adequately anonymising data and removing personally identifiable content [ 69 ]. For this reason, these data are often warehoused with strict data access regulations that necessarily inhibit reproducibility and replicability efforts.

Problems with data linkages and selection bias in EMR

Linking together information about the same individual across multiple data sources can further enhance existing data [ 85 , 86 ], improve the quality of information, and offer a relatively quick and low-cost means to exploit existing data sources. One benefit of data linkage in psychiatry is it can provide additional information on other non-psychiatric conditions and medications [ 87 ], allowing more detailed information about patient’s medical history, which can be used to reduce biases in research studies. Although data linkages can improve knowledge about psychiatric research [ 87 , 88 ], there are limitations. Errors in the data linkage process can introduce bias of unknown size and direction, which could feed through into final research results, leading to overestimating or underestimating results [ 89 ]. Missingness of different participant characteristics in EMR, such as age, gender and race, can also lead to systematic bias and issues with the validity and generalisability of research results [ 90 , 91 ].

Selection bias is a common problem in observational research and occurs when characteristics influence whether a person is included in a group. For example, in psychiatry, only those with extreme mental health conditions enter secondary care due to the different priorities of healthcare providers and government funds. Therefore, any research studies using EMR in secondary care will differ from the general population [ 92 ]. Furthermore, selection biases can exacerbate existing disparities, such as those relating to ethnicity, sexual orientation and socioeconomic status, that can lead to inequalities in treatment and healthcare [ 93 , 94 , 95 , 96 , 97 ]. Findings from psychiatric research conducted in selected groups should be interpreted with great caution unless selection bias has been explicitly addressed.

Phenotyping in psychiatric research

Phenotyping is the process of identifying specific patients with a clinical condition or characteristic(s) based on information in their EMR [ 98 ]. It can involve combining different types of data such as diagnosis codes, procedures, medication data, laboratory and test results, and unstructured text [ 99 ] with growing interest in using data from smartphones and other digital wearables [ 15 , 100 ]. Phenotypes can be derived using algorithms that use filters and rule-based algorithms or machine learning methods based on structured data [ 101 , 102 ]. The Electronic MEdical Records and GEnomics (eMERGE) Network [ 103 ] and CALIBER [ 104 ] have both shown that phenotypes can be identified and validated and consequently used in research. Patients identified with a specific phenotype can be included in cohort studies in order for further study of risk factors or drug safety surveillance, genetic studies as well as recruitment for clinical trials [ 105 , 106 , 107 , 108 , 109 , 110 ]. The psychiatry field presents a unique challenge for phenotyping as the majority of psychiatric diagnoses typically rely on self-reported symptoms, behaviour and clinical judgement, meaning a combination of structured and unstructured text has been shown to give rise to more accurate phenotypes with less misclassification of cases [ 111 , 112 ]. Problems arise with phenotyping when there is no consistency in the phenotyping process, only using structured data may not accurately represent the disease status of the patient, what types or combinations of data could be used from different healthcare datasets and the lack of translation of phenotypes to different health care settings and countries [ 113 ]. As phenotyping is a dynamic process, it requires clinical expertise and multiple cycles of review and can take many months of development [ 114 ]. Once a phenotype has been derived, validation of the phenotype is a critical process [ 115 ]. A phenotype must have high sensitivity and specificity, limiting both false positives and false negatives. Validation can be done using a variety of different approaches [ 104 ], such as by cross-referencing different data EMR sources and case note reviews by clinical experts to confirm a diagnosis based on the phenotype developed. Accuracy measures can then dictate how useful a phenotype will be for use in further research [ 106 , 116 , 117 ].

Future considerations for optimising the use of electronic medical records in psychiatry

There are many examples of the use of EMR to generate evidence in psychiatric research. However, to aid in the improvement and research applications of EMR we discuss future considerations which could optimise the use of EMR.

Design, statistical techniques to address biases and reporting in observational psychiatric research

The design, analysis and reporting are vital components for optimising the use of EMR. Observational research using EMR is utilised because RCTs that would answer causal questions are sometimes not feasible, unethical and take too long. By applying the study design principles of RCTs to observational studies, the causal effect of an intervention [ 118 , 119 ] can be estimated, and this helps avoid biases such as selection and immortal time biases [ 120 ]. This approach, called “target trial emulation” [ 118 ], uses EMR to emulate a clinical trial—the target trial—that would answer the causal research question. If target trial emulation is successful the results from observational data can yield similar results to the RCT [ 121 , 122 , 123 , 124 , 125 ]. Target trial emulation is now being used for a wide variety of conditions, such as showing potential beneficial effects of statins with dementia risk [ 126 ] and harmful effects of protein pump inhibitors with dementia risk [ 127 ]. Other applications include determining optimal drug plasma concentrations in bipolar disorder [ 128 ] and establishing the risk of diabetes with anticonvulsant mood stabilisers [ 129 ]. Target trial emulation cannot remove bias due to the lack of randomisation of observational data [ 118 ]. However, methods to address this, such as propensity scores, can be applied to reduce this confounding [ 130 , 131 ].

Clinical decision support tools for psychiatry could include identifying or detecting those at risk of certain disorders, illness progression/prognosis and using treatment response data to improve personalised care. However, specifically for predictive models, research has shown that over 90% were at high risk of bias [ 53 ]. Therefore, in order to optimise the use of EMR for developing clinical decision support, we require careful attention to model development, sample sizes [ 132 ], internal and external validation, including calibration and assessment of clinical utility and generalisability should be adopted [ 59 , 133 ].

Studies using EMR can be prone to publication bias and reporting bias [ 134 , 135 ]. On top of this, published research often omits important information or the information is unclear and very often, the nature of EMR data means it cannot be shared for interrogation, reproducibility and replication studies. These biases are a concern because they undermine the validity of studies. Study analysis plans and study results should be reported transparently, including what was planned, what was carried out, what was found, and what conclusions were drawn. Researchers can now register statistical analysis plans for a study prior to analysis (e.g. clinicaltrials.gov, researchregistry.com, encepp.eu) and the STROBE [ 136 ] and TRIPOD [ 137 ] guidelines offer a checklist of items that should be addressed in articles reporting studies to increase transparency [ 138 , 139 ]. Furthermore, in order to improve reproducibility and ambiguity, analytical code should also be freely available [ 140 ].

Precision medicine to provide individualised healthcare

With the increase in the availability of accurate deep phenotyping information from unstructured text researchers will be able to make more precise insights about disease outcomes from clinical information. This has expanded the scope of evidence-based prediction and tools designed to triangulate evidence from multiple sources are now being developed for applications in precision medicine. For example, the Petrushka [ 105 ] web-based tool uses data from multiple sources, including QResearch (primary care), EMR (secondary care) and available literature to make personalised medication recommendations in individuals with unipolar depression. Other projects seek to incorporate other data modalities, such as wearables to give a more detailed digital phenotype [ 141 ]. However, further validation is needed to convince clinicians of the benefits of supported clinical decision-making.

Triangulation of evidence from multimodal data for large-scale psychiatry research

There is a vast array of data acquired in research and healthcare which covers a variety of different modalities. These different modalities, such as omics, histology, imaging, clinical and smart technology, can help researchers unveil novel mechanistic insights to help understand crucial information about the complexity of mental health and neurological conditions. Triangulation of evidence is an approach where one can obtain more confidence in results by carrying out analyses integrating different statistical methodologies and/or data modalities [ 142 , 143 ]. The key is that each analysis has different sources of potential bias that may be unrelated to each other. If the results from each different analysis point to the same conclusion, this strengthens the confidence in the findings obtained. Examples of this triangulation approach in mental health research include assessing the relationship between cultural engagement and depression, where the authors used three different statistical methodologies with different strengths and weaknesses to show lower cultural engagement is associated with depression outcomes [ 144 ]. Other examples used observational data and genetic data to triangulate evidence between smoking and suicide ideation and attempts [ 145 ], and anxiety disorders and anorexia nervosa [ 146 ]; however, the triangulated results were inconsistent with each other, potentially questioning the causal relationships established using any of the sources. For psychiatry, triangulation could be used by applying different statistical approaches, using different EMRs across different countries and healthcare settings and/or integrating other non-EMR data as discussed below to help provide further understanding regarding causality and optimise big data in psychiatric research.

Incorporating biomarkers in EMR phenotyping

Biomarkers are biological measures utilised to better diagnose, track or predict psychiatric disorders. These can range from clinical assays and brain imaging to digital biomarkers from wearables. In EMR research, they can be used to help define disease phenotypes or better understand outcomes and applications in precision medicine. In Alzheimer’s Disease, fluid biomarkers (cerebrospinal fluid and blood plasma) for the tau protein are used to determine disease pathology to aid in trial recruitment. Biomarkers of neurodegeneration have been successful, neurofilament light measured in blood or CSF can be used to assess axon damage [ 147 ]. Measures of inflammation, such as C-reactive protein, have applications in many psychiatric disorders. Disorder-specific markers have been identified and replicated in meta-analyses for Vitamin B6 in schizophrenia and basal cortisol awakening in bipolar disorder [ 148 ]. However, in many psychiatric disorders, translation to clinical applications is limited [ 148 , 149 ] and further work will be necessary to validate these potential candidates in suitable cohorts.

The suitability of the marker modality should be considered when selecting a biomarker. In mental health conditions, the development of a digital marker captured by remote monitoring might aid in diagnosis by adding information to the self-reporting of symptoms from a patient, for example, if the marker can act as a proxy for behavioural signs of mental illness that cannot be captured by a single measurement of clinical state when consulting a clinician. The development of phone-based applications allows clinicians to collect data on changes through a series of symptom-based questions [ 150 ]. However, the future of biomarker discovery is likely the ability to measure, compare and combine multiple variables and here, resources are key. The Penn Medicine Biobank [ 151 ] includes genetics and biomarkers alongside EMR to enable precision medicine and the discovery of new phenotypes.

Interrogation of EMRs has revealed the potential value of routinely recorded data to identify and validate the use of existing and exploratory biomarkers. For example, in a study of sepsis, biomarkers were used alongside EMR to study progression [ 152 ]. Biomarkers were employed to study different time periods whereby early-life mental health impacts midlife using a panel of markers [ 153 ]. Elsewhere the combination of biomarkers and EMR has been recommended for aiding risk reduction profiles and identifying new clinical biomarkers [ 154 ].

The use of genetics for large-scale psychiatry research

The observational nature of many findings precludes drawing firm conclusions about causality due to residual confounding and reverse causation. With the recent explosion of large-scale genetic data available, methods using this data, such as polygenetic risk scores (PRS) and Mendelian randomisation (MR) allow the elucidation of causal relationships between psychiatric disorders, risk factors and drug treatments. Leveraging genetic data using PRS and MR offers a cost-effective approach with the future potential to embed genetic data into healthcare settings to help improve patient care [ 155 ] and could provide complementary evidence as part of the triangulation process.

PRS are weighted sums of genetic variants associated with a particular condition [ 156 ]. Therefore, PRS can estimate the genetic risk of an individual for a disease or trait. Due to the complex polygenic nature of many conditions (including mental health and neurological conditions [ 157 , 158 , 159 , 160 , 161 ], PRS can only capture a fraction of the overall risk, with clinical and demographic factors usually explaining most variance. This means that on their own, PRS is unlikely to definitively predict future diagnoses in a healthcare setting [ 162 , 163 ]. However, PRS could be included with other measures to predict future risk and may show promise in aiding clinical decision-making. Adding PRS to risk prediction models alongside other clinical factors such as age, gender and family history has been shown to improve model performance for predicting the risk of conditions such as dementia [ 164 ] and certain mental health conditions [ 162 , 165 , 166 ]. On top of this, PRS may have some potential to inform treatment response if polygenetic complex traits can be predicted from an individual’s genetics [ 167 ] and those traits are robustly associated with treatment response. PRS can be used in conjunction with other methods, such as Mendelian randomisation [ 168 ], to uncover casual insights between complex psychiatric traits and treatments.

Mendelian randomisation (MR) is a statistical approach that uses genetic variants to provide evidence that an exposure has a causal effect on an outcome [ 169 , 170 , 171 , 172 , 173 ] (Fig. 2 ). A genetic variant (or variants) is used which is associated with an exposure (e.g., insomnia) but not associated with any other risk factors which affect the outcome (e.g., depression). By doing so, any association of the genetic variant(s) with the outcome must act via the variants’ association with the exposure and imply a causal effect of the exposure on the outcome. As genetic variants are inherited randomly at conception, genetic variants are not susceptible to reverse causation and confounding, like observational studies using EMR. Results from MR can help support results from EMR by using data triangulation, as discussed previously.

There are a growing number of MR studies being published that show causal effects related to disease-disease associations and drug-disease associations for mental health and old age psychiatry [ 179 , 180 , 181 , 182 , 183 , 184 , 185 , 186 ] with extensions to traditional MR approaches, which could offer further insights [ 187 , 188 , 189 ].

Large-scale research approaches are at the forefront of EMR use in psychiatry. With the advances in interpretation using NLP and access to diverse data resources, the scope of research questions is rapidly expanding. However, care is needed to make sure that potential biases are considered. Not considering limitations with big data can lead to incorrect inferences about a population which could mean poorer care for high-risk populations such as those with mental health conditions and neurodegenerative conditions.

In order to optimise EMR for psychiatry a clear understanding of such biases in the data is vital. A researcher must carefully consider if the research question can be answered in the data source they want to use and develop the best study design and statistical analysis. By cautiously incorporating the strengths of the EMR format it will be possible to make exciting contributions to mental health and neurological research.

Koning NR, Büchner FL, van den Berg AW, Choi SYA, Leeuwenburgh NA, Paijmans IJM, et al. The usefulness of electronic health records from preventive youth healthcare in the recognition of child mental health problems. Front Public Health 2021;9. https://doi.org/10.3389/fpubh.2021.658240 .

Werbeloff N, Osborn DPJ, Patel R, Taylor M, Stewart R, Broadbent M, et al. The Camden & Islington Research Database: using electronic mental health records for research. PLoS ONE. 2018;13:e0190703.

Article   PubMed   PubMed Central   Google Scholar  

Patel R, Wee SN, Ramaswamy R, Thadani S, Tandi J, Garg R, et al. NeuroBlu, an electronic health record (EHR) trusted research environment (TRE) to support mental healthcare analytics with real-world data. BMJ Open. 2022;12:e057227.

Castillo EG, Olfson M, Pincus HA, Vawdrey D, Stroup TS. Electronic health records in mental health research: a framework for developing valid research methods. Psychiatr Serv. 2015;66:193–6.

Article   PubMed   Google Scholar  

Power MC, Engelman BC, Wei J, Glymour MM. Closing the gap between observational research and randomized controlled trials for prevention of Alzheimer disease and dementia. Epidemiol Rev. 2022;44:17–28.

Jankovic J, Parsons J, Jovanović N, Berrisford G, Copello A, Fazil Q, et al. Differences in access and utilisation of mental health services in the perinatal period for women from ethnic minorities—a population-based study. BMC Med. 2020;18:245.

Dennis M, Shine L, John A, Marchant A, McGregor J, Lyons RA, et al. Risk of adverse outcomes for older people with dementia prescribed antipsychotic medication: a population based e-cohort study. Neurol Ther. 2017;6:57–77.

Heikal SA, Salama M, Richard Y, Moustafa AA, Lawlor B. The impact of disease registries on advancing knowledge and understanding of dementia globally. Front Aging Neurosci 2022;14. https://www.frontiersin.org/articles/10.3389/fnagi.2022.774005 .

Hoque DME, Kumari V, Hoque M, Ruseckaite R, Romero L, Evans SM. Impact of clinical registries on quality of patient care and clinical outcomes: a systematic review. PLoS ONE. 2017;12:e0183667.

Berger K, Rietschel M, Rujescu D. The value of ‘mega cohorts’ for psychiatric research. World J Biol Psychiatry. 2023;24:860–4.

Davis K, Hotopf M. Mental health phenotyping in UK Biobank. Prog Neurol Psychiatry. 2019;23:4–7.

Article   Google Scholar  

Fry A, Littlejohns TJ, Sudlow C, Doherty N, Adamska L, Sprosen T, et al. Comparison of sociodemographic and health-related characteristics of UK Biobank participants with those of the general population. Am J Epidemiol. 2017;186:1026–34.

Davis KAS, Coleman JRI, Adams M, Allen N, Breen G, Cullen B, et al. Mental health in UK Biobank—development, implementation and results from an online questionnaire completed by 157 366 participants: a reanalysis. BJPsych Open. 2020;6:e18.

Bauer M, Glenn T, Geddes J, Gitlin M, Grof P, Kessing LV, et al. Smartphones in mental health: a critical review of background issues, current status and future concerns. Int J Bipolar Disord. 2020;8:2.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Brietzke E, Hawken ER, Idzikowski M, Pong J, Kennedy SH, Soares CN. Integrating digital phenotyping in clinical characterization of individuals with mood disorders. Neurosci Biobehav Rev. 2019;104:223–30.

Rykov Y, Thach T-Q, Bojic I, Christopoulos G, Car J. Digital biomarkers for depression screening with wearable devices: cross-sectional study with machine learning modeling. JMIR MHealth UHealth. 2021;9:e24872.

Lee S, Kim H, Park MJ, Jeon HJ. Current advances in wearable devices and their sensors in patients with depression. Front Psychiatry 2021;12. https://www.frontiersin.org/articles/10.3389/fpsyt.2021.672347 .

Torous J, Bucci S, Bell IH, Kessing LV, Faurholt‐Jepsen M, Whelan P, et al. The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality. World Psychiatry. 2021;20:318–35.

Merkow RP, Schwartz TA, Nathens AB. Practical guide to comparative effectiveness research using observational data. JAMA Surg. 2020;155:349–50.

Berger ML, Dreyer N, Anderson F, Towse A, Sedrakyan A, Normand S-L. Prospective observational studies to assess comparative effectiveness: the ISPOR Good Research Practices Task Force Report. Value Health. 2012;15:217–30.

Hammond GC, Croudace TJ, Radhakrishnan M, Lafortune L, Watson A, McMillan-Shields F, et al. Comparative effectiveness of cognitive therapies delivered face-to-face or over the telephone: an observational study using propensity methods. PLoS ONE. 2012;7:e42916.

Crescenzo FD, Garriga C, Tomlinson A, Coupland C, Efthimiou O, Fazel S, et al. Real-world effect of antidepressants for depressive disorder in primary care: protocol of a population-based cohort study. BMJ Ment Health. 2020;23:122–6.

Google Scholar  

Alemi F, Min H, Yousefi M, Becker LK, Hane CA, Nori VS et al. Effectiveness of common antidepressants: a post market release study. eClinicalMedicine 2021;41. https://doi.org/10.1016/j.eclinm.2021.101171 .

Centorrino F, Meyers AL, Ahl J, Cincotta SL, Zun L, Gulliver AH, et al. An observational study of the effectiveness and safety of intramuscular olanzapine in the treatment of acute agitation in patients with bipolar mania or schizophrenia/schizoaffective disorder. Hum Psychopharmacol Clin Exp. 2007;22:455–62.

Article   CAS   Google Scholar  

Singh B, Kung S, Pazdernik V, Schak KM, Geske J, Schulte PJ, et al. Comparative effectiveness of intravenous ketamine and intranasal esketamine in clinical practice among patients with treatment-refractory depression: an Observational Study. J Clin Psychiatry. 2023;84:45331.

Touriño AG, Feixas G, Medina JC, Paz C, Evans C. Effectiveness of integrated treatment for eating disorders in Spain: protocol for a multicentre, naturalistic, observational study. BMJ Open. 2021;11:e043152.

Ayton A, Ibrahim A, Dugan J, Galvin E, Wright OW. Ultra-processed foods and binge eating: a retrospective observational study. Nutrition. 2021;84:111023.

Molero Y, Kaddoura S, Kuja-Halkola R, Larsson H, Lichtenstein P, D’Onofrio BM, et al. Associations between β-blockers and psychiatric and behavioural outcomes: a population-based cohort study of 1.4 million individuals in Sweden. PLoS Med. 2023;20:e1004164.

Newby D, Linden AB, Fernandes M, Molero Y, Winchester L, Sproviero W, et al. Comparative effect of metformin versus sulfonylureas with dementia and Parkinson’s disease risk in US patients over 50 with type 2 diabetes mellitus. BMJ Open Diabetes Res Care. 2022;10:e003036.

Newby D, Prieto-Alhambra D, Duarte-Salles T, Ansell D, Pedersen L, van der Lei J, et al. Methotrexate and relative risk of dementia amongst patients with rheumatoid arthritis: a multi-national multi-database case-control study. Alzheimers Res Ther. 2020;12:38.

Desai RJ, Mahesri M, Lee SB, Varma VR, Loeffler T, Schilcher I, et al. No association between initiation of phosphodiesterase-5 inhibitors and risk of incident Alzheimer’s disease and related dementia: results from the Drug Repurposing for Effective Alzheimer’s Medicines study. Brain Commun. 2022;4:fcac247.

Desai RJ, Varma VR, Gerhard T, Segal J, Mahesri M, Chin K, et al. Comparative risk of Alzheimer disease and related dementia among medicare beneficiaries with rheumatoid arthritis treated with targeted disease-modifying antirheumatic agents. JAMA Netw Open. 2022;5:e226567.

Ding J, Davis-Plourde KL, Sedaghat S, Tully PJ, Wang W, Phillips C, et al. Antihypertensive medications and risk for incident dementia and Alzheimer’s disease: a meta-analysis of individual participant data from prospective cohort studies. Lancet Neurol. 2020;19:61–70.

Article   CAS   PubMed   Google Scholar  

Ballard C, Aarsland D, Cummings J, O’Brien J, Mills R, Molinuevo JL, et al. Drug repositioning and repurposing for Alzheimer disease. Nat Rev Neurol. 2020;16:661–73.

Livingston G, Huntley J, Sommerlad A, Ames D, Ballard C, Banerjee S, et al. Dementia prevention, intervention, and care: 2020 report of the Lancet Commission. The Lancet. 2020;396:413–46.

Coupland CAC, Hill T, Dening T, Morriss R, Moore M, Hippisley-Cox J. Anticholinergic drug exposure and the risk of dementia: a Nested Case-Control Study. JAMA Intern Med. 2019;179:1084–93.

Group BMJP. Anticholinergic drugs and risk of dementia: case-control study. BMJ. 2019;367:l6213.

Joyce G, Ferido P, Thunell J, Tysinger B, Zissimopoulos J. Benzodiazepine use and the risk of dementia. Alzheimers Dement Transl Res Clin Interv. 2022;8:e12309.

He Q, Chen X, Wu T, Li L, Fei X. Risk of dementia in long-term benzodiazepine users: evidence from a meta-analysis of observational studies. J Clin Neurol Seoul Korea. 2019;15:9–19.

Fox MP, Murray EJ, Lesko CR, Sealy-Jefferson S. On the need to revitalize descriptive epidemiology. Am J Epidemiol. 2022;191:1174–9.

Lesko CR, Fox MP, Edwards JK. A framework for descriptive epidemiology. Am J Epidemiol. 2022;191:2063–70.

Moreno-Agostino D, Wu Y-T, Daskalopoulou C, Hasan MT, Huisman M, Prina M. Global trends in the prevalence and incidence of depression: a systematic review and meta-analysis. J Affect Disord. 2021;281:235–43.

Kessler RC, Wang PS. The descriptive epidemiology of commonly occurring mental disorders in the United States. Annu Rev Public Health. 2008;29:115–29.

Stauffacher MWD, Stiefel F, Dorogi Y, Michaud L. Observational study of suicide in Switzerland: comparison between psychiatric in- and outpatients. Swiss Med Wkly. 2022;152:w30140.

Mejareh ZN, Abdollahi B, Hoseinipalangi Z, Jeze MS, Hosseinifard H, Rafiei S, et al. Global, regional, and national prevalence of depression among cancer patients: a systematic review and meta-analysis. Indian J Psychiatry. 2021;63:527–35.

Santomauro DF, Herrera AMM, Shadid J, Zheng P, Ashbaugh C, Pigott DM, et al. Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic. The Lancet. 2021;398:1700–12.

Raventós B, Pistillo A, Reyes C, Fernández-Bertolín S, Aragón M, Berenguera A, et al. Impact of the COVID-19 pandemic on diagnoses of common mental health disorders in adults in Catalonia, Spain: a population-based cohort study. BMJ Open. 2022;12:e057866.

Walby FA, Myhre MØ, Mehlum L. Suicide among users of mental health and addiction services in the first 10 months of the COVID-19 pandemic: observational study using national registry data. BJPsych Open. 2022;8:e111.

Doherty AS, Shahid F, Moriarty F, Boland F, Clyne B, Dreischulte T, et al. Prescribing cascades in community‐dwelling adults: a systematic review. Pharmacol Res Perspect. 2022;10:e01008.

Luo Y, Kataoka Y, Ostinelli EG, Cipriani A, Furukawa TA. National prescription patterns of antidepressants in the treatment of adults with major depression in the US between 1996 and 2015: a population representative survey based analysis. Front Psychiatry. 2020;11:35.

McCarthy LM, Savage R, Dalton K, Mason R, Li J, Lawson A, et al. ThinkCascades: a tool for identifying clinically important prescribing cascades affecting older people. Drugs Aging. 2022;39:829–40.

Trenaman SC, Bowles SK, Kirkland S, Andrew MK. An examination of three prescribing cascades in a cohort of older adults with dementia. BMC Geriatr. 2021;21:297.

Meehan AJ, Lewis SJ, Fazel S, Fusar-Poli P, Steyerberg EW, Stahl D, et al. Clinical prediction models in psychiatry: a systematic review of two decades of progress and challenges. Mol Psychiatry. 2022;27:2700–8.

Salazar de Pablo G, Studerus E, Vaquerizo-Serrano J, Irving J, Catalan A, Oliver D, et al. Implementing precision psychiatry: a systematic review of individualized prediction models for clinical practice. Schizophr Bull. 2021;47:284–97.

Gradus JL, Rosellini AJ, Horváth-Puhó E, Jiang T, Street AE, Galatzer-Levy I, et al. Predicting sex-specific nonfatal suicide attempt risk using machine learning and data from Danish National Registries. Am J Epidemiol. 2021;190:2517–27.

Jørgensen TSH, Osler M, Jorgensen MB, Jorgensen A. Mapping diagnostic trajectories from the first hospital diagnosis of a psychiatric disorder: a Danish nationwide cohort study using sequence analysis. Lancet Psychiatry. 2023;10:12–20.

Sajjadian M, Lam RW, Milev R, Rotzinger S, Frey BN, Soares CN, et al. Machine learning in the prediction of depression treatment outcomes: a systematic review and meta-analysis. Psychol Med. 2021;51:2742–51.

Vyas A, Aisopos F, Vidal M-E, Garrard P, Paliouras G. Identifying the presence and severity of dementia by applying interpretable machine learning techniques on structured clinical records. BMC Med Inform Decis Mak. 2022;22:271.

Zhang Y, Wang S, Hermann A, Joly R, Pathak J. Development and validation of a machine learning algorithm for predicting the risk of postpartum depression among pregnant women. J Affect Disord. 2021;279:1–8.

Huang SH, LePendu P, Iyer SV, Tai-Seale M, Carrell D, Shah NH. Toward personalizing treatment for depression: predicting diagnosis and severity. J Am Med Inform Assoc JAMIA. 2014;21:1069–75.

Bayramli I, Castro V, Barak-Corren Y, Madsen EM, Nock MK, Smoller JW, et al. Predictive structured-unstructured interactions in EHR models: a case study of suicide prediction. NPJ Digit Med. 2022;5:15.

De Mauro A, Greco M, Grimaldi M. A formal definition of Big Data based on its essential features. Libr Rev. 2016;65:122–35.

Pastorino R, De Vito C, Migliara G, Glocker K, Binenbaum I, Ricciardi W, et al. Benefits and challenges of Big Data in healthcare: an overview of the European initiatives. Eur J Public Health. 2019;29:23–27.

Monteith S, Glenn T, Geddes J, Bauer M. Big data are coming to psychiatry: a general introduction. Int J Bipolar Disord. 2015;3:21.

Meng X-L. Statistical paradises and paradoxes in big data (I): law of large populations, big data paradox, and the 2016 US presidential election. Ann Appl Stat. 2018;12:685–726.

Msaouel P. The big data paradox in clinical practice. Cancer Investig. 2022;40:567–76.

Lazer D, Kennedy R, King G, Vespignani A. The parable of Google flu: traps in big data analysis. Science. 2014;343:1203–5.

Bradley VC, Kuriwaki S, Isakov M, Sejdinovic D, Meng X-L, Flaxman S. Unrepresentative big surveys significantly overestimated US vaccine uptake. Nature. 2021;600:695–700.

Tayefi M, Ngo P, Chomutare T, Dalianis H, Salvi E, Budrionis A, et al. Challenges and opportunities beyond structured data in analysis of electronic health records. WIREs Comput Stat 2021;13. https://doi.org/10.1002/wics.1549 .

Bodenreider O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32:D267–D270.

Searle T, Ibrahim Z, Teo J, Dobson R. Estimating redundancy in clinical text. J Biomed Inform. 2021;124:103938.

Charniak E. Statistical language learning (language, speech, and communication). (The MIT Press; 1996).

Jones KS. Natural language processing: a historical review. In: Zampolli A, Calzolari N, Palmer M, editors. Current issues in computational linguistics: in honour of Don Walker. Springer Netherlands: Dordrecht; 1994. pp. 3–16.

Devlin J, Chang M-W, Lee K, Toutanova K. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” North American Chapter of the Association for Computational Linguistics. 2019.

Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P et al. Language models are few-shot learners. [Preprint]. 2020. https://doi.org/10.48550/arXiv.2005.14165 .

Chan JY-L, Bea KT, Leow SMH, Phoong SW, Cheng WK. State of the art: a review of sentiment analysis based on sequential transfer learning. Artif Intell Rev. 2023;56:749–80.

Mars M. From word embeddings to pre-trained language models: a state-of-the-art walkthrough. Appl Sci. 2022;12:8805.

Kraljevic Z, Searle T, Shek A, Roguski L, Noor K, Bean D et al. Multi-domain clinical natural language processing with MedCAT: the Medical Concept Annotation Toolkit. [Preprint] 2021. https://doi.org/10.48550/arXiv.2010.01165 .

Kormilitzin A, Vaci N, Liu Q, Nevado-Holgado A. Med7: a transferable clinical natural language processing model for electronic health records. Artif Intell Med. 2021;118:102086.

Bender EM, Gebru T, McMillan-Major A, Shmitchell S. On the dangers of stochastic parrots: can language models be too big? In Conference on Fairness, Accountability, and Transparency(FAccT ’21), March 3–10, 2021, Virtual Event, Canada. 14 pages (ACM, New York, NY, USA, 2021). https://doi.org/10.1145/3442188.3445922 .

Sharir O, Peleg B, Shoham Y. The cost of training NLP models: a concise overview. [Preprint] 2020. http://arxiv.org/abs/2004.08900 .

Joyce DW, Kormilitzin A, Smith KA, Cipriani A. Explainable artificial intelligence for mental health through transparency and interpretability for understandability. Npj Digit Med. 2023;6:1–7.

Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell. 2019;1:206–15.

Li B, Qi P, Liu B, Di S, Liu J, Pei J, et al. Trustworthy AI: from principles to practices. ACM Comput Surv. 2023;55:177.

Gilbert R, Lafferty R, Hagger-Johnson G, Harron K, Zhang L-C, Smith P, et al. GUILD: GUidance for Information about Linking Data sets†. J Public Health. 2018;40:191–8.

Green E, Ritchie F, Mytton J, Webber DJ, Deave T, Montgomery A, et al. Enabling data linkage to maximise the value of public health research data: summary report. 2015. https://uwe-repository.worktribe.com/output/836851/enabling-data-linkage-to-maximise-the-value-of-public-health-research-data-summary-report .

Perera G, Broadbent M, Callard F, Chang C-K, Downs J, Dutta R, et al. Cohort profile of the South London and Maudsley NHS Foundation Trust Biomedical Research Centre (SLaM BRC) case register: current status and recent enhancement of an electronic mental health record-derived data resource. BMJ Open. 2016;6:e008721.

Carson L, Jewell A, Downs J, Stewart R. Multisite data linkage projects in mental health research. Lancet Psychiatry. 2020;7:e61.

Hagger-Johnson G, Harron K, Fleming T, Gilbert R, Goldstein H, Landy R, et al. Data linkage errors in hospital administrative data when applying a pseudonymisation algorithm to paediatric intensive care records. BMJ Open. 2015;5:e008118.

Bohensky MA, Jolley D, Sundararajan V, Evans S, Pilcher DV, Scott I, et al. Data Linkage: a powerful research tool with potential problems. BMC Health Serv Res. 2010;10:346.

Grath-Lone LM, Libuy N, Etoori D, Blackburn R, Gilbert R, Harron K. Ethnic bias in data linkage. Lancet Digit Health. 2021;3:e339.

Patten SB. Selection bias in studies of major depression using clinical subjects. J Clin Epidemiol. 2000;53:351–7.

Yu S. Uncovering the hidden impacts of inequality on mental health: a global study. Transl Psychiatry. 2018;8:98.

Semlyen J, King M, Varney J, Hagger-Johnson G. Sexual orientation and symptoms of common mental disorder or low wellbeing: combined meta-analysis of 12 UK population health surveys. BMC Psychiatry. 2016;16:67.

Reiss F, Meyrose A-K, Otto C, Lampert T, Klasen F, Ravens-Sieberer U. Socioeconomic status, stressful life situations and mental health problems in children and adolescents: results of the German BELLA cohort-study. PLoS ONE. 2019;14:e0213700.

Howe CJ, Robinson WR. Survival-related selection bias in studies of racial health disparities: the importance of the target population and study design. Epidemiol Camb Mass. 2018;29:521–4.

Thornton J. Ethnic minority patients receive worse mental healthcare than white patients, review finds. BMJ. 2020;368:m1058.

Pendergrass SA, Crawford DC. Using electronic health records to generate phenotypes for research. Curr Protoc Hum Genet. 2019;100:e80.

Hemingway H, Asselbergs FW, Danesh J, Dobson R, Maniadakis N, Maggioni A, et al. Big data from electronic health records for early and late translational cardiovascular research: challenges and potential. Eur Heart J. 2018;39:1481–95.

Wisniewski H, Henson P, Torous J. Using a smartphone app to identify clinically relevant behavior trends via symptom report, cognition scores, and exercise levels: a case series. Front Psychiatry. 2019;10:652.

Yang S, Varghese P, Stephenson E, Tu K, Gronsbell J. Machine learning approaches for electronic health records phenotyping: a methodical review. J Am Med Inform Assoc JAMIA. 2023;30:367–81.

Banda JM, Seneviratne M, Hernandez-Boussard T, Shah NH. Advances in electronic phenotyping: from rule-based definitions to machine learning models. Annu Rev Biomed Data Sci. 2018;1:53.

Gottesman O, Kuivaniemi H, Tromp G, Faucett WA, Li R, Manolio TA, et al. The Electronic Medical Records and Genomics (eMERGE) network: past, present, and future. Genet Med. 2013;15:761–71.

Denaxas S, Gonzalez-Izquierdo A, Direk K, Fitzpatrick NK, Fatemifar G, Banerjee A, et al. UK phenomics platform for developing and validating electronic health record phenotypes: CALIBER. J Am Med Inform Assoc. 2019;26:1545–59.

Tomlinson A, Furukawa TA, Efthimiou O, Salanti G, Crescenzo FD, Singh I, et al. Personalise antidepressant treatment for unipolar depression combining individual choices, risks and big data (PETRUSHKA): rationale and protocol. BMJ Ment Health. 2020;23:52–56.

Ingram WM, Baker AM, Bauer CR, Brown JP, Goes FS, Larson S, et al. Defining major depressive disorder cohorts using the EHR: multiple phenotypes based on ICD-9 codes and medication orders. Neurol Psychiatry Brain Res. 2020;36:18–26.

Mayer MA, Gutierrez-Sacristan A, Leis A, De La Peña S, Sanz F, Furlong LI. Using electronic health records to assess depression and cancer comorbidities. Stud Health Technol Inform. 2017;235:236–40.

PubMed   Google Scholar  

Deferio JJ, Levin TT, Cukor J, Banerjee S, Abdulrahman R, Sheth A, et al. Using electronic health records to characterize prescription patterns: focus on antidepressants in nonpsychiatric outpatient settings. JAMIA Open. 2018;1:233.

James G, Collin E, Lawrance M, Mueller A, Podhorna J, Zaremba-Pechmann L, et al. Treatment pathway analysis of newly diagnosed dementia patients in four electronic health record databases in Europe. Soc Psychiatry Psychiatr Epidemiol. 2021;56:409–16.

Smoller JW. The use of electronic health records for psychiatric phenotyping and genomics. Am J Med Genet Part B. 2018;177:601.

Ford E, Carroll JA, Smith HE, Scott D, Cassell JA. Extracting information from the text of electronic medical records to improve case detection: a systematic review. J Am Med Inform Assoc JAMIA. 2016;23:1007–15.

Moldwin A, Demner-Fushman D, Goodwin TR. Empirical findings on the role of structured data, unstructured data, and their combination for automatic clinical phenotyping. AMIA Summits Transl Sci Proc. 2021;2021:445–54.

PubMed   PubMed Central   Google Scholar  

Morley KI, Wallace J, Denaxas SC, Hunter RJ, Patel RS, Perel P, et al. Defining disease phenotypes using national linked electronic health records: a case study of atrial fibrillation. PLoS ONE. 2014;9:e110900.

Kirby JC, Speltz P, Rasmussen LV, Basford M, Gottesman O, Peissig PL, et al. PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportability. J Am Med Inform Assoc JAMIA. 2016;23:1046–52.

Newton KM, Peissig PL, Kho AN, Bielinski SJ, Berg RL, Choudhary V, et al. Validation of electronic medical record-based phenotyping algorithms: results and lessons learned from the eMERGE network. J Am Med Inform Assoc JAMIA. 2013;20:e147–154.

McGuinness LA, Warren‐Gash C, Moorhouse LR, Thomas SL. The validity of dementia diagnoses in routinely collected electronic health records in the United Kingdom: a systematic review. Pharmacoepidemiol Drug Saf. 2019;28:244–55.

Bellows BK, LaFleur J, Kamauu AWC, Ginter T, Forbush TB, Agbor S, et al. Automated identification of patients with a diagnosis of binge eating disorder from narrative electronic health records. J Am Med Inform Assoc JAMIA. 2014;21:e163–e168.

Hernán MA, Wang W, Leaf DE. Target trial emulation: a framework for causal inference from observational data. JAMA. 2022;328:2446–7.

Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol. 2016;183:758–64.

Hernán MA, Sauer BC, Hernández-Díaz S, Platt R, Shrier I. Specifying a target trial prevents immortal time bias and other self-inflicted injuries in observational analyses. J Clin Epidemiol. 2016;79:70–75.

Admon AJ, Donnelly JP, Casey JD, Janz DR, Russell DW, Joffe AM, et al. Emulating a novel clinical trial using existing observational data. predicting results of the prevent study. Ann Am Thorac Soc. 2019;16:998–1007.

Dickerman BA, García-Albéniz X, Logan RW, Denaxas S, Hernán MA. Avoidable flaws in observational analyses: an application to statins and cancer. Nat Med. 2019;25:1601–6.

Boyne DJ, Cheung WY, Hilsden RJ, Sajobi TT, Batra A, Friedenreich CM, et al. Association of a shortened duration of adjuvant chemotherapy with overall survival among individuals with Stage III colon cancer. JAMA Netw Open. 2021;4:e213587.

Matthews AA, Szummer K, Dahabreh IJ, Lindahl B, Erlinge D, Feychting M, et al. Comparing effect estimates in randomized trials and observational studies from the same population: an application to percutaneous coronary intervention. J Am Heart Assoc. 2021;10:e020357.

Hoffman KL, Schenck EJ, Satlin MJ, Whalen W, Pan D, Williams N, et al. Comparison of a target trial emulation framework vs. Cox regression to estimate the association of corticosteroids with COVID-19 mortality. JAMA Netw Open. 2022;5:e2234425.

Caniglia EC, Rojas-Saunero LP, Hilal S, Licher S, Logan R, Stricker B, et al. Emulating a target trial of statin use and risk of dementia using cohort data. Neurology. 2020;95:e1322–e1332.

Ahn N, Nolde M, Günter A, Güntner F, Gerlach R, Tauscher M, et al. Emulating a target trial of proton pump inhibitors and dementia risk using claims data. Eur J Neurol. 2022;29:1335–43.

Chen Y-CB, Liang C-S, Wang L-J, Hung K-C, Carvalho AF, Solmi M, et al. Comparative effectiveness of valproic acid in different serum concentrations for maintenance treatment of bipolar disorder: a retrospective cohort study using target trial emulation framework. EClinicalMedicine. 2022;54:101678.

Sun JW, Young JG, Sarvet AL, Bailey LC, Heerman WJ, Janicke DM, et al. Comparison of rates of type 2 diabetes in adults and children treated with anticonvulsant mood stabilizers. JAMA Netw Open. 2022;5:e226484.

Tian Y, Schuemie MJ, Suchard MA. Evaluating large-scale propensity score performance through real-world and synthetic data experiments. Int J Epidemiol. 2018;47:2005–14.

Austin PC. An Introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivar Behav Res. 2011;46:399–424.

Riley RD, Ensor J, Snell KIE, Harrell FE, Martin GP, Reitsma JB, et al. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020;368:m441.

Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. Springer International Publishing: Cham; 2019.

Dalton JE, Bolen SD, Mascha EJ. Publication bias: the elephant in the review. Anesth Analg. 2016;123:812–3.

Nguyen VT, Engleton M, Davison M, Ravaud P, Porcher R, Boutron I. Risk of bias in observational studies using routinely collected data of comparative effectiveness research: a meta-research study. BMC Med. 2021;19:279.

von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, et al. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet Lond Engl. 2007;370:1453–7.

Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med. 2015;13:1.

Williams RJ, Tse T, Harlan WR, Zarin DA. Registration of observational studies: Is it time?. Can Med Assoc J. 2010;182:1638–42.

Thor M, Oh JH, Apte AP, Deasy JO. Registering study analysis plans (SAPs) before dissecting your data—updating and standardizing outcome modeling. Front Oncol. 2020;10:978.

Goldacre B, Morton CE, DeVito NJ. Why researchers should share their analytic code. BMJ. 2019;367:l6365.

Koutsouleris N, Hauser TU, Skvortsova V, Choudhury MD. From promise to practice: towards the realisation of AI-informed mental health care. Lancet Digit Health. 2022;4:e829–e840.

Lawlor DA, Tilling K, Davey Smith G. Triangulation in aetiological epidemiology. Int J Epidemiol. 2016;45:1866–86.

Hammerton G, Munafò MR. Causal inference with observational data: the need for triangulation of evidence. Psychol Med. 2021;51:563–78.

Fancourt D, Steptoe A. Cultural engagement and mental health: does socio-economic status explain the association? Soc Sci Med 1982. 2019;236:112425.

Harrison R, Munafò MR, Davey Smith G, Wootton RE. Examining the effect of smoking on suicidal ideation and attempts: triangulation of epidemiological approaches. Br J Psychiatry J Ment Sci. 2020;217:701–7.

Lloyd EC, Sallis HM, Verplanken B, Haase AM, Munafò MR. Understanding the nature of association between anxiety phenotypes and anorexia nervosa: a triangulation approach. BMC Psychiatry. 2020;20:495.

Gaetani L, Blennow K, Calabresi P, Filippo MD, Parnetti L, Zetterberg H. Neurofilament light chain as a biomarker in neurological disorders. J Neurol Neurosurg Psychiatry. 2019;90:870–81.

Carvalho AF, Solmi M, Sanches M, Machado MO, Stubbs B, Ajnakina O, et al. Evidence-based umbrella review of 162 peripheral biomarkers for major mental disorders. Transl Psychiatry. 2020;10:1–13.

García-Gutiérrez MS, Navarrete F, Sala F, Gasparyan A, Austrich-Olivares A, Manzanares J. Biomarkers in psychiatry: concept, definition, types and relevance to the clinical reality. Front Psychiatry 2020;11. https://www.frontiersin.org/articles/10.3389/fpsyt.2020.00432 .

Goodday SM, Atkinson L, Goodwin G, Saunders K, South M, Mackay C, et al. The true colours remote symptom monitoring system: a decade of evolution. J Med Internet Res. 2020;22:e15188.

Verma A, Damrauer SM, Naseer N, Weaver J, Kripke CM, Guare L, et al. The Penn Medicine BioBank: towards a genomics-enabled learning healthcare system to accelerate precision medicine in a diverse population. J Pers Med. 2022;12:1974.

Taneja I, Reddy B, Damhorst G, Dave Zhao S, Hassan U, Price Z, et al. Combining biomarkers with EMR data to identify patients in different phases of sepsis. Sci Rep. 2017;7:10800.

Ploubidis GB, Batty GD, Patalay P, Bann D, Goodman A. Association of early-life mental health with biomarkers in midlife and premature mortality: evidence from the 1958 British Birth Cohort. JAMA Psychiatry. 2021;78:38–46.

Ryan D, Blakey J, Chisholm A, Price D, Thomas M, Ställberg B, et al. Use of electronic medical records and biomarkers to manage risk and resource efficiencies. Eur Clin Respir J. 2017;4:1293386.

Stark Z, Dolman L, Manolio TA, Ozenberger B, Hill SL, Caulfied MJ, et al. Integrating genomics into healthcare: a global responsibility. Am J Hum Genet. 2019;104:13–20.

Wray NR, Lin T, Austin J, McGrath JJ, Hickie IB, Murray GK, et al. From basic science to clinical application of polygenic risk scores: a primer. JAMA Psychiatry. 2021;78:101–9.

Levine ME, Crimmins EM, Prescott CA, Phillips D, Arpawong TE, Lee J. A polygenic risk score associated with measures of depressive symptoms among older adults. Biodemography Soc Biol. 2014;60:199–211.

Baker E, Escott-Price V. Polygenic risk scores in Alzheimer’s disease: current applications and future directions. Front Digit Health 2020;2. https://www.frontiersin.org/articles/10.3389/fdgth.2020.00014 .

Escudero I, Johnstone M. Genetics of schizophrenia. Curr Psychiatry Rep. 2014;16:502.

Smeland OB, Frei O, Dale AM, Andreassen OA. The polygenic architecture of schizophrenia—rethinking pathogenesis and nosology. Nat Rev Neurol. 2020;16:366–79.

Lopes FL, Zhu K, Purves KL, Song C, Ahn K, Hou L, et al. Polygenic risk for anxiety influences anxiety comorbidity and suicidal behavior in bipolar disorder. Transl Psychiatry. 2020;10:1–7.

Murray GK, Lin T, Austin J, McGrath JJ, Hickie IB, Wray NR. Could polygenic risk scores be useful in psychiatry?: a review. JAMA Psychiatry. 2021;78:210–9.

Kember RL, Merikangas AK, Verma SS, Verma A, Judy R, Abecasis G, et al. Polygenic risk of psychiatric disorders exhibits cross-trait associations in electronic health record data from European ancestry individuals. Biol Psychiatry. 2021;89:236–45.

Gao XR, Chiariglione M, Qin K, Nuytemans K, Scharre DW, Li Y-J, et al. Explainable machine learning aggregates polygenic risk scores and electronic health records for Alzheimer’s disease prediction. Sci Rep. 2023;13:450.

Wang M, Hu K, Fan L, Yan H, Li P, Jiang T et al. Predicting treatment response in schizophrenia with magnetic resonance imaging and polygenic risk score. Front Genet 2022;13. https://www.frontiersin.org/articles/10.3389/fgene.2022.848205 .

Landi I, Kaji DA, Cotter L, Van Vleck T, Belbin G, Preuss M, et al. Prognostic value of polygenic risk scores for adults with psychosis. Nat Med. 2021;27:1576–81.

Meerman JJ, ter Hark SE, Janzing JGE, Coenen MJH. The potential of polygenic risk scores to predict antidepressant treatment response in major depression: a systematic review. J Affect Disord. 2022;304:1–11.

Burgess S, Thompson SG. Use of allele scores as instrumental variables for Mendelian randomization. Int J Epidemiol. 2013;42:1134–44.

Wootton RE, Jones HJ, Sallis HM. Mendelian randomisation for psychiatry: how does it work, and what can it tell us? Mol Psychiatry. 2022;27:53–57.

Davies NM, Holmes MV, Smith GD. Reading Mendelian randomisation studies: a guide, glossary, and checklist for clinicians. BMJ. 2018;362:k601.

Cai L, Bao Y, Fu X, Cao H, Baranova A, Zhang X, et al. Causal links between major depressive disorder and insomnia: a Mendelian randomisation study. Gene. 2021;768:145271.

Nassan M, Daghlas I, Winkelman JW, Dashti HS, Saxena R. Genetic evidence for a potential causal relationship between insomnia symptoms and suicidal behavior: a Mendelian randomization study. Neuropsychopharmacology. 2022;47:1672–9.

Sun X, Liu B, Liu S, Wu DJH, Wang J, Qian Y, et al. Sleep disturbance and psychiatric disorders: a bidirectional Mendelian randomisation study. Epidemiol Psychiatr Sci. 2022;31:e26.

Herrett E, Gallagher AM, Bhaskaran K, Forbes H, Mathur R, van Staa T, et al. Data resource profile: Clinical Practice Research Datalink (CPRD). Int J Epidemiol. 2015;44:827–36.

Davies MR, Kalsi G, Armour C, Jones IR, McIntosh AM, Smith DJ, et al. The Genetic Links to Anxiety and Depression (GLAD) Study: online recruitment into the largest recontactable study of depression and anxiety. Behav Res Ther. 2019;123:103503.

Religa D, Fereshtehnejad S-M, Cermakova P, Edlund A-K, Garcia-Ptacek S, Granqvist N, et al. SveDem, the Swedish Dementia Registry—a tool for improving the quality of diagnostics, treatment and care of dementia patients in clinical practice. PLoS ONE. 2015;10:e0116538.

Sudlow C, Gallacher J, Allen N, Beral V, Burton P, Danesh J, et al. UK Biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med. 2015;12:e1001779.

German National Cohort (GNC) Consortium. The German National Cohort: aims, study design and organization. Eur J Epidemiol. 2014;29:371–82.

Choi KW, Stein MB, Nishimi KM, Ge T, Coleman JRI, Chen C-Y, et al. An exposure-wide and Mendelian randomization approach to identifying modifiable factors for the prevention of depression. Am J Psychiatry. 2020;177:944–54.

Choi KW, Chen C-Y, Stein MB, Klimentidis YC, Wang M-J, Koenen KC, et al. Assessment of bidirectional relationships between physical activity and depression among adults. JAMA Psychiatry. 2019;76:399–408.

Jefsen OH, Speed M, Speed D, Østergaard SD. Bipolar disorder and cannabis use: A bidirectional two-sample Mendelian randomization study. Addict Biol. 2021;26:e13030.

Huang P, Zou Y, Zhang X, Ye X, Wang Y, Yu R, et al. The causal effects of insomnia on bipolar disorder, depression, and schizophrenia: a two-sample Mendelian Randomization Study. Front Genet 2021;12. https://www.frontiersin.org/articles/10.3389/fgene.2021.763259 .

Yang J, Fan Y, Yan B, Zhao B, Qian L, Gao F, et al. Mendelian randomization analyses reveal novel drug targets for anorexia nervosa. Prog Neuropsychopharmacol Biol Psychiatry. 2022;112:110427.

Sproviero W, Winchester L, Newby D, Fernandes M, Shi L, Goodday SM, et al. High blood pressure and risk of dementia: a two-sample Mendelian randomization study in the UK Biobank. Biol Psychiatry. 2021;89:817–24.

Baird DA, Liu JZ, Zheng J, Sieberts SK, Perumal T, Elsworth B, et al. Identifying drug targets for neurological and psychiatric disease via genetics and the brain transcriptome. PLoS Genet. 2021;17:e1009224.

Ryan DK, Karhunen V, Su B, Traylor M, Richardson TG, Burgess S, et al. Genetic evidence for protective effects of angiotensin-converting enzyme against Alzheimer disease but not other neurodegenerative diseases in European populations. Neurol Genet. 2022;8:e200014.

Burgess S, Thompson SG. Multivariable Mendelian randomization: the use of pleiotropic genetic variants to estimate causal effects. Am J Epidemiol. 2015;181:251–60.

Burgess S, Davies NM, Thompson SG, Consortium on behalf of E-I. Instrumental variable analysis with a nonlinear exposure–outcome relationship. Epidemiology. 2014;25:877.

Rees JMB, Foley CN, Burgess S. Factorial Mendelian randomization: using genetic variants to assess interactions. Int J Epidemiol. 2019;49:1147–58.

Article   PubMed Central   Google Scholar  

Download references

Acknowledgements

LW is supported by an Alzheimer’s Research UK fellowship (ARUK-RF2020A-005) and funding from Virtual Brain Cloud from the European Commission [grant number H2020-SC1-DTH-2018-1] and Rosetrees Trust (M937). NT is supported by the EPSRC Center for Doctoral Training in Health Data Science (EP/S02428X/1). DWJ is supported in part by the NIHR AI Award for Health and Social Care (NIHR-AI-AWARD0-2183). DWJ is supported by the NIHR Oxford Health Biomedical Research Centre (grant BRC-1215-20005). The views expressed are those of the authors and not necessarily those of the UK National Health Service, the NIHR, the UK Department of Health, or the University of Oxford.

Author information

Authors and affiliations.

Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Centre for Statistics in Medicine, University of Oxford, Oxford, UK

Danielle Newby

Department of Psychiatry, University of Oxford, Oxford, UK

Niall Taylor & Laura M. Winchester

Department of Primary Care and Mental Health and Civic Health, Innovation Labs, Institute of Population Health, University of Liverpool, Liverpool, UK

Dan W. Joyce

You can also search for this author in PubMed   Google Scholar

Contributions

DN, NT, and LW drafted the manuscript. LW and DWJ critically revised and approved the final manuscript.

Corresponding author

Correspondence to Laura M. Winchester .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Newby, D., Taylor, N., Joyce, D.W. et al. Optimising the use of electronic medical records for large scale research in psychiatry. Transl Psychiatry 14 , 232 (2024). https://doi.org/10.1038/s41398-024-02911-1

Download citation

Received : 17 February 2023

Revised : 13 April 2024

Accepted : 15 April 2024

Published : 01 June 2024

DOI : https://doi.org/10.1038/s41398-024-02911-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Search Menu

Sign in through your institution

• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Papyrology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Evolution
• Language Reference
• Language Acquisition
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Media
• Music and Religion
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Science
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Ethics
• Business Strategy
• Business History
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic History
• Economic Systems
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• Ethnic Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Behaviour
• Political Economy
• Political Institutions
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Theory
• Politics and Law
• Politics of Development
• Public Policy
• Public Administration
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

53 Quantitative research methods in medical education

• Published: October 2013
• Cite Icon Cite
• Permissions Icon Permissions

Quantitative research in medical education tends to be predominantly observational research based on survey or correlational studies. As researchers strive towards making inferences about the impact of education interventions, a shift towards experimental research designs may enhance the quality and conclusions made in medical education. The establishment of experiment research designs, where interventions (i.e. curriculum, teaching or assessment interventions) are tested with an experimental group and either a comparison or controlled group of learners, may allow researchers to overcome validity concerns and infer potential cause–effect generalizations. There are a number of internal and external validity concerns that researchers need to be conscious of when designing their own or looking at others’ experimental research studies. The selection of a research design for any study should fit within the parameters of the stated research question or hypothesis. In quantitative research, the findings will reflect the reliability and validity (psychometric characteristics) of the measured outcomes or dependent variables (such as changes in knowledge, skills, or attitudes) used to assess the effectiveness of the medical education intervention (the independent variable of interest). It is important to remember that not all quantitative research involves experimental studies—important results can also be drawn from quantitative observational studies. This chapter outlines commonly used quantitative methods in medical education research. It explains their theoretical underpinnings, the evidence base for their use, and gives practical guidance on their application. It concludes with a section on the role of meta-analyses of quantitative research in medical education.

Personal account

• Sign in with email/username & password
• Get email alerts
• Save searches
• Purchase content
• Activate your purchase/trial code
• Add your ORCID iD

Institutional access

Sign in with a library card.

• Sign in with username/password
• Recommend to your librarian
• Institutional account management
• Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

• Click Sign in through your institution.
• Select your institution from the list provided, which will take you to your institution's website to sign in.
• When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
• Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

• Click Sign in through society site.
• When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

• View your signed in personal account and access account management features.
• View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Our books are available by subscription or purchase to libraries and institutions.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Rights and permissions
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Get new issue alerts Get alerts

Secondary Logo

Journal logo.

Colleague's E-mail is Invalid

Your message has been successfully sent to your colleague.

Save my selection

Establishment of a Quantitative Medical Technology Evaluation System and Indicators within Medical Institutions

Wu, Suo-Wei; Chen, Tong; Pan, Qi; Wei, Liang-Yu; Wang, Qin; Li, Chao; Song, Jing-Chen; Luo, Ji

Department of Medical Administration, Beijing Hospital, National Center of Gerontology, Beijing 100730, China

Address for correspondence: Dr. Qi Pan, Department of Medical Administration, Beijing Hospital, National Center of Gerontology, Beijing 100730, China E-Mail: [email protected]

Received February 10, 2018

Background: 

The development and application of medical technologies reflect the medical quality and clinical capacity of a hospital. It is also an effective approach in upgrading medical service and core competitiveness among medical institutions. This study aimed to build a quantitative medical technology evaluation system through questionnaire survey within medical institutions to perform an assessment to medical technologies more objectively and accurately, and promote the management of medical quality technologies and ensure the medical safety of various operations among the hospitals.

Methods: 

A two-leveled quantitative medical technology evaluation system was built through a two-round questionnaire survey of chosen experts. The Delphi method was applied in identifying the structure of evaluation system and indicators. The judgment of the experts on the indicators was adopted in building the matrix so that the weight coefficient and maximum eigenvalue (λ max), consistency index (CI), and random consistency ratio (CR) could be obtained and collected. The results were verified through consistency tests, and the index weight coefficient of each indicator was conducted and calculated through analytical hierarchy process.

Results: 

Twenty-six experts of different medical fields were involved in the questionnaire survey, 25 of whom successfully responded to the two-round research. Altogether, 4 primary indicators (safety, effectiveness, innovativeness, and benefits), as well as 13 secondary indicators, were included in the evaluation system. The matrix is built to conduct the λ max, CI , and CR of each expert in the survey, and the index weight coefficients of primary indicators were 0.33, 0.28, 0.27, and 0.12, respectively, and the index weight coefficients of secondary indicators were conducted and calculated accordingly.

Conclusions: 

As the two-round questionnaire survey of experts and statistical analysis were performed and credibility of the results was verified through consistency evaluation test, the study established a quantitative medical technology evaluation system model and assessment indicators within medical institutions based on the Delphi method and analytical hierarchy process. Moreover, further verifications, adjustments, and optimizations of the system and indicators will be performed in follow-up studies.

INTRODUCTION

The development and application of medical technologies reflect the medical quality and clinical capacity of a hospital. Meanwhile, the operation of advanced medical technologies is also an effective approach in upgrading medical service and core competitiveness among medical institutions.[ 1 ] Therefore, the regulation on medical technologies is a crucial part of medical quality management and hospital management, and it is the primary task for the medical administrative department to ensure the safety and effectiveness of medical technologies in clinical applications.[ 2 ] To be specific, the key contents of medical technology management are the evaluation and assessment of the operation procedures and protocols, the technical competency of medical staffs and medical units as well as the external conditions such as the equipment and materials required in the process to determine the safety, effectiveness as well as the adaptability in clinical practice.[ 3 ] Thus, the establishment of a comprehensive evaluation system and related indicators of clinical technology are vital in enhancing the management and supervision within medical institutions.[ 4 ] According to recent studies, some domestic and foreign medical administrative departments and medical institutions have already established relatively mature admission procedures and management regulations as well as assessment methodology of medical technology.[ 5 ] However, researchers and outcomes on the accurate evaluation systems and standards, especially on quantitative assessment systems that can assess to an objective, unified evaluation are still rare, which make it almost impossible for different medical technology for comparison.[ 6 ]

On this basis, the current study aimed to build a quantitative medical technology evaluation system through questionnaire survey using the Delphi method and set index weight coefficient to each indicator through analytical hierarchy process within medical institutions. Moreover, it is hoped with the establishment of the evaluation system and further verifications in clinical practice, medical institutions can make a better assessment to clinical technologies, promote the management of medical quality technologies, and ensure the medical safety of various operations among the hospitals.[ 7 ]

Selection of questionnaire survey experts

According to the principle of the Delphi method, a certain number of experts were selected to participate in the questionnaire survey. In general, the accuracy and constancy of the results are related to the number of experts joining the investigation.[ 8 ] Normally, to ensure the credibility and authority of the results, the optimum number of experts investigated was among 15–50.[ 9 ] All the experts selected should be involved in the related specialty fields and that are acquainted with the contents of the survey. To ensure the accuracy and objectivity of the results, the professional levels, specialty majors as well as the length of occupations of the experts should be evenly distributed.[ 10 ]

Establishment of the evaluation system and two-leveled indicators

According to previous literature reviews and expert consultations, it is generally suggested to build a medical technology evaluation system of the two-leveled indicators.[ 11 ] Based on the current reports and studies, practical experiences of medical technology operations in combination with the results of the expert consultation, the research drafted the initial evaluation system of four primary level indicator and 14 secondary indicators. Then, the first round of survey questionnaires was developed according to the system and indexes we drafted and distributed to experts selected. Based on the principle of the Delphi method, experts would be responding to the questionnaire and grade each indicator based on the Likert scale (containing 5 scales, which are “strongly agree”, “agree”, “uncertain”, “disagree”, and “strongly disagree”, with the grade of 5, 4, 3, 2, and 1, respectively, in statistical analysis) by the importance of the subjects according to their expertise and experience.[ 12 ] Besides, suggestions of the experts in adjusting the structure of the evaluation system through adding, deleting, or merging indicators would also be faithfully recorded. The results of the first round questionnaires were collected and processed, and the outcomes and suggestions of statistical significance were summed up in rebuilding the second round of survey questionnaires for the same expert team of the same procedures.

Allocation of the index weight

To set the allocation of index weight, analytical hierarchy process was adopted in determining the weight coefficient of each indicator. In the second round of the survey, experts were asked to assign weights to each level of indicators according to their experience and expertise, and the matrix was built in the principle of analytical hierarchy process.[ 13 ] In this period, experts were supposed to make comparisons of specific indicators in each judgment matrix, and Saaty 1–9 scale relative materiality table was applied in the grading the importance of the indicators between one and another, the matrix was completed based on the value of relative importance of each subject.[ 14 ] Calculate the primary and secondary index weight coefficient by the survey results of the matrix, and the index weight coefficient was used in judging the importance of the indicators. The judgment of the experts on primary level indicators was input into Microsoft Excel 2010 (Microsoft Corporation, Washington, USA) to build the matrix so that the weight coefficient and maximum eigenvalue (λ max), consistency index (CI), and random consistency ratio (CR) were obtained and collected.[ 15 ] Consistency test was performed according to the calculated value, as the results of CR <0.1 was considered adaptable, and the average value of the results was calculated to determine the index weight primary level indicators.[ 16 ] Then collected the weight coefficient of secondary indicators according to the same principle and with the index weight obtained, calculated the integrated index weight of secondary indicators according to the formula N ij = X i Y ij ( i = 1, 2…; j = 1, 2,…). As X i refers to the primary level index weight, and Y ij is the j level index weight under the i primary level index.[ 17 ] All the data and material collected were entered into Excel 2010 software for Microsoft (Microsoft Corporation), and statistical analyses were performed using SPSS version 14.0 (SPSS Inc., Chicago, IL, USA).

General situation of experts and the distribution of questionnaires

In this study, 26 experts of different occupational lengths and professional levels majoring in clinical medicine, hospital management, nursery as well as pharmacy management were invited in the research, as the general situation of the experts are shown in Table 1 . According to the initial index system as well as the evaluation indicators we drafted, the first round of survey questionnaires was designed as shown in Supplementary Table 1 . In the first round of investigation, 26 questionnaires were distributed and 25 copies were recovered, with a recovery rate of 96.15%. While in the second round, a total of 25 questionnaires were allocated to the 25 experts responded in the last round, and all the questionnaires returned were examined to be effective.

Establishment of the evaluation system and indicators

The first round of survey results questionnaires of the 25 survey experts were collected, analyzed and verified through consistency tests. According to the grading of experts, it was suggested that the initial structure of the evaluation system are basically recognized. Meanwhile, modifications on some indicators were made according to suggestions given by the experts [ Table 2 ].

Indicators suggested to be added

During the survey, 6 experts out of 25 (24.0%) suggested that the indicator “the priority of the technology in professional field” were of great importance that should be included in the system. Over careful consideration, we decided to include the subject as the secondary indicator under the primary column of “innovativeness”.

Indicators suggested to be combined or separated

Four experts out of 25 (16.0%) proposed the combination of secondary indicators “the establishment of practice guideline and operation regulations” and “the formulation of risk disposal plans” under the primary column of “safety” since they could all be classified as the preoperational systematic requirements of medical technologies. Besides, 3 experts (12.0%) suggested the indicator “definitions on indications and contraindications” be separated in consideration for the aptness of grading. After further discussion, we separated the original indicator into “definitions on indications of diseases” and “occurrence of contraindications” two parts.

Indicators suggested to be deleted

According to the survey results of 3 experts (8.0%), the secondary indicator “the origination of innovations” under the primary column “innovativeness” was thought hard to be quantitatively analyzed and evaluated, and hence that it was suggested to be deleted, and the proposal was accepted by the research group.

With the adjustments made of the evaluation system and the indicators, the second round of survey questionnaires was developed.

The aim of the second round survey was carried out to identify the index weight of each indicator as we distributed the questionnaires to the 25 experts responded in the first round. After the results were collected and analyzed, verification on the consistency of the experts was made, and results showed that all the CR values were below 0.1, suggesting the opinions of experts were relatively stable. Besides, the experts had no revisions on the setting of the indicators in the second round of survey. Thus, the assignments of primary and secondary indicators were analyzed and calculated by building the matrix according to the grading of experts on each individual indicator by Saaty 1–9 scale relative materiality table, as the results of index weight of indicator are calculated as shown in Table 3 . According to the results, the maximum index weight of primary indicator was the safety of the medical technology, with the coefficient of 0.33, followed by effectiveness and innovativeness, with the index weight coefficient of 0.28 and 0.27 respectively, benefits of medical technology ranked last in the survey, with the index weight coefficient of 0.12. The index weight of each secondary indicator was conducted according to the formula N ij = X i Y ij ( i = 1, 2…; j = 1, 2,…), and the results are shown in Table 4 .

Selection of initial indicators of the system

According to general consensus, the basis and core elements of medical technologies management within medical institutions are consist of safety, effectiveness, innovativeness, and the benefits.[ 18 ] In the light of the principle, the study initially set the four basic subjects as the primary indicators of the evaluations system. For the record, since all the medical equipment and material applied in hospitals went through the safety and effectiveness assessment by medical administration departments, the evaluation on safety and effectiveness of medical technologies mainly refers to the operations performed by medical staffs. Similarly, the secondary indicators are drafted according to clinical experience and preliminary research reports, and the acquisition of all the indicators should be linear indexes (such as the secondary indicator “quantity of operated cases” that could be rated by number and “recovery rate” that could be marked by ratio under the column of “effectiveness”) or that could be valued through unified and objective grading standards (such as the secondary indicator “the priority of the technology in professional field” that could be graded by “globally advanced”, “domestic advanced”, “regional advanced”, “horizontal advanced”, “well-developed” as well as “under-developed” and marked by arithmetic sequence number). On this basis, some initial indicators that are hard to be quantitatively estimated (such as “the origination of innovations”) are eliminated or replaced by other indexes.

Application and advantages of the evaluation system and its indicators

The evaluation system we built provided a methodology for medical institutions to analyze the comprehensive adaptability of clinical technologies since all the indicators could be measured quantitatively that made it possible for different medical technologies comparable with one and another.[ 19 ] Compared with previous studies, the advantage and innovativeness of the system lie in its accuracy and objectiveness. Besides, the evaluation system along with the indicators enable medical institution to make assessment in medical technologies of all fields (including surgical operations, noninvasive operations, laboratory operations as well as image inspection operations and so on), all classes ( first-, second-, and third-class technologies), and all phases (preadmission and postadmission phase), which provided a standardized, comparable platform in medical technology management for medical institutions.[ 20 ] Moreover, the results could be adopted in the assessment of individuals, departments, and institutions of medical fields.

Verification of the evaluation system and adjustment of indicators

To better apply the evaluation system into practical operation, further verification of the indicators should be made in the clinical application. Besides, since the availability and suitability of all the indicators might be changing with time, the reassessment and adjustment of all the indicators should be dynamically made over a certain period.[ 21 ]

Study limitations

The accuracy and objectiveness of the evaluation system and the indicators are closely related to the practical situation of medical institutions among different regions.[ 22 ] The degree of development, structure, and scale of medical institutions might all have a direct impact on indicators.[ 13 ] Since the research was performed within a regional medical institution and all the experts we surveyed are from medical fields in Beijing, considering the different situation in various regions, as well as the limitation of the data and material, the generalization of the results should be more careful in other circumstances.[ 23 ]

In conclusions, though a two-round questionnaire survey of experts and statistical analysis, along with the verification on the credibility of the results through consistency evaluation test, the study established a quantitative medical technology evaluation system model and assessment indicators within medical institutions based on the Delphi method and analytical hierarchy process.[ 24 ] It is believed that the system could be better applied to clinical practice through continuous revalidation and dynamic adjustment with time. Furthermore, the methodology of the study could be extended to other subjects of hospital evaluation.

Supplementary information is linked to the online version of the paper on the Chinese Medical Journal website .

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

• Cited Here |
• View Full Text | PubMed | CrossRef
• PubMed | CrossRef

Edited by: Qiang Shi

Evaluation System; Medical Management; Medical Technology

• + Favorites
• View in Gallery

Advertisement

• Previous Issue
• Previous Article
• Next Article
• Box 1. What to Look for in Research Using This Method

What Is Qualitative Research?

Qualitative versus quantitative research, conducting and appraising qualitative research, conclusions, research support, competing interests, qualitative research methods in medical education.

Submitted for publication January 5, 2018. Accepted for publication November 29, 2018.

• Split-Screen
• Article contents
• Figures & tables
• Supplementary Data
• Peer Review
• Open the PDF for in another window
• Cite Icon Cite
• Get Permissions
• Search Site

Adam P. Sawatsky , John T. Ratelle , Thomas J. Beckman; Qualitative Research Methods in Medical Education. Anesthesiology 2019; 131:14–22 doi: https://doi.org/10.1097/ALN.0000000000002728

Download citation file:

• Ris (Zotero)
• Reference Manager

Qualitative research was originally developed within the social sciences. Medical education is a field that comprises multiple disciplines, including the social sciences, and utilizes qualitative research to gain a broader understanding of key phenomena within the field. Many clinician educators are unfamiliar with qualitative research. This article provides a primer for clinician educators who want to appraise or conduct qualitative research in medical education. This article discusses a definition and the philosophical underpinnings for qualitative research. Using the Standards for Reporting Qualitative Research as a guide, this article provides a step-wise approach for conducting and evaluating qualitative research in medical education. This review will enable the reader to understand when to utilize qualitative research in medical education and how to interpret reports using qualitative approaches.

Image: J. P. Rathmell and Terri Navarette.

Qualitative research provides approaches to explore and characterize the education of future anesthesiologists. For example, the practice of anesthesiology is increasingly team-based; core members of the anesthesia care team include physicians, trainees, nurse anesthetists, anesthesiologist assistants, and other healthcare team members. 1   Understanding how to work within and how to teach learners about anesthesia care teams requires the ability to conceptualize the complexity of individual psychology and social interactions that occur within teams. Qualitative research is well suited to investigate complex issues like team-based care. For example, one qualitative study observed the interactions between members of the anesthesia care team during simulated stressful situations and conducted interviews of team members; they described limited understanding of each team member’s role and perceptions about appropriate roles and responsibilities, which provided insight for interprofessional team training. 2   Another qualitative study explored the hierarchy within the anesthesia care team, highlighting residents’ reluctance to challenge the established hierarchy and outlining the strategies they use to cope with fear and intimidation. 3   Key issues in medical education and anesthesiology, particularly when exploring human experience and social interactions, may be best studied using qualitative research methodologies and methods.

Medical education is a complex field, and medical education research and practice fittingly draws from many disciplines ( e.g. , medicine, psychology, sociology, education) and synthesizes multiple perspectives to explain how people learn and how medicine should be taught. 4 , 5   The concept of a field was well described by Cristancho and Varpio 5   in their tips for early career medical educators: “A discipline is usually guided by shared paradigms, assumptions, rules and methods to present their knowledge claims— i.e. , people from the same discipline speak the same language. A field brings people from multiple disciplines together.” Qualitative research draws from the perspectives of multiple disciplines and has provided methodologies to explore the complex research questions inherent to medical education.

When appraising qualitative research in medical education, do the authors:

Clearly state the study purpose and research question?

Describe the conceptual framework that inform the study and guide analysis?

Identify their qualitative methodology and research paradigm?

Demonstrate adequate reflexivity, conveying to the reader their values, assumptions and way of thinking, being explicit about the effects these ways of thinking have on the research process?

Choose data collection methods that are congruent with the research purpose and qualitative methodology?

Select an appropriate sampling strategy, choosing participants whose perspectives or experiences are relevant to the study question?

Define their method for determining saturation, how they decided to stop data collection?

Outline their process for data processing, including the management and coding of study data?

Conduct data analysis consistent with their chosen methodology?

Consider techniques to enhance trustworthiness of their study findings?

Synthesize and interpret their data with sufficient detail and supporting quotations to explain the phenomenon of study?

Current medical training is heavily influenced by the practice of evidence-based medicine. 6   Trainees are taught the “hierarchy of evidence” for evaluating studies of clinical interventions. 7   This hierarchy prioritizes knowledge gained through systematic reviews and meta-analyses, randomized controlled trials, and observational studies, but it does not include qualitative research methodologies. This means that because of their medical training and exposure to quantitative medical literature, clinician educators may be more familiar with quantitative research and feel more comfortable engaging in studies utilizing quantitative methodologies. However, many clinician educators are not familiar with the language and application of qualitative research and feel less comfortable engaging in studies using qualitative methodologies.

Because medical education is a diverse and complex field, qualitative research is a common approach in medical education research. Clinician educators who wish to understand the medical education literature need to be familiar with qualitative research. Clinician educators involved in research may also find themselves asking questions best answered by qualitative methodologies. Our goal is to provide a broad, practical overview of qualitative research in medical education. Our objectives are to:

1) Define qualitative research.

2) Compare and contrast qualitative and quantitative research.

3) Provide a framework for conducting and appraising qualitative research in medical education.

Qualitative research in medical education has a distinct vocabulary with terminology not commonly used in other biomedical research fields. Therefore, we have provided a glossary and definitions of the common terms that are used throughout this article ( table 1 ).

Glossary of Common Terms Used in Qualitative Research

Of the many attempts to provide a comprehensive definition of qualitative research, our favorite definition comes from Denzin and Lincoln:

“Qualitative research is a situated activity that locates the observer in the world. Qualitative research consists of a set of interpretive, material practices that make the world visible. These practices…turn the world into a series of representations, including field notes, interviews, conversations, photographs, recordings, and memos to the self. At this level, qualitative research involves an interpretive, naturalistic approach to the world. This means that qualitative researchers study things in their natural settings, attempting to make sense of or interpret phenomena in terms of the meanings people bring to them.” 12  

This definition reveals the following points: first, qualitative research is a “situated activity,” meaning that the research and observations are made in the real world, in this case a real life clinical or educational situation. Second, qualitative research “turns the world into a series of representations” by representing the observations, in this case of a clinical or educational situation, with qualitative data, usually taking the form of words, pictures, documents, and other symbols. Last, qualitative researchers seek to “make sense” of the meanings that research participants bring to different phenomena to allow for a greater understanding of those phenomena. Through qualitative research, observers comprehend participants’ beliefs and values and the way these beliefs and values are shaped by the context in which they are studied.

Because most clinician educators are familiar with quantitative methods, we will start by comparing qualitative and quantitative methods to gain a better understanding of qualitative research ( table 2 ). To illustrate the difference between qualitative and quantitative research in medical education, we pose the question: “What makes noon conference lectures effective for resident learning?” A qualitative approach might explore the learner perspective on learning in noon conference lectures during residency and conduct an exploratory thematic analysis to better understand what the learner thinks is effective. 13   A qualitative approach is useful to answer this question, especially if the phenomenon of interest is incompletely understood. If we wanted to compare types or attributes of conferences to assess the most effective methods of teaching in a noon conference setting, then a quantitative approach might be more appropriate, though a qualitative approach could be helpful as well. We could use qualitative data to inform the design of a survey 14   or even inform the design of a randomized control trial to compare two types of learning during noon conference. 15   Therefore, when discussing qualitative and quantitative research, the issue is not which research approach is stronger, because it is understood that each approach yields different types of knowledge when answering the research question.

Comparisons of Quantitative and Qualitative Research in Medical Education

Similarities

The first step of any research project, qualitative or quantitative, is to determine and refine the study question; this includes conducting a thorough literature review, crafting a problem statement, establishing a conceptual framework for the study, and declaring a statement of intent. 16   A common pitfall in medical education research is to start by identifying the desired methods ( e.g. , “I want to do a focus group study with medical students.”) without having a clearly refined research question, which is like putting the cart before the horse. In other words, the research question should guide the methodology and methods for both qualitative and quantitative research.

Acknowledging the conceptual framework for a study is equally important for both qualitative and quantitative research. In a systematic review of medical education research, only 55% of studies provided a conceptual framework, limiting the interpretation and meaning of the results. 17   Conceptual frameworks are often theories that represent a way of thinking about the phenomenon being studied. Conceptual frameworks guide the interpretation of data and situate the study within the larger body of literature on a specific topic. 9   Because qualitative research was developed within the social sciences, many qualitative research studies in medical education are framed by theories from social sciences. Theories from social science disciplines have the ability to “open up new ways of seeing the world and, in turn, new questions to ask, new assumptions to unearth, and new possibilities for change.” 18   Qualitative research in medical education has benefitted from these new perspectives to help understand fundamental and complex problems within medical education such as culture, power, identity, and meaning.

Differences

The fundamental difference between qualitative and quantitative methodologies centers on epistemology ( i.e. , differing views on truth and knowledge). Cleland 19   describes the differences between qualitative and quantitative philosophies of scientific inquiry: “quantitative and qualitative approaches make different assumptions about the world, about how science should be conducted and about what constitutes legitimate problems, solutions and criteria of ‘proof.’”

Quantitative research comes from objectivism , an epistemology asserting that there is an absolute truth that can be discovered; this way of thinking about knowledge leads researchers to conduct experimental study designs aimed to test hypotheses about cause and effect. 10   Qualitative research, on the other hand, comes from constructivism , an epistemology asserting that reality is constructed by our social, historical, and individual contexts, and leads researchers to utilize more naturalistic or exploratory study designs to provide explanations about phenomenon in the context that they are being studied. 10   This leads researchers to ask fundamentally different questions about a given phenomenon; quantitative research often asks questions of “What?” and “Why?” to understand causation, whereas qualitative research often asks the questions “Why?” and “How?” to understand explanations. Cook et al. 20   provide a framework for classifying the purpose of medical education research to reflect the steps in the scientific method—description (“What was done?”), justification (“Did it work?”), and clarification (“Why or how did it work?”). Qualitative research nicely fits into the categories of “description” and “clarification” by describing observations in natural settings and developing models or theories to help explain “how” and “why” educational methods work. 20  

Another difference between quantitative and qualitative research is the role of the researcher in the research process. Experimental studies have explicitly stated methods for creating an “unbiased” study in which the researcher is detached ( i.e. , “blinded”) from the analysis process so that their biases do not shape the outcome of the research. 21   The term “bias” comes from the positivist paradigm underpinning quantitative research. Assessing and addressing “bias” in qualitative research is incongruous. 22   Qualitative research, based largely on a constructivist paradigm, acknowledges the role of the researcher as a “coconstructer” of knowledge and utilizes the concept of “reflexivity.” Because researchers act as coconstructors of knowledge, they must be explicit about the perspectives they bring to the research process. A reflexive researcher is one who challenges their own values, assumptions, and way of thinking and who is explicit about the effects these ways of thinking have on the research process. 23   For example, when we conducted a study on self-directed learning in residency training, we were overt regarding our roles in the residency program as core faculty, our belief in the importance of self-directed learning, and our assumptions that residents actually engaged in self-directed learning. 24 , 25   We also needed to challenge these assumptions and open ourselves to alternative questions, methods of data collection, and interpretations of the data, to ultimately ensure that we created a research team with varied perspectives. Therefore, qualitative researchers do not strive for “unbiased” research but to understand their own roles in the coconstruction of knowledge. When assessing reflexivity, it is important for the authors to define their roles, explain how those roles may affect the collection and analysis of data, and how the researchers accounted for that effect and, if needed, challenged any assumptions during the research process. Because of the role of the researcher in qualitative research, it is vital to have a member of the research team with qualitative research experience.

A Word on Mixed Methods

In mixed methods research, the researcher collects and analyzes both qualitative and quantitative data rigorously and integrates both forms of data in the results of the study. 26   Medical education research often involves complex questions that may be best addressed through both quantitative and qualitative approaches. Combining methods can complement the strengths and limitations of each method and provide data from multiple sources to create a more detailed understanding of the phenomenon of interest. Examples of uses of mixed methods that would be applicable to medical education research include: collecting qualitative and quantitative data for more complete program evaluation, collecting qualitative data to inform the research design or instrument development of a quantitative study, or collecting qualitative data to explain the meaning behind the results of a quantitative study. 26   The keys to conducting mixed methods studies are to clearly articulate your research questions, explain your rationale for use of each approach, build an appropriate research team, and carefully follow guidelines for methodologic rigor for each approach. 27  

Toward Asking More “Why” Questions

We presented similarities and differences between qualitative and quantitative research to introduce the clinician educator to qualitative research but not to suggest the relative value of one these research methods over the other. Whether conducting qualitative or quantitative research in medical education, researchers should move toward asking more “why” questions to gain deeper understanding of the key phenomena and theories in medical education to move the field of medical education forward. 28   By understanding the theories and assumptions behind qualitative and quantitative research, clinicians can decide how to use these approaches to answer important questions in medical education.

There are substantial differences between qualitative and quantitative research with respect to the assessment of rigor; here we provide a framework for reading, understanding, and assessing the quality of qualitative research. O’Brien et al. 29   created a useful 21-item guide for reporting qualitative research in medical education, based upon a systematic review of reporting standards for qualitative research—the Standards for Reporting Qualitative Research. It should be noted, however, that just performing and reporting each step in these standards do not ensure research quality.

Using the Standards for Reporting Qualitative Research as a backdrop, we will highlight basic steps for clinician educators wanting to engage with qualitative research. If you use this framework to conduct qualitative research in medical education, then you should address these steps; if you are evaluating qualitative research in medical education, then you can assess whether the study investigators addressed these steps. Table 3 underscores each step and provides examples from our research in resident self-directed learning. 25  

Components of Qualitative Research: Examples from a Single Research Study

Refine the study question. As with any research project, investigators should clearly define the topic of research, describe what is already known about the phenomenon that is being studied, identify gaps in the literature, and clearly state how the study will fill that gap. Considering theoretical underpinnings of qualitative research in medical education often means searching for sources outside of the biomedical literature and utilizing theories from education, sociology, psychology, or other disciplines. This is also a critical time to engage people from other disciplines to identify theories or sources of information that can help define the problem and theoretical frameworks for data collection and analysis. When evaluating the introduction of a qualitative study, the researchers should demonstrate a clear understanding of the phenomenon being studied, the previous research on the phenomenon, and conceptual frameworks that contextualize the study. Last, the problem statement and purpose of the study should be clearly stated.

Identify the qualitative methodology and research paradigm. The qualitative methodology should be chosen based on the stated purpose of the research. The qualitative methodology represents the overarching philosophy guiding the collection and analysis of data and is distinct from the research methods ( i.e. , how the data will be collected). There are a number of qualitative methodologies; we have included a list of some of the most common methodologies in table 4 . Choosing a qualitative methodology involves examining the existing literature, involving colleagues with qualitative research expertise, and considering the goals of each approach. 32   For example, explaining the processes, relationships, and theoretical understanding of a phenomenon would point the researcher to grounded theory as an appropriate approach to conducting research. Alternatively, describing the lived experiences of participants may point the researcher to a phenomenological approach. Ultimately, qualitative research should explicitly state the qualitative methodology along with the supporting rationale. Qualitative research is challenging, and you should consult or collaborate with a qualitative research expert as you shape your research question and choose an appropriate methodology. 32  

Choose data collection methods. The choice of data collection methods is driven by the research question, methodology, and practical considerations. Sources of data for qualitative studies would include open-ended survey questions, interviews, focus groups, observations, and documents. Among the most important aspects of choosing the data collection method is alignment with the chosen methodology and study purpose. 33   For interviews and focus groups, there are specific methods for designing the instruments. 34 , 35   Remarkably, these instruments can change throughout the course of the study, because data analysis often informs future data collection in an iterative fashion.

Select a sampling strategy. After identifying the types of data to be collected, the next step is deciding how to sample the data sources to obtain a representative sample. Most qualitative methodologies utilize purposive sampling, which is choosing participants whose perspectives or experiences are relevant to the study question. 11   Although random sampling and convenience sampling may be simpler and less costly for the researcher than purposeful sampling, these approaches often do not provide sufficient information to answer the study question. 36   For example, in grounded theory, theoretical sampling means that the choice of subsequent participants is purposeful to aid in the building and refinement of developing theory. The criteria for selecting participants should be stated clearly. One key difference between qualitative and quantitative research is sample size: in qualitative research, sample size is usually determined during the data collection process, whereas in quantitative research, the sample size is determined a priori . Saturation is verified when the analysis of newly collected data no longer provides additional insights into the data analysis process. 10  

Plan and outline a strategy for data processing. Data processing refers to how the researcher organizes, manages, and dissects the study data. Although data processing serves data analysis, it is not the analysis itself. Data processing includes practical aspects of data management, like transcribing interviews, collecting field notes, and organizing data for analysis. The next step is coding the data, which begins with organizing the raw data into chunks to allow for the identification of themes and patterns. A code is a “word or short phrase that symbolically assigns a summative, salient, essence-capturing, and/or evocative attribute for a portion of language-based or visual data.” 8   There is an artificial breakdown between data processing and analysis, because these steps may be conducted simultaneously; many consider coding as different from—yet a necessary step to facilitating—the analysis of data. 8   Qualitative software can support this process, by making it easier to organize, access, search, and code your data. However, it is noteworthy that these programs do not do the work for you, they are merely tools for supporting data processing and analysis.

Conduct the data analysis. When analyzing the data, there are several factors to consider. First, the process of data analysis begins with the initial data collection, which often informs future data collection. Researchers should be intentional when reading, reviewing, and analyzing data as it is collected, so that they can shape and enrich subsequent data collection ( e.g. , modify the interview questions). Second, data analysis is often conducted by a research team that should have the appropriate expertise and perspectives to bring to the analysis process. Therefore, when evaluating a qualitative study, you should consider the team’s composition and their reflexivity with respect to their potential biases and influences on their study subjects. Third, the overall goal is to move from the raw data to abstractions of the data that answer the research question. For example, in grounded theory, the research moves from the raw data, to the identification of themes, to categorization of themes, to identifying relationships between themes, and ultimately to the development of theoretical explanations of the phenomenon. 30   Consequently, the primary researcher or research team should be intimately involved with the data analysis, interrogating the data, writing analytic memos, and ultimately make meaning out of the data. There are differing opinions about the use of “counting” of codes or themes in qualitative research. In general, counting of themes is used during the analysis process to recognize patterns and themes; often these are not reported as numbers and percentages as in quantitative research, but may be represented by words like few , some , or many . 37  

Recognize techniques to enhance trustworthiness of your study findings. Ensuring consistency between the data and the results of data analysis, along with ensuring that the data and results accurately represent the perspectives and contexts related to the data source, are crucial to ensuring trustworthiness of study findings. Methods for enhancing trustworthiness include triangulation , which is comparing findings from different methods or perspectives, and member-checking , which is presenting research findings to study participants to provide opportunities to ensure that the analysis is representative. 10  

Synthesize and interpret your data. Synthesis of qualitative research is determined by the depth of the analysis and involves moving beyond description of the data to explaining the findings and situating the results within the larger body of literature on the phenomenon of interest. The reporting of data synthesis should match the research methodology. For instance, if the study is using grounded theory, does the study advance the theoretical understanding of the phenomenon being studied? It is also important to acknowledge that clarity and organization are paramount. 10   Qualitative data are rich and extensive; therefore, researchers must organize and tell a compelling story from the data. 38   This process includes the selection of representative data ( e.g. , quotations from interviews) to substantiate claims made by the research team.

Common Methodologies Used in Qualitative Research

For more information on qualitative research in medical education:

Qualitative Research and Evaluation Methods: Integrating Theory and Practice, by Michael Q. Patton (SAGE Publications, Inc., 2014)

Qualitative Inquiry and Research Design: Choosing Among Five Approaches, by John W. Cresswell (SAGE Publications, Inc. 2017)

Researching Medical Education, by Jennifer Cleland and Steven J. Durning (Wiley-Blackwell, 2015)

Qualitative Research in Medical Education, by Patricia McNally, in Oxford Textbook of Medical Education, edited by Kieren Walsh (Oxford University Press, 2013)

The Journal of Graduate Medical Education “Qualitative Rip Out Series” (Available at: http://www.jgme.org/page/ripouts )

The Standards for Reporting Qualitative Research (O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89(9):1245-51.)

The Wilson Centre Qualitative Atelier (For more information: http://thewilsoncentre.ca/atelier/ )

Qualitative research is commonly used in medical education but may be unfamiliar to many clinician educators. In this article, we provided a definition of qualitative research, explored the similarities and differences between qualitative and quantitative research, and outlined a framework for conducting or appraising qualitative research in medical education. Even with advanced training, it can be difficult for clinician educators to understand and conduct qualitative research. Leaders in medical education research have proposed the following advice to clinician educators wanting to engage in qualitative medical education research: (1) clinician educators should find collaborators with knowledge of theories from other disciplines ( e.g. , sociology, cognitive psychology) and experience in qualitative research to utilize their complementary knowledge and experience to conduct research—in this way, clinician educators can identify important research questions; collaborators can inform research methodology and theoretical perspectives; and (2) clinician educators should engage with a diverse range disciplines to generate new questions and perspectives on research. 4  

Support was provided solely from institutional and/or departmental sources.

The authors declare no competing interests.

Citing articles via

Most viewed, email alerts, related articles, social media, affiliations.

• ASA Practice Parameters
• Online First
• Author Resource Center
• About the Journal
• Editorial Board
• Rights & Permissions
• Online ISSN 1528-1175
• Print ISSN 0003-3022
• Anesthesiology
• ASA Monitor

• Terms & Conditions Privacy Policy
• Manage Cookie Preferences
• © Copyright 2024 American Society of Anesthesiologists

This Feature Is Available To Subscribers Only

Sign In or Create an Account

More From Forbes

Leveraging technology for patient safety: the role of knowledge management in healthcare.

• Share to Facebook
• Share to Twitter
• Share to Linkedin

CEO of KMS Lighthouse . Leading the company's vision to disrupt the knowledge management market.

In the rapidly evolving landscape of healthcare technology, ensuring patient safety remains a paramount concern. By harnessing innovative technologies such as AI, data analytics and digital health platforms, healthcare providers can implement robust knowledge management (KM) systems to capture, disseminate and apply critical insights. I'll explore how technology-enabled initiatives empower healthcare teams to identify adverse events, implement evidence-based practices and foster a culture of continuous learning.

At a time when healthcare providers must thoroughly serve their patients' needs while protecting their personal information, technologies like electronic health records (EHRs), clinical decision support systems (CDSS) and telemedicine are enhancing patient safety practices.

Knowledge management is critical in making these technologies effective by streamlining access to up-to-date medical knowledge, evidence-based guidelines and patient data, enabling healthcare professionals to make informed decisions, reduce errors and improve outcomes.

Hospitals, clinics and doctors' offices owe it to their patients and themselves to leverage every available tool to uphold the highest standards of care. I'll explore the dynamic relationship between KM and healthcare technologies, delving into the transformative impact their integration is having on creating cultures of continuous learning and quality improvement while safeguarding patient well-being in the digital age.

Best High-Yield Savings Accounts Of 2024

Best 5% interest savings accounts of 2024.

Knowledge management’s role in healthcare technology improves the creation, sharing and application of knowledge to support patient safety and improve healthcare outcomes. Over the years, it has evolved to include the following.

Electronic Health Records

Before: Patient records were primarily paper-based, making it challenging to share information across different healthcare providers and settings and increasing the risk of medical errors due to incomplete or unavailable patient information.

Today: EHRs have digitized patient records, allowing healthcare professionals to access comprehensive patient information, including medical histories, test results and prescribed medications, from any authorized location. Communication, coordination and decision-making are improved , and the likelihood of medical errors is reduced.

Clinical Decision Support Systems

Before: Healthcare professionals relied heavily on individual knowledge and experience, which could result in clinical practice variations and potential oversights.

Today: CDSS integrates evidence-based guidelines, best practices and patient-specific data to provide real-time alerts, reminders and recommendations to healthcare professionals, ensuring they adhere to standardized protocols while reducing the risk of medication errors, adverse drug interactions and other preventable mistakes.

Knowledge Repositories And Expert Systems

Before: Medical knowledge was primarily stored in textbooks, journals and the collective experience of healthcare professionals, making it more challenging to consistently access and apply the latest research and best practices.

Today: Healthcare organizations use knowledge repositories and expert systems to consolidate and disseminate up-to-date medical knowledge, clinical guidelines and best practices. This means that healthcare professionals have easier access to the latest evidence-based information to enhance treatment effectiveness.

Telemedicine And Remote Monitoring

Before: Patients in remote or underserved areas had limited access to specialized healthcare services, increasing the risk of delayed diagnoses and suboptimal treatment.

Today: Telemedicine and remote monitoring technologies enable healthcare professionals to monitor and provide care to patients remotely, leveraging the knowledge and expertise of specialists across geographical borders. Access to high-quality care is improved , bridging the gap between remote and centralized healthcare and reducing the risk of complications due to delayed interventions.

Continuing Medical Education (CME) And E-Learning

Before: Healthcare professionals relied heavily on in-person training and seminars to stay current with the latest medical knowledge and best practices—an often time-consuming and logistically challenging endeavor.

Today: Online CME platforms and e-learning resources provide convenient access to educational materials, allowing providers to continuously enhance their knowledge and skills. This, in turn, helps ensure they’re equipped with the latest knowledge when making decisions on a patient’s healthcare options.

By using an integrated knowledge approach, healthcare providers can create, share and apply institutional knowledge to promote evidence-based practices, standardize care protocols and empower medical professionals with the information they need to optimize patient care outcomes.

Three Ways Technology Is Improving Patient Safety

Many researchers claim that medical error is the third-leading cause of death in the U.S., often due to poor communication, a lack of access to comprehensive patient data and best practice deviations. Healthcare providers can improve patient care by investing in innovative technologies and implementing a robust knowledge management system.

1. Artificial Intelligence: AI’s predictive capabilities enable providers to offer proactive, preventative care that improves outcomes and reduces costs. In hospitals, AI-powered clinical decision support systems integrate evidence-based guidelines, best practices and patient-specific data to provide healthcare professionals with real-time reminders and recommendations.

For instance, AI systems can analyze a patient's medical history, current medications and lab results and then alert physicians to potential adverse drug interactions or contraindications , reducing the risk of medication errors.

2. Data Analytics: Advanced healthcare analytics can potentially revolutionize the medical sector . Clinics can use data analytics tools to identify patterns and trends in patient data, enabling them to make data-driven decisions and improve care processes. For example, by analyzing patient outcomes data, a clinic can identify risk factors for certain conditions and tailor preventive care strategies accordingly.

3. Digital Healthcare Platforms: These online services moved to the forefront during the pandemic. Today, doctors' offices can use telemedicine and remote monitoring technologies to leverage knowledge management systems and provide patients with access to specialized care and expertise, regardless of geographic location.

For instance, a rural-based primary care physician might consult with a major medical center specialist, sharing patient data and medical records through a secure digital platform, ensuring timely and accurate diagnosis and treatment for a patient.

Every patient deserves a healthcare provider they can trust to keep them safe. When finding the right information at the right time can literally save lives, it’s crucial for the healthcare sector to implement solutions that facilitate timely access to accurate medical knowledge, evidence-based guidelines and patient data.

Knowledge is capital for the healthcare industry, and information technology can have a significant impact on patient safety. Healthcare-focused knowledge management systems align people, data and technologies to provide patients with secure environments where informed decisions can be made to prevent errors and improve the quality of care—a win-win for patients and providers.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?

• Editorial Standards
• Reprints & Permissions

share this!

June 14, 2024

This article has been reviewed according to Science X's editorial process and policies . Editors have highlighted the following attributes while ensuring the content's credibility:

fact-checked

peer-reviewed publication

trusted source

Training AI models to answer 'what if?' questions could improve medical treatments

by University of Cambridge

Machines can learn not only to make predictions, but to handle causal relationships. An international research team shows how this could make medical treatments safer, more efficient, and more personalized.

Artificial intelligence techniques can be helpful for multiple medical applications , such as radiology or oncology, where the ability to recognize patterns in large volumes of data is vital. For these types of applications, the AI compares information against learned examples, draws conclusions, and makes extrapolations.

Now, an international team led by researchers from Ludwig-Maximilians-Universität München (LMU) and including researchers from the University of Cambridge, is exploring the potential of a comparatively new branch of AI for diagnostics and therapy.

The researchers found that causal machine learning (ML) can estimate treatment outcomes—and do so better than the machine learning methods generally used to date. Causal machine learning makes it easier for clinicians to personalize treatment strategies, which individually improves the health of patients.

The results, reported in the journal Nature Medicine , suggest how causal machine learning could improve the effectiveness and safety of a variety of medical treatments .

Classical machine learning recognizes patterns and discovers correlations. However, the principle of cause and effect remains closed to machines as a rule; they cannot address the question of why. When making therapy decisions for a patient, the 'why' is vital to achieve the best outcomes.

"Developing machine learning tools to address 'why?' and 'what if?' questions is empowering for clinicians, because it can strengthen their decision-making processes," said senior author Professor Mihaela van der Schaar, Director of the Cambridge Center for AI in Medicine. "But this sort of machine learning is far more complex than assessing personalized risk."

For example, when attempting to determine therapy decisions for someone at risk of developing diabetes, classical ML would aim to predict how probable it is for a given patient with a range of risk factors to develop the disease.

With causal ML, it would be possible to answer how the risk changes if the patient receives an anti-diabetes drug; that is, gauge the effect of a cause. It would also be possible to estimate whether metformin, the commonly-prescribed medication, would be the best treatment , or whether another treatment plan would be better.

To be able to estimate the effect of a hypothetical treatment, the AI models must learn to answer "what if?" questions. "We give the machine rules for recognizing the causal structure and correctly formalizing the problem," said Professor Stefan Feuerriegel from LMU, who led the research. "Then the machine has to learn to recognize the effects of interventions and understand, so to speak, how real-life consequences are mirrored in the data that has been fed into the computers."

Even in situations for which reliable treatment standards do not yet exist or where randomized studies are not possible for ethical reasons because they always contain a placebo group , machines could still gauge potential treatment outcomes from the available patient data and form hypotheses for possible treatment plans, so the researchers hope.

With such real-world data, it should generally be possible to describe the patient cohorts with ever greater precision in the estimates, bringing individualized therapy decisions that much closer. Naturally, there would still be the challenge of ensuring the reliability and robustness of the methods.

"The software we need for causal ML methods in medicine doesn't exist out of the box," says Professor Feuerriegel. "Rather, complex modeling of the respective problem is required, involving close collaboration between AI experts and doctors."

In other fields, such as marketing, explains Professor Feuerriegel, the work with causal ML has already been in the testing phase for some years now. "Our goal is to bring the methods a step closer to practice," he said. The paper describes the direction in which things could move over the coming years."

"I have worked in this area for almost 10 years, working relentlessly in our lab with generations of students to crack this problem," said Professor van der Schaar, who is affiliated with the Departments of Engineering, Applied Mathematics and Theoretical Physics, and Medicine.

"It's an extremely challenging area of machine learning, and seeing it come closer to clinical use, where it will empower clinicians and patients alike, is very satisfying."

Professor Van der Schaar is continuing to work closely with clinicians to validate these tools in diverse clinical settings, including transplantation, cancer and cardiovascular disease.

Explore further

Feedback to editors

A new large-scale simulation platform to train robots on everyday tasks

20 hours ago

Transitioning to renewable energy systems might not entail a decline in net energy

21 hours ago

New carbon nitride membrane improves lithium extraction from salt lakes

Jun 14, 2024

Mineralizing emissions: Advanced reactor designs for CO₂ capture

Microsoft delays controversial AI Recall feature on new Windows computers

Concrete-nitrogen mix may provide major health and environment benefits

Novel strategy proposed for all-climate zinc-ion batteries

Simplicity versus adaptability: Scientists propose AI method that integrates habitual and goal-directed behaviors

A fully edible robot could soon end up on our plate, say scientists

From wearables to swallowables: Engineers create GPS-like smart pills with AI

Related stories.

Researchers explore causal machine learning, a new advancement for AI in health care

Apr 19, 2024

With huge patient dataset, AI accurately predicts treatment outcomes

May 1, 2024

A theoretical model for reliability assessment of machine learning systems

Feb 1, 2024

AI-based surgical prediction models have limits

May 15, 2024

Using machine learning to battle COVID-19 bacterial co-infection

Feb 2, 2024

AI in medical research: Promise and challenges

Dec 19, 2023

Recommended for you

An open-source robotic system that can play chess with humans

New technique improves the reasoning capabilities of large language models

Let us know if there is a problem with our content.

Use this form if you have come across a typo, inaccuracy or would like to send an edit request for the content on this page. For general inquiries, please use our contact form . For general feedback, use the public comments section below (please adhere to guidelines ).

Please select the most appropriate category to facilitate processing of your request

Thank you for taking time to provide your feedback to the editors.

Your feedback is important to us. However, we do not guarantee individual replies due to the high volume of messages.

E-mail the story

Your email address is used only to let the recipient know who sent the email. Neither your address nor the recipient's address will be used for any other purpose. The information you enter will appear in your e-mail message and is not retained by Tech Xplore in any form.

Your Privacy

This site uses cookies to assist with navigation, analyse your use of our services, collect data for ads personalisation and provide content from third parties. By using our site, you acknowledge that you have read and understand our Privacy Policy and Terms of Use .

E-mail newsletter

Life Science Research Professional 1

🔍 school of medicine, stanford, california, united states.

Stanford University is seeking a Life Science Research Professional 1 to perform basic functions and activities involved in defined research projects, and independently conduct and analyze experiments.

Stanford University is one of the world’s most renowned universities.  Sitting in the heart of the San Francisco Bay Area among the valley’s most progressive companies.

You will be working with an unparalleled leading-edge community of faculty and staff that are fundamentally changing the world of health care. You will have the opportunity to influence and drive change with your innovative ideas, the ability to make a difference and participate in human advancements. Our culture is fast-paced, energetic, and growing all of the time. 

We offer a variety of benefits beyond traditional medical, dental, retirement, and savings options:

• Events and program for children, sports camps, tuition options
• World-class intellectual stimulation through learning and development classes, workshops, and onsite conferences from leading-edge speakers and faculty
• Work/life and family-friendly policies and reimbursement
• Participation in Stanford’s social responsibility and sustainable programs for a better world
• A vibrant university culture that values the uniqueness of each individual

We are seeking candidates who are progressive thinkers, see challenges as simply problems to solve, and have the spirit and energy to change the world.

For more information about the department visit  http://pathology.stanford.edu/ About the Lab:

The LSRP will work in the lab of Dr. Ansuman Satpathy MD PhD (Department of Pathology and Parker Institute for Cancer Immunotherapy at Stanford University School of Medicine). The goal of the lab is to apply cutting-edge immunologic and genomic technologies to study the immune response to cancer, and to design the next-generation of cancer immunotherapies for patients. The laboratory consists of an interdisciplinary team of physicians and scientists with backgrounds in medicine, immunology, cancer, and genetics. The Satpathy Lab studies fundamental principles of the immune system in health, infection, and cancer (Parker et al, Cell 2020; Flynn et al, Cell 2021). The lab particularly focuses on single-cell genomic and multi-omic technologies (Yost et al, Nature Medicine 2019; Rubin et al, Cell 2019) to study immunological mechanisms of disease directly in samples isolated from patients. For more information about our multidisciplinary research laboratory, see https://satpathylab.com/ .

About the Position: 

We are recruiting a Life Science Research Professional to join a new research group within the lab focused on using high throughput functional genomics experiments to understand and engineer novel gene regulation paradigms. The group has a history of developing such systems (..Stickels, Science 2019, Stickels Nature Biotech 2021, ..Stickels biorxiv 2024) and we are looking to expand our team. Working as part of a multidisciplinary group, the LSRP will design and construct DNA libraries for cellular engineering experiments, isolate and culture primary human immune cells, and perform functional analysis of engineered cells via flow cytometry, cancer co-culture assays, and prepare and analyze next generation sequencing libraries. This position represents an excellent opportunity to gain exposure to basic laboratory skills as well as advanced genetics and genome engineering technologies in a dynamic research environment, and will allow the LSRP to interface with an exceptional set of experimental and computational biologists.

Duties include: 

• Plan approach to experiments in support of research projects in lab and/or field based on knowledge of scientific theory.
• Independently conduct experiments; maintain detailed records of experiments and outcomes.
• Apply the theories and methods of a life science discipline to interpret and perform analyses of experiment results; offer suggestions regarding modifications to procedures and protocols in collaboration with senior researcher.
• Review literature on an ongoing basis to remain current with new procedures and apply learnings to related research.
• Contribute to publication of findings as needed. Participate in the preparation of written documents, including procedures, presentations, and proposals.
• Help with general lab maintenance as needed; maintain lab stock, manage chemical inventory and safety records, and provide general lab support as needed.
• Assist with orientation and training of new staff or students on lab procedures or techniques.

*- Other duties may also be assigned

Desired Qualifications: 

Bachelor’s degree in a quantitative field, life science discipline, or related field.

• Knowledge of basic molecular biology workflows in a laboratory such as molecular cloning procedures, PCR, and Sequencing Library preparation is required.  
• Knowledge and experience in any of the following fields: immunology, molecular biology, cell biology, genomics, or related biology and bio-engineering fields.
• Other qualities of a successful application would include evidence of being self-motivated, possessing strong communication skills, and the ability and desire to perform as part of a team.
• Fluency in a programming language for biological data science analyses, including R (preferred), Python (preferred), Julia, C++, Perl, and/or Matlab is a plus but not required.
• Basic knowledge of the unix command line environment is a plus but not required.

Education & Experience (Required):

Bachelor's degree in related scientific field.

Knowledge, Skills, and Abilities (Required): 

• General understanding of scientific principles. Demonstrated performance to use knowledge and skills when needed.
• Demonstrated ability to apply theoretical knowledge of science principals to problem solve work.
• Ability to maintain detailed records of experiments and outcomes.
• General computer skills and ability to quickly learn and master computer programs, databases, and scientific applications.
• Ability to work under deadlines with general guidance.
• Excellent organizational skills and demonstrated ability to accurately complete detailed work.

Physical Requirements*:

• Frequently stand, walk, twist, bend, stoop, squat, grasp lightly, use fine manipulation, grasp forcefully, perform desk-based computer tasks, use telephone, write by hand, lift, carry, push and pull objects weighing over 50 pounds. 
• Occasionally sit, kneel, crawl, reach and work above shoulders, sort and file paperwork or parts.
• Rarely climb, scrub, sweep, mop, chop and mix or operate hand and foot controls.
• Must have correctible vision to perform duties of the job.
• Ability to bend, squat, kneel, stand, reach above shoulder level, and move on hard surfaces for up to eight hours.
• Ability to lift heavy objects weighing up to 50 pounds.
• Ability to work in a dusty, dirty, and odorous environment.
• Position may require repetitive motion.

*- Consistent with its obligations under the law, the University will provide reasonable accommodation to any employee with a disability who requires accommodation to perform the essential functions of his or her job.

Working Conditions:

• May require working in close proximity to blood borne pathogens.
• May require work in an environment where animals are used for teaching and research.
• Position may at times require the employee to work with or be in areas where hazardous materials and/or infectious diseases are present.
• Employee must perform tasks that require the use of personal protective equipment, such as safety glasses and shoes, protective clothing and gloves, and possibly a respirator.
• May require extended or unusual work hours based on research requirements and business needs.
• Due to the nature of the work, this position will work fully on-site.

The expected pay range for this position is $26.44 to $36.54 per hour.

Stanford University provides pay ranges representing its good faith estimate of what the university reasonably expects to pay for a position. The pay offered to a selected candidate will be determined based on factors such as (but not limited to) the scope and responsibilities of the position, the qualifications of the selected candidate, departmental budget availability, internal equity, geographic location and external market pay for comparable jobs.

At Stanford University, base pay represents only one aspect of the comprehensive rewards package. The Cardinal at Work website ( https://cardinalatwork.stanford.edu/benefits-rewards ) provides detailed information on Stanford’s extensive range of benefits and rewards offered to employees. Specifics about the rewards package for this position may be discussed during the hiring process.

Why Stanford is for You   Imagine a world without search engines or social platforms. Consider lives saved through first-ever organ transplants and research to cure illnesses. Stanford University has revolutionized the way we live and enrich the world. Supporting this mission is our diverse and dedicated 17,000 staff. We seek talent driven to impact the future of our legacy. Our culture and unique perks empower you with:

• Freedom to grow. We offer career development programs, tuition reimbursement, or course auditing. Join a TedTalk, film screening, or listen to a renowned author or global leader speak.
• A caring culture. We provide superb retirement plans, generous time-off, and family care resources.
• A healthier you. Climb our rock wall or choose from hundreds of health or fitness classes at our world-class exercise facilities. We also provide excellent health care benefits.
• Discovery and fun. Stroll through historic sculptures, trails, and museums.
• Enviable resources. Enjoy free commuter programs, ridesharing incentives, discounts and more

The job duties listed are typical examples of work performed by positions in this job classification and are not designed to contain or be interpreted as a comprehensive inventory of all duties, tasks, and responsibilities. Specific duties and responsibilities may vary depending on department or program needs without changing the general nature and scope of the job or level of responsibility. Employees may also perform other duties as assigned.   Consistent with its obligations under the law, the university will provide reasonable accommodation to any employee with a disability who requires accommodation to perform the essential functions of his or her job.   Stanford is an equal employment opportunity and affirmative action employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, protected veteran status, or any other characteristic protected by law.

• Schedule: Full-time
• Job Code: 4943
• Employee Status: Fixed-Term
• Requisition ID: 103621
• Work Arrangement : On Site

My Submissions

Track your opportunities.

Similar Listings

 School of Medicine, Stanford, California, United States

📁 Research

Post Date: May 28, 2024

Post Date: May 07, 2024

Post Date: Jan 29, 2024

Global Impact We believe in having a global impact

Climate and sustainability.

Stanford's deep commitment to sustainability practices has earned us a Platinum rating and inspired a new school aimed at tackling climate change.

Medical Innovations

Stanford's Innovative Medicines Accelerator is currently focused entirely on helping faculty generate and test new medicines that can slow the spread of COVID-19.

From Google and PayPal to Netflix and Snapchat, Stanford has housed some of the most celebrated innovations in Silicon Valley.

Advancing Education

Through rigorous research, model training programs and partnerships with educators worldwide, Stanford is pursuing equitable, accessible and effective learning for all.

Working Here We believe you matter as much as the work

I love that Stanford is supportive of learning, and as an education institution, that pursuit of knowledge extends to staff members through professional development, wellness, financial planning and staff affinity groups.

School of Engineering

I get to apply my real-world experiences in a setting that welcomes diversity in thinking and offers support in applying new methods. In my short time at Stanford, I've been able to streamline processes that provide better and faster information to our students.

Phillip Cheng

Office of the Vice Provost for Student Affairs

Besides its contributions to science, health, and medicine, Stanford is also the home of pioneers across disciplines. Joining Stanford has been a great way to contribute to our society by supporting emerging leaders.

Denisha Clark

School of Medicine

I like working in a place where ideas matter. Working at Stanford means being part of a vibrant, international culture in addition to getting to do meaningful work.

Office of the President and Provost

Getting Started We believe that you can love your job

Join Stanford in shaping a better tomorrow for your community, humanity and the planet we call home.

• 4.2 Review Ratings
• 81% Recommend to a Friend

View All Jobs

Wild African elephants address each other with name-like rumbling calls: study

Science Wild African elephants address each other with name-like rumbling calls: study

African elephants are a highly social bunch with tonnes of different ways to communicate.

They greet each other by intertwining trunks and express strong emotions by trumpeting. They are even thought to send messages over large distances with ground-based vibrations, which could be detected through their feet.

Now researchers behind a new study in Nature Ecology and Evolution believe wild elephants may be able to communicate with other members of their group by using sounds akin to names.

Dolphins and some birds are reported to develop a signature call which other members of their species can used to identify them or copy to get their attention.

But Colorado State University behavioural ecologist Mickey Pardo, lead author of the new study, believes elephant calls are more similar to how humans use arbitrary names rather than imitations used by other species.

"Elephants are among the few mammals that are capable of learning to produce new sounds , which is a prerequisite for having names," he said.

Elephant talk 101

Human names are conveyed through speech, but an elephant "name" sounds a lot different.

One of an African elephant's most common forms of communication are vocalisations known as rumbles, which can sound a bit like a growl or roar.

"It's actually easier for elephants to learn to produce rumbles on command than it is for them to learn to produce trumpets on command," Dr Pardo said.

To the inexperienced ear, elephant rumbles can sound the same or very similar, but Dr Pardo said that's akin to an alien assuming humans have a modest vocal repertoire of crying, laughing, screaming or speech.

"If you thought about it that way, you would be missing all the interesting stuff about human communication," he said.

Inspired by the work of behavioural biologist Stephanie King , who found dolphins could identify others by repeating signature calls , Dr Pardo wanted to dig deep into the rumbles of elephants, another big-brained mammal, to find if something similar was going on.

To do this, he and his colleagues analysed recordings of 469 rumbles made by groups of wild African elephant females and their young at Amboseli National Park and Samburu and Buffalo Springs National Reserves in Kenya over 36 years.

While there are many different contexts for rumbles, the research team focused on three:

• Contact — where elephants try to make contact with other family members over a long distance or when out of sight
• Greeting — when elephants approach each other within touching distance
• Caregiving — when adult or adolescent females give comfort to or communicate with a young calf.

They used an artificial intelligence model to figure out which elephant was being addressed in each recorded call.

The next step was to play calls containing "names" to 17 wild elephants.

"I found that they reacted much more strongly to calls that were originally addressed to them than to calls from the same caller that were originally addressed to someone else," Dr Pardo said.

"That indicates that they can tell just by hearing a call whether it was addressed to them or not."

Behaviours explained

For some of the long-time elephant researchers involved with the study, the findings appear to confirm unexplained interactions they had witnessed before.

Joyce Poole — a conservationist, biologist and co-founder of not-for-profit Elephant Voices — said she had seen an elephant rumble out a contact call in the past, but noticed only one member of their group responded while others ignored them.

"Other times a different individual would answer, or perhaps none of the elephants that I was with answered," she said.

"Since the caller listened after calling, I knew she was waiting for an answer. 

"It made me wonder whether the elephants were 'rude' and simply ignoring one another or whether she wasn't calling the particular family members that I was with."

Harvard Medical School behavioural ecologist Caitlin O'Connell-Rodwell, who was not involved in the study, thought the findings were the tip of the iceberg for understanding elephant vocal complexity.

"[The study] puts a lot in perspective and totally makes sense as far as elephants trying to communicate over a kilometre or so, and wanting to address a specific individual," Dr O'Connell-Rodwell said.

Dr O'Connell-Rodwell said she had previously assumed elephants which reacted without delay to a call must have been an immediate family member. 

"Over time, I saw a more complex pattern emerging, but didn't know exactly what individuals were cueing in on," she said.

"This study very carefully assessed these situations and vocalisations and provided a quantitative answer to my long-standing question of why certain individuals responded more intently, other than the possibility of being more closely related."

What's in a name?

Although the new study's researchers believe they have found evidence of elephant names, they can't really tell how they are structured within the calls.

The human ear has a hard time detecting elements of elephant rumbles, which can be low on the decibel register.

Colorado State University wildlife conservation biologist George Wittemyer, a study co-author, said not understanding how elephant calls were encoded was a bit like not understanding an alphabet in human language.

Further research could also help understand if each elephant has a specific "name", or if they have a few different names that are used by different callers.

With room for interpretation, University of Western Australia behavioural ecologist Amanda Ridley, who was not part of the study, said whether name-like calls were truly being used by the African elephants was unclear.

"What I do think is convincing in the study is that they are using specific vocal information to address specific individuals," she said.

"This could be considered 'names' in some contexts."

But Dr Ridley said she thought it was the most convincing case of non-human name-use, which wasn't imitated , so far discovered in the animal kingdom.

"I think it highly possible other species do this as well, but the specific research to test this hasn't been conducted on them yet," she said.

Science in your inbox

• X (formerly Twitter)

Related Stories

Work smarter, not harder: study finds magpies who are bullied are likely to be smarter than their bullies.

Secret language of dolphins discovered during COVID-19 quiet

'This is how they learn': Rare footage captures a lyrebird singing lesson

• Animal Behaviour
• Animal Science
• Environment
• Science and Technology

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• JMIR Res Protoc
• v.7(12); 2018 Dec

Acceptance and Resistance of New Digital Technologies in Medicine: Qualitative Study

1 Medical School Brandenburg, digilog, Center for Connected Health Care UG, Neuruppin, Germany

Thomas Thiessen

2 BSP Business School Berlin, digilog, Center for Connected Health Care UG, Berlin, Germany

Kurt JG Schmailzl

This study discusses the acceptance of new medical technologies in health care settings and resistance to these technologies from hospitals, doctors’ surgical centers, electronic health (eHealth) centers, and related institutions. We suggest a novel method of identifying factors that influence the acceptance of, and resistance to, new technologies by medical staff and patients.

The objective of this study was to determine and evaluate the factors that influence acceptance and resistance to achieve a successful implementation of new technologies.

The target group was patients residing in Brandenburg and major stakeholders in the local health care structure, for instance, medical institutions and medical professionals. The process relies on 3 models: the technology acceptance model, the unified technology acceptance and use of technology model, and the theory of technical innovation diffusion. Qualitative methodology was employed in this study, and an exploratory design was adopted to gain new insights into a poorly understood phenomenon in the German context. This enabled the researcher to take a flexible approach toward exploring a wide range of secondary data and to choose a different approach when unexpected information emerged. Content analysis was used to identify and interpret the data, and the researcher assured that the meaning associated with the information has concurred with that of the original source.

This study confirmed that adoption of new technologies in health care depended on individual opinions of the factors relating to them. Some medical professionals believed that technology would interfere with their ability to make independent diagnoses and their relationships with patients. Doctors also feared that technology was a means of management control. In contrast, other medical staff welcomed technology because it provided them with more opportunities to interact with patients and their carers. Generally, patients were more enthusiastic about technology than medical professionals and health care managers because it allowed them to have greater autonomy in selecting health care options. The need for all groups to be involved in the development of the new health care approach was an important outcome, otherwise resistance to it was likely to be greater. In other words, the strategy for change management was the indicator of success or failure. Therefore, following our analysis, a number of practical precepts emerged that could facilitate user acceptance of digital solutions and innovative medical technologies.

Conclusions

The acceptance of digital solutions and innovative medical technology by patients and professionals relies on understanding their anxieties and feelings of insecurity. The process will take time because individuals accept change at different rates. Hence, the development of an extensive user community to fully and successfully implement eHealth is less likely in the short term; however, this should not prevent the push for changes in health care technology.

Introduction

The term electronic health ( eHealth ) encompasses all apps that integrate modern information and communication technologies (ICT) to treat and care for patients. Therefore, eHealth is considered to be a general term for a wide range of ICT-based applications, which process information electronically. This information can be exchanged to support patient treatment and care processes because medical data obtained from the eHealth card can be easily communicated. This information consists of emergency data, treatment plans, medications, and the electronic patient file or telemedicine applications. This health information is communicated via a telematics infrastructure [ 1 ], and eHealth is summarized as follows: “a new term used to describe the combined use of electronic communication and information technology in the health sector” [ 2 ].

Therefore, eHealth centers are institutions or workstations that are fully equipped with specific medical diagnostic equipment and information technologies (ITs). These features enable members to run diagnostic tests and communicate results to doctors in real time. Thus, eHealth centers demonstrate how modern health care can be delivered to underserved populations and can encourage healthier communities.

Health care innovations seek to achieve stability, security, sustainability, and high qualitative value through networked structures, technological solutions, and analog interaction spaces. Innovative medical technology and digital solutions are components of innovative patterns and models. However, the successful implementation of innovative medical technologies, in particular digital companions, depends on acceptance by medical staff, such as doctors and nurses. These individuals directly confront new technologies and their implementation, with patients positioned as customers. To determine if innovation benefits these groups, resistance must be identified and reduced by creating greater awareness of the technologies and convincing potential users of advantages associated with the use of these technologies. Therefore, in medical settings, the implementation of new medical technologies should also consider psychological indicators that indicate developing acceptance, thus supporting a win-win situation. Resistance to new technologies or procedures should be recognized by medical professionals and their customers. Once recognized, these forms of resistance can be overcome by carefully planned and appropriate interventions.

Therefore, we sought to identify drivers of and barriers to the adoption of new health care technologies by referencing existing studies and then generated recommendations for improving technology uptake and diffusion. The resultant recommendations will be tested in future research projects.

Acceptance is a perceptual phenomenon that involves evaluation of new experiences and arriving at a final decision with respect to the benefits and limitations of that experience. Acceptance outcomes are derived from attitudes or courses of action. The development of acceptance depends on the interaction of 3 elements: subjective acceptance, objective acceptance, and the context in which acceptance occurs. Acceptance is an unpredictable construct because modified perceptions or general conditions can lead to different levels of acceptance. The decision to accept or reject a certain technology depends on various influencing factors [ 3 ]. Psychological approaches focus on attitudes, positions, norms, and value system factors that influence acceptance. Emotions and sociodemographic factors, such as age, gender, and educational level, similarly influence acceptance. Objective acceptance relies on how relevant an individual evaluates an innovation’s characteristics, acknowledging that identical characteristics may lead to different responses. This is because major influencing factors vary among individuals and may include financial impact, cost-benefit analysis, acquisition of necessary skills, or opportunities for work facilitation.

Technology acceptance also depends on perceived risks, such as whether the technology delivers secure care that is reliable and effective. Ease of use is of particular concern to users as new technology should facilitate more efficient execution of health-related tasks. The content factors related to acceptance are not directly related to users; rather, they externally influence users. One example of this could be jobs that are supported by new technologies and social processes in organizations, groups, or communities involved in implementing new technologies. Other contextual factors include organizational and social environments, including existing routines, political climates, participation cultures, the state of the economy, legal frameworks, and the processes by which innovation is introduced.

The technology acceptance model (TAM) is frequently used to explain acceptance [ 4 ] and provides insight into an individual’s decision to use or reject technological innovations [ 5 ]. The TAM proposes that the use of technology depends on 2 variables: how useful the technology is perceived to be and how easily it can be employed. Perceived usefulness is defined as an individual’s subjective evaluation of the new technology in relation to how much it will enhance their job performance. In contrast, perceived ease of use is the assessment of the effort required to learn and use technology. The balance of effort and usefulness underpins the development of user acceptance and influences user motivation. In general, an individual’s motivation to use a technology is higher if that technology is easy to use. External factors, such as support measures, have a positive effect on the perception of usefulness and on understanding a technology. In general, individuals adjust to new procedures quickly.

The unified technology acceptance and use of technology (UTAUT) model is an extension of the TAM that bases the growth of acceptance on 4 factors: performance expectancy, effort expectancy, social influence, and facilitating conditions [ 6 ]. Performance expectancy is a person’s perception of the extent to which a new technology brings about improvement and is the strongest predictor for the development of acceptance. As with the TAM, effort expectancy in the UTAUT model is the perceived usefulness and complexity of the technology. Social influence describes an individual’s perception of the extent to which others believe a new technology should be used and facilitating conditions under which individuals recognize interventions that support the use of the technology, for example, organizational or technical infrastructures. These models are used more than any other methods to explain acceptance, and some technology specialists suggest that studying their effectiveness has detracted from new research fields in technology acceptance. Their major shortcomings include the fact that research related to TAM was conducted under the assumption that a positive relationship exists between new technology use and user satisfaction, quality, and productivity; however, this was not proven [ 7 ].

The innovation diffusion theory examines technology uptake by individuals and organizations, focusing on the technological innovation development process from the invention stage to general acceptance or rejection. Five characteristics of innovation influence the diffusion of a new technology: relative advantage, compatibility, complexity, opportunity for a trial, and observability. Relative advantage is a technology’s perceived superiority over current methodologies, whereas compatibility is a social factor related to how well the technology matches social norms and behaviors. The remaining 3 factors describe its practical usage, with complexity indicating how easy it is to learn, opportunity for a trial being the technology’s amenability to evaluation and the chances of this occurring before adaptation is determined, and observability is associated with the capacity to observe the new technology’s outputs and advantages over alternatives. Although all 5 factors affect the rate of technology diffusion and no single aspect is strong enough to predict acceptance, the rate of innovation diffusion is the most affected by low complexity, the opportunity to try out novel technologies, and observability [ 8 ]. Empirical studies have also found that relative advantage and compatibility have positive associations with technology adoption, whereas complexity is negatively related to adoption [ 9 ]. The limitations of the innovation diffusion theory include the interpretation of relative advantage, which is a subjective factor because a cost versus benefits comparison is most important to some individuals, whereas ease of use is more appreciated by others.

The indicators of the development of acceptance, characteristics of the actors, features of the technological innovation, planned interventions implemented for its introduction, and observable benefits for patients and practitioners (including social factors and diverse environmental characteristics) decide the likelihood of individual adoption and dissemination of innovations. The weaknesses associated with the 3 theories, however, represent a gap in the current literature. There is a need to verify whether the assumptions underlying TAM result in positive outcomes and whether the subjective nature of relative advantage affects the rate of technology uptake purported by the innovation diffusion theory.

As environmental factors do not change easily, they are considered to be exogenous. Endogenous components, such as the actors and innovation, are critical for the development of acceptance. These factors are explored in the proposed methodology.

Methodological Process

This study is the precursor to a future project; therefore, the methodology indicates how the entire study will proceed. This study gathers major findings from previous research on eHealth technology acceptance to make recommendations for enhancing uptake of technology diffusion, which will be later tested empirically.

Research Philosophy, Design, and Strategy

Social science research may adopt 2 diverse philosophies: objective and subjective stances. The objective stance seeks to determine objective cause-and-effect links between variables, whereas the subjective approach seeks to gather deep insight into how human beings interpret the same phenomenon differently. The objective approach is associated with a positivist research philosophy, which is similar to scientific approaches. It suggests that acceptable knowledge is generated from 1 source, that reality based on the phenomenon is independent of its context, and that the researcher has no effect on the outcomes of their research work; in other words, the research findings are value free. This stance is less important to resolving the research problem in this study, but its cause-and-effect links do form part of the solution. However, the subjective reasons as to why these links occur are of more practical value to health care providers and their patients; therefore, the emphasis of this study is on the subjective stance.

The subjective stance assumes that knowledge is derived from many sources because individuals observing the same phenomenon will tend to interpret it in diverse ways. These individualized interpretations exist as a consequence of each individual’s diverse values, beliefs, and experiences. Hence, reality is socially constructed, and the researcher is a major part of the study; thus, research is value-laden. Traditionally, although research has been based on either an objective or a subjective stance, researchers have recently focused on a combination of both stances; however, these combinations can vary in their relative proportions. Therefore, the main research design for this study is exploratory, which means that the researcher takes a flexible approach toward obtaining in-depth insights into a new or poorly understood phenomenon, as is the case in this study. If the researcher discovers unexpected information, this information can be pursued and the direction of research altered. In contrast, an objective stance demands stringent adherence to a pre-established design, which can be of much less value for answering certain research questions. The study, which has 2 parts, intends to generate new findings, an inductive approach, while confirming known theory, which is referred to as deduction [ 10 ].

The research philosophy is, therefore, interpretative as the subjective and objective stances can be combined in 1 study [ 11 ]. The research strategy for this study is archival and documentary because secondary data are used to establish the current status of technology acceptance in eHealth. However, moving forward, we will adopt a survey strategy using semistructured interviews to gain the opinions of medical staff and patients on the basis of the recommendations made by this initial study.

Methodology for Data Collection and Analysis

Qualitative research methodology, which is well established with the interpretivist research philosophy and with the theory-building inductive approach, was selected for the study. Therefore, qualitative data will be the main priority; however, some numerical quantitative data may also emerge.

In this study, secondary data will be collected from robust sources such as books, journal articles, specialist magazines, and reliable websites that are noncommercial. These archived data are freely available in electronic form and comprise specialist articles, quality newspaper reports, and interview transcripts, for instance [ 10 ]. The future study will employ semistructured interviews for the collection of empirical data suitable for identifying resistance from employees and patients in Brandenburg, Germany. In this case, feelings of fear and insecurity can be identified by means of standardized interviews as part of the development of acceptance because interview questions will be based on the findings of this study.

The sample for the second study is a purposive, nonprobability sample that uses the opinions of experts to answer the research question; the researcher selects the interviewees and invites them to participate. In-depth knowledge is more important than the generation of findings that are applicable to an entire population, particularly because populations in different communities are likely to hold diverse views [ 11 ]. The sample comprises a combination of the key stakeholders in the Brandenburg local health care structures and its patients, including doctors’ practices, accident and emergency hospital departments, medical professionals, and other health care professionals. Local health care structures play a key role in technology adoption because they can serve as multipliers and act as partners for digital care solutions. The communication target groups, which are the focus of efforts to build new technology acceptance, include key medical opinion leaders: The Brandenburg Chamber of Physicians; The Brandenburg Association of Statutory Health Insurance Physicians; Associations of General Practitioners in Brandenburg; other relevant professional and trade groups; and representatives of the local, county, and state governments.

Data analysis for this study and the empirical study to test its findings will be conducted using content analysis. This means that the data collected and transcribed will be scrutinized to identify keywords and phrases that are associated with technology acceptance and how the development of acceptance might be enhanced and its diffusion rate accelerated. These words and phrases will be organized into major themes, which are interpreted for meaning by the originator or originators, and can then be discussed, summarized, and presented in tables and charts [ 12 ].

All ethical standards associated with social science research are applied to this study. Although the first part is solely informed by secondary data, the researchers aim to interpret the data such that it reflects its original emphasis rather than the researchers’ own preferences. In the future empirical study, the researchers will ensure that participants do not suffer any harm as a consequence of expressing their opinions, and strict confidentiality will be maintained; in other words, the report will not indicate the source of any view expressed within study interviews [ 11 ].

The increased number of patients being treated is generally caused by the aging society, which incremented the complexity of health care systems. This is due to the number of terminal diseases experienced by these patients and the introduction of new technologies that enable medical professionals to diagnose illnesses more precisely. These technologies also give rise to surgical interventions that are more effective and less invasive. However, to benefit from new technology app, sustainable financial resources must be first organized in a cross-sectoral manner in primary care institutions, specialist clinics, and rehabilitation centers. However, the concept of a boundary-less hospital , although achievable, is hindered internally by insufficient, ineffective network design, silos resulting in poor communication, lack of an interdisciplinary approach, and inefficient processes. eHealth services have the potential to resolve the challenges of treating increasing numbers of patients, including those with chronic diseases, and creating efficient communication between departments. Many studies have demonstrated the benefits of telemonitoring, reducing hospital emissions, and controlling chronic conditions remotely for patients. However, despite these positive facts, uptake has been slow [ 13 ] because the potential cost-saving advantages of new technologies are not always evident to major stakeholders. In some cases, new technologies that comprise eHealth solutions are initially associated with higher costs and more time compared with traditional alternatives [ 14 ].

Major influencers in the adoption of eHealth are reported in empirical studies, for instance, the extent of trust that the patient has in a service provider, perceived user-friendliness of tools, health condition severity, and anonymity when using self-diagnosis tools. Medical professionals have concerns regarding the design of eHealth services and the technologies on which it will rely. Medical professionals also hold subjective opinions of the usefulness of new technology, its complexity, and/or how familiar technology is to end users. Hospital culture, location, and size have impacted the decision makers’ consideration for the relevance of tools such as eHealth applications for radiology and patient scheduling. Hence, there are 3 groups of main stakeholders comprising important subgroups, and these subgroups affect both acceptance and the development of acceptance.

Empirical research on patient acceptance factors affirms the importance of age. Although older people tend to need health care services the most, this group is often averse to technology, to the point where customized interventions are needed to support tool adoption. Despite widespread adoption of mobile technologies, such as smartphones, in Germany, with millions of people downloading apps, very few older adults use eHealth apps, preferring websites and email [ 15 ]. This suggests a lack of awareness of the benefits of these applications. In general, the study demonstrated that acceptance is a multistage process and that patients developed acceptance according to defined stages and at different speeds. Various organizations and medical professionals serve to raise awareness within the health care system; therefore, service providers should increase their marketing efforts. This might include highlighting benefits to patients through enhanced communication with medical professionals and greater access to support and 24/7 monitoring of known illnesses. Medical professionals could also leverage patient awareness of the potential for individualized service because they hold access to electronically organized patient information, which can be continuously updated.

Medical professionals and organizations could also inform patients of reliable medical websites, which provide information on the benefits and costs of eHealth. In addition, health service and medical professionals should elicit feedback from patients to support more effective use of eHealth tools and help improve the quality of these tools. In effect, patients need to be involved in the development of eHealth acceptance [ 13 ].

eHealth cards were introduced in Germany approximately 10 years ago upon their mandated use. An empirical study found that primary care doctors felt that eHealth could lead to fewer prescription errors and improve communication among various individuals and groups providing patient care. However, doctors also stated that their involvement in technology development and their ICT expertise were very low. The study also found that 46% of the variance in the perceived usefulness of eHealth cards was related to IT capability [ 16 ]. Health care professionals’ motivation to use eHealth records depended on the quality of interaction with the patient; however, lack of time, workload volume, perception of technology as a major threat to medical professional autonomy, and potential use of technology as a management control tool were significant barriers, according to a systematic literature review of 52 studies [ 17 ]. The extent of IT support and training had a substantial impact on the acceptance and implementation of eHealth technology by medical professionals. If there was no standard process and procedure for the health care organizations at the local, national, and regional levels, doctors and managers were less motivated to use the system.

Patients tended to be more positive about eHealth technologies than the other 2 user groups, recognizing that they had autonomy in their health management. If managers simply imposed eHealth techniques and processes for health professionals and other staff, the failure rate was high. In contrast, when a planning and implementation process involved user groups and a bottom-up development system, enthusiasm and commitment were generated. Hence, the actual change management process was the driver of success or failure. This review also identified that the most frequent reasons for acceptance of eHealth records were design, technical concerns, privacy and security, capacity for fully integrated health information systems within and across organizational boundaries, ease of use, costs, familiarity, and productivity. A total of 4 health care user groups were the subject of 3 linked studies: doctors, other health care professionals, health information professionals, and managers [ 17 ]. The participants were asked to rate the importance of and potential for implementation of 10 factors. Here, participants agreed that importance and applicability were criteria for success. There was a high agreement among managers that interoperability and outcome expectancy were the most important factors, whereas high levels of consensus among health care and health information professionals focused on perceived usefulness, productivity, motivation, and participation of end users in implementation. In addition, although health care professionals agreed that patient and health professional interaction, time constraints, workload, and available resources were important, an additional area of high agreement among them was management. These findings illustrate the differing priorities of the user groups, who therefore have different roles to play in the implementation process.

The volume of data associated with eHealth necessitates greater cognitive effort and creates a higher administrative burden. Consequently, many key players perceive eHealth solutions as an additional time-consuming effort rather than as a source of useful applications. Moreover, 1 reason for this opinion is that these individuals were not invited to participate in the process of developing technical solutions, and therefore, their needs were not considered. Consequently, they failed to understand how innovation supports their daily tasks. In Germany, awareness of eHealth solutions is lower than that in other countries. Therefore, new medical technologies are not widespread, leading to an information deficit. Older adults are not sufficiently familiar with telemedicine supplies and products, and this lack of awareness is aggravated by the paucity of cross-functional interactions among various health care sectors. As no reliable and protected nationwide infrastructure exists, deficiencies in the quality of care and efficiency of administrative and delivery processes occur. Manual collection and transmission of data also generate administrative delays and can be a source of error, which results in the real potential of eHealth solutions being underappreciated. In most cases, low technology–related expectations act as a self-fulfilling prophecy because major stakeholders, such as doctors, imagine that their peers will not fully support eHealth solutions and will not exchange data consistently. Therefore, developing an extensive user community to fully and successfully implement eHealth is less likely to occur in the short term [ 18 ].

Principal Findings

The discussion of the research findings focuses on the challenges to the effective development of acceptance, which were revealed by this study and compared with the theoretical framework presented in the Introduction. In health care, the decision to use or to avoid new technologies depends on various factors, as suggested in previous studies [ 3 ]. Although some factors are common to patients, medical professionals, and health care organizations and their managers, there exist substantial differences in users’ perceptions of the importance of each factor. This observation agrees with earlier research that posits that decision making regarding acceptance is a subjective process [ 4 , 5 ]. For example, some doctors expressed concerns that technology could affect professional autonomy while diagnosing or treating patients. Another concern was that the organizations might use eHealth tools as means of controlling doctors. These perceptions succeed in generating negative attitudes toward implementing change. Doctors perceived technology as a positive factor by potentially reducing errors when prescribing patient treatments and as a means of improving communication with other groups and individuals caring for patients. However, doctors stated that their involvement in the development of technology and their ICT expertise were very low. The study also found that 46% of the variance in the perceived usefulness of the eHealth card related to IT capability [ 19 ].

Importantly, there were different levels of agreement among user groups on the 10 criteria considered to be important for eHealth adoption. Outcome expectancy and interoperability were the most important to managers, whereas perceived usefulness, productivity, and motivation were important to health care professionals. However, there was a high consensus among medical professionals regarding the importance of patient-carer interactions, available time, workload, and available resources. Interestingly, they also emphasized the importance of end-user involvement in implementation. Managers and medical professionals considered that the lack of standardization and integration among health care systems was a huge demotivating factor for eHealth implementation; therefore, this was not a facilitating condition. Based from the UTAUT model, the social condition factor is the most important aspect as it is represented by the age of patients. This factor therefore measures the complexity of the technology when compared with traditional methods shown by the way change was introduced, which can either be imposed by a top-down process or ushered in by a bottom-up process [ 6 ]. Ease of use was a general factor that was reinforced by these findings and support measures such as IT training and support for medical professionals, as reflected by the TAM [ 4 , 5 ]. Doctors also suggested that their involvement in technology implementation was low and that ICT expertise was an issue for them (and for all stakeholders generally) because the capacity to use the electronic system had a 46% effect on how useful they felt the technology to be. These findings also align with the innovation diffusion theory [ 9 ] in terms of relative advantage, perceived complexity, time, and opportunity to evaluate the technologies while undergoing training, with the opportunity to observe its potential advantages indicating how easy it is to learn. The opportunity to evaluate technology before deciding whether to adopt it and the observability of technology-related advantages were also inferred in the training and support that health professionals felt were necessary for acceptance. This study also suggested that the patients were encouraged by the eHealth-related capacity for health self-management; however, trust in the care provider, system’s ease of use, severity of the medical condition, and data security were additional concerns [ 6 ]. In addition, technology did not appear to be an issue for patients because they had already used it; rather, issues centered around lack of awareness of the usefulness of the technology.

Recommendations for the Planned Electronic Health Center

The findings of this study have generated a range of recommendations for the planned eHealth center, which are presented in this final section.

The major stakeholders, who will be the users of the telemedicine processes, must be involved in the design of the eHealth centers and associated technologies. All those who are involved should be active participants in each phase of the innovation process as part of responsible research and innovation. To make an informed contribution, all medical professionals need to be informed about the major features of the innovation and its major benefits, especially effective treatment of more patients, with lower effort per patient. A transparent, accurate, user-centered ICT strategy that acknowledges feelings of insecurity and ensures that the information provided meets the needs of the target group must be devised. Merely instructing users on how to use the technology is insufficient for gaining their interest and commitment. Transfer of knowledge and skills in terms of the practical impact that technology can have on health care outcomes must be an integral part of the learning process. The implementation strategy must also include interventions that build a positive attitude toward the technology among various target groups of patients and the general population. Hence, the implementation strategy must be integrated into the overall eHealth strategy with a prolonged rollout period. This will enable all stakeholder groups to adapt and acknowledge the fact that technology diffusion occurs at different rates. Involving all stakeholder groups appropriately in the development of the change interventions will reduce their resistance to change and enable introspection of the groups’ perceived barriers to implementation. The advantages will become more observable to each group, and the realization that they each have different ideas about what those advantages are will be better appreciated. Individuals responsible for the implementation process must be regarded as trustworthy and proficient. This will encourage them to visibly demonstrate their support for the change and their role in accomplishing technology implementation.

With regard to the effective use of technology, professional learning and development personnel must introduce the various applications and explain their functionality to potential users, medical institution employees, and patients.

To address the perceived lack of cross-functionality, the communication among various key players must be improved and simplified. This could bring about a change in the traditional structures. The creation of a high level of acceptance through communication, participation, and support is an important condition for countrywide care delivery through eHealth solutions. Each innovation needs to be adapted to the wishes of the target group. Patient adherence is obtained after acceptance has been secured among employees of medical facilities, reinforcing the need for the acknowledgment of different rates of technology acceptance. Patients’ acceptance of innovation depends on their perceived ability to both use and directly benefit from it. Prejudices have a negative effect on patients’ readiness to deal intensively with digital companions , for instance. However, patient acceptance can be enhanced if doctors, surgery staff, community nurses, and other patients convey a positive perception toward the respective innovations. Conversations regarding change should take place with patients to increase their awareness and provide an opportunity to identify their resistance factors and overcome them. As patients’ positive perception depends on positive emotions and moods, their emotional participation must be encouraged, potentially through enjoyable elements that can be integrated into health care apps.

Patients’ acceptance of innovation could also be improved by offering sessions in health or community centers where the technology could be tested. Trained individuals would be available to offer support as the patients test the technical innovation. These settings also offer the potential for observability as other patients discuss the usefulness of the devices. During this time, a trained individual should be available to answer patients’ questions and explain the hardware and functions. A positive experience could be the first step toward patients developing a connection with the innovation because, in these settings, patients can be made to feel safe as their use is monitored. Users must interact with technologies and test their multipurpose options. Simulation environments can also be useful, including living labs. Technology users are important sources of information throughout all phases of product development. The simulation environment, with support from trained personnel, is important for developing acceptance, particularly among older adults residing in Brandenburg. Fortunately, these individuals are usually interested test users. The transdisciplinary experience data obtained from a living lab can subsequently be integrated into the structural concept of the eHealth center.

The demonstration of added value for patients and care providers is important, although some patients and medical staff react with skepticism to technical innovations and fear excessive external control. These feelings of insecurity among patients can be reduced by interactions with qualified and aware health care staff in living lab settings. Targeted health promotion through regional media, advertisements, or radio spots can create awareness of the advantages of eHealth centers. Related marketing objectives should include generating as much persuasion, memory value, and attention value as possible. Knowledge-imparting campaigns and information seminars that notify target stakeholder groups of relevant technology features are an additional option. As many patients view Web-based information before a consultation, there is danger that they could receive incorrect information or apply the information in the wrong context. The eHealth center gives patients quality-assured information and serves as an informational health platform; it can also recommend robust online sources.

As described in the Introduction, service providers view innovations in a positive way when the innovations have a positive effect on their daily activities. The acceptance of innovation among health professionals in Brandenburg will be achieved when the patient care goals are achieved more quickly and rendered at a higher quality through the use of new service delivery processes. This also applies to interface management and information flow. Employees will accept a technical innovation if resource use during service provision is lower and/or the revenue obtained by accessing new target groups is higher. Therefore, the eHealth center must help reduce professional burdens such as time pressure or the steadily rising mobility and documentation requirements in Brandenburg. Confident health professionals can convey their positive attitudes regarding innovation to their patients.

Conservative organizational forms, such as hospitals and their employees, often fail to easily adapt to technical innovations. Doctors, in particular, may hide their opposition to change. This could present a challenge for the planned eHealth center that is characterized by technological and procedural innovations. The benefits of innovation must be presented to hospitals in a very tactful manner; the reality of the need for economic efficiency exposes hospitals to challenges that relate to cost savings and competition. The eHealth center could help hospitals to stay competitive in the long run by facilitating the delivery of high-quality care while producing cost-effective services.

In view of the less-developed IT infrastructure in Brandenburg compared with other federal states, the creators of the eHealth center should lobby for internet access, broadband expansion, and rapid data transmission. Population groups with low digital affinity should be assisted in their efforts to acquire digital competence and suitable equipment through cooperation and coordination. Special attention should be given to data protection. Acceptance by the key players depends on the degree to which special encryption methods ensure the protection of personal data. However, patients with significant illnesses care less about the storage and protection of their data and more about their health care [ 20 ].

Further Questions or Research Issues

The research on acceptance involves understanding the feelings of insecurity, fear, and apprehension experienced by different stakeholders. These feelings can be identified through suitable data collection processes. Comparing stakeholder perceptions is important to this ongoing study: initially, a sample from the target group will be interviewed about their feelings of insecurity and subjective perceptions of how technological innovations are implemented. Subsequently, the participants will be asked to test the innovation within a simulation or other environment that facilitates innovation testing. After a designated period, the participants will be interviewed again to reassess their feelings and perceptions. This process is intended to provide valuable insights into how testing an innovation positively influences key players’ perceptions.

This research into the psychological indicators of acceptance shows that acceptance is critically dependent on the subject, object, and the general conditions that surround acceptance. In the case of the construction of an eHealth center, the acceptance subjects include target groups of patients (or the general population of individuals interested in preventative medicine issues) and medical professionals. The acceptance object is the technical innovation. The technical and social conditions are considered exogenous as they cannot be influenced easily or quickly. These general conditions are diverse. The first step toward securing the acceptance of digital solutions and innovative medical technology by patients and professionals is to understand their anxieties and feelings of insecurity on the basis of empirical study findings. This insight will create an opportunity to further categorize and evaluate the specific issues of the target group of disabled and elderly persons in the federal state of Brandenburg. The final step will be the generation of reliable recommendations for action for the eHealth center of the Federal State of Brandenburg. For both groups, acceptance can be generated only through a directed, transparent awareness campaign that provides users with sufficient information and the opportunity to test new technologies. Hence, users can directly experience the benefits of the technologies and acquire a positive attitude toward the new products.

Abbreviations

Conflicts of Interest: None declared.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Immersive technology and building information modeling (bim) for sustainable smart cities.

1. Introduction

2. methodology, 2.1. phase 1: determine the research problem, 2.2. phase 2: search the database, 2.3. phase 3: define the criteria and quality standards, 2.4. phase 4: macro-quantitative analysis, 2.5. phase 5: micro-qualitative analysis, 3.1. literature publications, 3.2. keyword network overlay (time period), 3.3. co-occurrence keyword network visualization (clustering), 3.4. keyword burst detection, 3.5. immersive technology, 3.6. sustainable smart city, 3.7. lifecycle assessment.

• Beginning stage: design, cost estimation, tendering, decision making, procurement.
• Intermediate stage: manufacturing stage (manufacturing and construction, transportation, packaging, and other production activities) and utilization stage (operation, facility management, consumer use, maintenance, and renovation).
• End stage: deconstruction/disassembly, reuse/remanufacturing/recycling, and waste disposal, optimization/iterative.

3.7.1. Beginning Stage

3.7.2. intermediate stage, 3.7.3. end stage, 4. discussion, 4.1. research hotspots and development trends, 4.1.1. research hotspots of immersive technology and bim in sustainable smart cities.

• Intelligent Solutions for Public Services

4.1.2. Development Trends of Immersive Technology and BIM in Sustainable Smart Cities

4.2. challenges and limitations, 4.2.1. challenges in implementing immersive technology for sustainable smart cities, 4.2.2. challenges in improving bim in sustainable smart cities, 4.2.3. limitations, 5. conclusions.

• In terms of research methodology and context, this paper employs a triangulation research method combining a quantitative method via bibliometric analysis and a qualitative method via content analysis to investigate the relationship between immersive technology and BIM in the context of sustainable development. It is found that immersive technology and BIM have sustained potential in addressing sustainable urban development via various fields of sustainable smart cities, including transportation systems and public services, energy conservation and sustainability, medical care, cultural heritage and tourism, and education, which is applied to the whole process of the lifecycle of architecture and engineering, provides a comprehensive perspective and foundation on urban planning and decision making, and promotes the design, construction, and management of sustainable smart cities.
• In terms of research technique, although the bibliometric analysis method is used to reveal the current state of research and application areas of the content of the immersive technology and BIM for sustainable smart cities, it only provides a rough measure due to the complexity of the scientific development. Hence, this paper adopts the visual bibliometric map tool VOSviewer and Citespace to assist the analysis in order to minimize the impact of this limitation on the research, to make the research more rigorous, and to suggest future research with a reliable methodological reference.
• In terms of research approach, this paper utilizes software tools in a multidimensional and systematic perspective by generating two overlay knowledge maps i.e., overlay knowledge mapping and time period knowledge mapping (knowledge top view and side view) with various perspectives such as specific knowledge network cluster, citation burst, and other detailed highlighted knowledge maps, to form a knowledge universe map at a “macro-knowledge” level stage followed by a “micro-knowledge” level stage comprising a standard knowledge system, such as lifecycle assessment, to systematically generate knowledge and establish relationships ( Figure 1 ), which provids a way for future research.

Author Contributions

Data availability statement, acknowledgments, conflicts of interest.

• Suh, A.; Prophet, J. The state of immersive technology research: A literature analysis. Comput. Hum. Behav. 2018 , 86 , 77–90. [ Google Scholar ] [ CrossRef ]
• Hyarat, E.; Hyarat, T.; Al Kuisi, M. Barriers to the Implementation of Building Information Modeling among Jordanian AEC Companies. Buildings 2022 , 12 , 150. [ Google Scholar ] [ CrossRef ]
• Dong, R.R.; Martin, A. Research on Barriers and Government Driving Force in Technological Innovation of Architecture Based on BIM. Eurasia J. Math. Sci. Technol. Educ. 2017 , 13 , 5757–5763. [ Google Scholar ] [ CrossRef ]
• Shi, Y.; Du, J.; Lavy, S.; Zhao, D. A Multiuser Shared Virtual Environment for Facility Management. In Proceedings of the International Conference on Sustainable Design, Engineering and Construction (ICSDEC), Arizona State University, Coll Avenue Commons, Tempe, AZ, USA, 18–20 May 2016. [ Google Scholar ]
• Alizadehsalehi, S.; Hadavi, A.; Huang, J.C. Assessment of AEC Students’ Performance Using BIM-into-VR. Appl. Sci. 2021 , 11 , 3225. [ Google Scholar ] [ CrossRef ]
• Porsani, G.B.; de Lersundi, K.D.; Gutiérrez, A.S.O.; Bandera, C.F. Interoperability between Building Information Modelling (BIM) and Building Energy Model (BEM). Appl. Sci. 2021 , 11 , 2167. [ Google Scholar ] [ CrossRef ]
• GhaffarianHoseini, A.; Zhang, T.; Nwadigo, O.; Ghaffarian Hoseini, A.; Naismith, N.; Tookey, J.; Raahemifar, K. Application of nD BIM Integrated Knowledge-based Building Management System (BIM-IKBMS) for inspecting post-construction energy efficiency. Renew. Sustain. Energy Rev. 2017 , 72 , 935–949. [ Google Scholar ] [ CrossRef ]
• Panchanathan, S.; McDaniel, T.; Tadayon, R.; Rukkila, A.; Venkateswara, H.; IEEE. Smart Stadia as Testbeds for Smart Cities: Enriching Fan Experiences and Improving Accessibility. In Proceedings of the International Conference on Computing, Networking and Communications (ICNC), Honolulu, HI, USA, 18–21 February 2019. [ Google Scholar ]
• Zhou, Y. The Application Trend of Digital Finance and Technological Innovation in the Development of Green Economy. J. Environ. Public Health 2022 , 2022 , 1064558. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Wu, Z.Z.; Chen, C.H.; Cai, Y.Z.; Lu, C.; Wang, H.; Yu, T. BIM-Based Visualization Research in the Construction Industry: A Network Analysis. Int. J. Environ. Res. Public Health 2019 , 16 , 3473. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ali, K.N.; Alhajlah, H.H.; Kassem, M.A. Collaboration and Risk in Building Information Modelling (BIM): A Systematic Literature Review. Buildings 2022 , 12 , 571. [ Google Scholar ] [ CrossRef ]
• Choi, J.; Lee, S. A Suggestion of the Alternatives Evaluation Method through IFC-Based Building Energy Performance Analysis. Sustainability 2023 , 15 , 1797. [ Google Scholar ] [ CrossRef ]
• Guedes, A.L.A.; Alvarenga, J.C.; Goulart, M.D.S.; Rodriguez, M.; Soares, C.A.P. Smart Cities: The Main Drivers for Increasing the Intelligence of Cities. Sustainability 2018 , 10 , 3121. [ Google Scholar ] [ CrossRef ]
• Ramaprasad, A.; Sánchez-Ortiz, A.; Syn, T. A Unified Definition of a Smart City. In Proceedings of the 16th IFIP WG 8.5 International Conference on Electronic Government (EGOV), St Petersburg, Russia, 4–7 September 2017; pp. 13–24. [ Google Scholar ]
• Ismagilova, E.; Hughes, L.; Rana, N.P.; Dwivedi, Y.K. Security, Privacy and Risks Within Smart Cities: Literature Review and Development of a Smart City Interaction Framework. Inf. Syst. Front. 2022 , 24 , 393–414. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Liu, W.S. The data source of this study is Web of Science Core Collection? Not enough. Scientometrics 2019 , 121 , 1815–1824. [ Google Scholar ] [ CrossRef ]
• van Eck, N.J.; Waltman, L. Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics 2010 , 84 , 523–538. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ding, X.; Yang, Z. Knowledge mapping of platform research: A visual analysis using VOSviewer and CiteSpace. Electron. Commer. Res. 2022 , 22 , 787–809. [ Google Scholar ] [ CrossRef ]
• Jamei, E.; Mortimer, M.; Seyedmahmoudian, M.; Horan, B.; Stojcevski, A. Investigating the Role of Virtual Reality in Planning for Sustainable Smart Cities. Sustainability 2017 , 9 , 2006. [ Google Scholar ] [ CrossRef ]
• Briciu, A.; Briciu, V.-A.; Kavoura, A. Evaluating How ‘Smart’ Brașov, Romania Can Be Virtually via a Mobile Application for Cultural Tourism. Sustainability 2020 , 12 , 5324. [ Google Scholar ] [ CrossRef ]
• Nasralla, M.M. Sustainable Virtual Reality Patient Rehabilitation Systems with IoT Sensors Using Virtual Smart Cities. Sustainability 2021 , 13 , 4716. [ Google Scholar ] [ CrossRef ]
• Kamari, A.; Paari, A.; Torvund, H.O. BIM-Enabled Virtual Reality (VR) for Sustainability Life Cycle and Cost Assessment. Sustainability 2021 , 13 , 249. [ Google Scholar ] [ CrossRef ]
• de la Plata, A.R.M.; Franco, P.A.C.; Sánchez, J.A.R. Architectural Survey, Diagnostic, and Constructive Analysis Strategies for Monumental Preservation of Cultural Heritage and Sustainable Management of Tourism. Buildings 2022 , 12 , 1156. [ Google Scholar ] [ CrossRef ]
• Chiabrando, F.; Sammartano, G.; Spanò, A. Historical Buildings Models And Their Handling Via 3d Survey: From Points Clouds To User-Oriented Hbim. In Proceedings of the 23rd Congress of the International-Society-for-Photogrammetry-and-Remote-Sensing (ISPRS), Prague, Czech Republic, 12–19 July 2016. [ Google Scholar ]
• Wiberg, A.H.; Lovhaug, S.; Mathisen, M.; Tschoerner, B.; Resch, E.; Erdt, M.; Prasolova-Forland, E. Visualisation of KPIs in zero emission neighbourhoods for improved stakeholder participation using Virtual Reality. In Proceedings of the Sustainable Built Environment D-A-CH Conference (SBE), Graz University Technol, Graz, Austria, 11–14 September 2019. [ Google Scholar ]
• De Luca, D.; Del Giudice, M.; Dellosta, M.; Fonsati, A.; Osello, A.; Ugliotti, F.M. Augmented And Virtual Reality For Smart Cities Users’ Awareness. In Proceedings of the 11th International Conference on Technology, Education and Development (INTED), Valencia, Spain, 6–8 March 2017. [ Google Scholar ]
• Predescu, A.; Mocanu, M. Increasing Collaboration and Participation Through Serious Gaming for Improving the Quality of Service in Urban Water Infrastructure. In Proceedings of the 22nd International Conference on Business Information Systems (BIS), Seville, Spain, 26–28 June 2019. [ Google Scholar ]
• Shih, N.-J.; Chen, Y. LiDAR-and AR-Based Monitoring of Evolved Building Facades upon Zoning Conflicts. Sensors 2020 , 20 , 5628. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Vassileva, S.; Kovatcheva, E. The Innovative Model For Interactivity In Bulgarian Museums. In Proceedings of the 10th Annual International Conference of Education, Research and Innovation (ICERI), Seville, Spain, 16–18 November 2017. [ Google Scholar ]
• Alvarez-Marin, A.; Velazquez-Iturbide, J.A. Augmented Reality and Engineering Education: A Systematic Review. Ieee Trans. Learn. Technol. 2021 , 14 , 817–831. [ Google Scholar ] [ CrossRef ]
• Chew, M.Y.L.; Teo, E.A.L.; Shah, K.W.; Kumar, V.; Hussein, G.F. Evaluating the Roadmap of 5G Technology Implementation for Smart Building and Facilities Management in Singapore. Sustainability 2020 , 12 , 10259. [ Google Scholar ] [ CrossRef ]
• Khan, A.; Sepasgozar, S.; Liu, T.; Yu, R. Integration of BIM and Immersive Technologies for AEC: A Scientometric-SWOT Analysis and Critical Content Review. Buildings 2021 , 11 , 126. [ Google Scholar ] [ CrossRef ]
• Srivastava, A.; Jawaid, S.; Singh, R.; Gehlot, A.; Akram, S.V.; Priyadarshi, N.; Khan, B. Imperative Role of Technology Intervention and Implementation for Automation in the Construction Industry. Adv. Civ. Eng. 2022 , 2022 , 6716987. [ Google Scholar ] [ CrossRef ]
• Bourhim, E.M.; Cherkaoui, A. How Can the Virtual Reality Help in Implementation of The Smart City? In Proceedings of the 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT), Kanpur, India, 6–8 July 2019. [ Google Scholar ]
• Alabdali, S.A.; Pileggi, S.F.; Cetindamar, D. Influential Factors, Enablers, and Barriers to Adopting Smart Technology in Rural Regions: A Literature Review. Sustainability 2023 , 15 , 7908. [ Google Scholar ] [ CrossRef ]
• Shahrokni, H.; Årman, L.; Lazarevic, D.; Nilsson, A.; Brandt, N. Implementing Smart Urban Metabolism in the Stockholm Royal Seaport: Smart City SRS. J. Ind. Ecol. 2015 , 19 , 917–929. [ Google Scholar ] [ CrossRef ]
• Pournaras, E. Proof of witness presence: Blockchain consensus for augmented democracy in smart cities. J. Parallel Distrib. Comput. 2020 , 145 , 160–175. [ Google Scholar ] [ CrossRef ]
• Vitello, P.; Capponi, A.; Fiandrino, C.; Giaccone, P.; Kliazovich, D.; Bouvry, P.; IEEE. High-Precision Design of Pedestrian Mobility for Smart City Simulators. In Proceedings of the IEEE International Conference on Communications (ICC), Workshop on Integrating UAVs into 5G, Kansas City, MO, USA, 20–24 May 2018. [ Google Scholar ]
• Allam, Z.; Jones, D.S. Future (post-COVID) digital, smart and sustainable cities in the wake of 6G: Digital twins, immersive realities and new urban economies. Land Use Policy 2021 , 101 , 105201. [ Google Scholar ] [ CrossRef ]
• Vázquez-Rowe, I.; Córdova-Arias, C.; Brioso, X.; Santa-Cruz, S. A Method to Include Life Cycle Assessment Results in Choosing by Advantage (CBA) Multicriteria Decision Analysis. A Case Study for Seismic Retrofit in Peruvian Primary Schools. Sustainability 2021 , 13 , 8139. [ Google Scholar ] [ CrossRef ]
• Lenfers, U.A.; Ahmady-Moghaddam, N.; Glake, D.; Ocker, F.; Osterholz, D.; Ströbele, J.; Clemen, T. Improving Model Predictions-Integration of Real-Time Sensor Data into a Running Simulation of an Agent-Based Model. Sustainability 2021 , 13 , 7000. [ Google Scholar ] [ CrossRef ]
• Osello, A.; Del Giudice, M.; Guinea, A.M.; Rapetti, N.; Ronzino, A.; Ugliotti, F.M.; Migliarino, L. Augmented Reality and Gamification Approach within The DIMMER Project. In Proceedings of the 9th International Technology, Education and Development Conference (INTED), Madrid, Spain, 2–4 March 2015. [ Google Scholar ]
• Gutierrez-Bucheli, L.A.; Caldarón, M.C.M.; Londoño-Acevedo, M.C.; Ponz-Tienda, J.L. BIM and IPD as vehicle in the teaching and learning process of Project delivery in civil engineering. In Proceedings of the 9th Annual International Conference of Education, Research and Innovation (iCERi), Seville, Spain, 14–16 November 2016. [ Google Scholar ]
• Carbonari, A.; Franco, C.; Naticchia, B.; Spegni, F.; Vaccarini, M. A Mixed Reality Application for the On-Site Assessment of Building Renovation: Development and Testing. Sustainability 2022 , 14 , 3239. [ Google Scholar ] [ CrossRef ]
• Vite, C.; Horvath, A.S.; Neff, G.; Moller, N.L.H.; ACM. Bringing Human-Centredness to Technologies for Buildings An agenda for linking new types of data to the challenge of sustainability. In Proceedings of the 14th Biannual Conference of the Italian-SIGCHI-Chapter (CHIItaly), Bozen Bolzano, Italy, 11–13 July 2021. [ Google Scholar ]
• Setiawan, R.; Devadass, M.M.V.; Rajan, R.; Sharma, D.K.; Singh, N.P.; Amarendra, K.; Ganga, R.K.R.; Manoharan, R.R.; Subramaniyaswamy, V.; Sengan, S. IoT Based Virtual E-Learning System for Sustainable Development of Smart Cities. J. Grid Comput. 2022 , 20 , 24. [ Google Scholar ] [ CrossRef ]
• Buyukdemircioglu, M.; Kocaman, S. Development Of A Smart City Concept In Virtual Reality Environment. In Proceedings of the 24th ISPRS Congress on Imaging Today, Foreseeing Tomorrow, Nice, France, 6–11 June 2022. [ Google Scholar ]
• Manogaran, G.; Rawal, B.S.; Song, H.B.; Wang, H.H.; Hsu, C.H.; Saravanan, V.; Kadry, S.N.; Shakeel, P.M. Optimal Energy-Centric Resource Allocation and Offloading Scheme for Green Internet of Things Using Machine Learning. ACM Trans. Internet Technol. 2022 , 22 , 36. [ Google Scholar ] [ CrossRef ]
• Wu, J.; Sun, L.; Peng, D.D.; Siuly, S. A Micro Neural Network for Healthcare Sensor Data Stream Classification in Sustainable and Smart Cities. Comput. Intell. Neurosci. 2022 , 2022 , 4270295. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Kim, D.; Kim, S. The Role of Mobile Technology in Tourism: Patents, Articles, News, and Mobile Tour App Reviews. Sustainability 2017 , 9 , 2082. [ Google Scholar ] [ CrossRef ]
• Anand, A.; Rufuss, D.D.W.; Rajkumar, V.; Suganthi, L. Evaluation of Sustainability Indicators in Smart Cities for India Using MCDM Approach. In Proceedings of the 4th International Conference on Power and Energy Systems Engineering, (CPESE), Berlin, Germany, 25–29 September 2017; pp. 211–215. [ Google Scholar ]
• Khoshdelnezamiha, G.; Liew, S.C.; Bong, V.N.S.; Ong, D.E.L. A BIM-Based Automated Assessment Tool for Green Building Index. In Proceedings of the 2nd International Conference on Materials Technology and Energy (ICMTE), Curtin University Malaysia, Sarawak, Malaysia, 6–8 November 2019. [ Google Scholar ]
• Li, H.; Fang, S.; Chen, L.; Menadue, V.; Martin, S. Extended reality (XR)-A magic box of digitalization in driving sustainable development of the construction industry: A critical review. Sustain. Dev. 2023 , 32 , 2830–2845. [ Google Scholar ] [ CrossRef ]
• Yaqoob, I.; Salah, K.; Jayaraman, R.; Omar, M. Metaverse applications in smart cities: Enabling technologies, opportunities, challenges, and future directions. Internet Things 2023 , 23 , 100884. [ Google Scholar ] [ CrossRef ]
• Li, X.; Wu, P.; Shen, G.Q.; Wang, X.; Teng, Y. Mapping the knowledge domains of Building Information Modeling (BIM): A bibliometric approach. Autom. Constr. 2017 , 84 , 195–206. [ Google Scholar ] [ CrossRef ]
• Santos, R.; Costa, A.A.; Grilo, A. Bibliometric analysis and review of Building Information Modelling literature published between 2005 and 2015. Autom. Constr. 2017 , 80 , 118–136. [ Google Scholar ] [ CrossRef ]
• Prebanic, K.R.; Vukomanovic, M. Realizing the Need for Digital Transformation of Stakeholder Management: A Systematic Review in the Construction Industry. Sustainability 2021 , 13 , 12690. [ Google Scholar ] [ CrossRef ]
• Meng, Q.F.; Zhang, Y.F.; Li, Z.; Shi, W.X.; Wang, J.; Sun, Y.H.; Xu, L.; Wang, X.Y. A review of integrated applications of BIM and related technologies in whole building life cycle. Eng. Constr. Archit. Manag. 2020 , 27 , 1647–1677. [ Google Scholar ] [ CrossRef ]
• Jiao, Z.; Du, X.; Liu, Z.; Liu, L.; Sun, Z.; Shi, G.; Liu, R. A Review of Theory and Application Development of Intelligent Operation Methods for Large Public Buildings. Sustainability 2023 , 15 , 9680. [ Google Scholar ] [ CrossRef ]
• Afzal, M.; Li, R.Y.M.; Shoaib, M.; Ayyub, M.F.; Tagliabue, L.C.; Bilal, M.; Ghafoor, H.; Manta, O. Delving into the Digital Twin Developments and Applications in the Construction Industry: A PRISMA Approach. Sustainability 2023 , 15 , 16436. [ Google Scholar ] [ CrossRef ]
• Darko, A.; Chan, A.P.C.; Yang, Y.; Tetteh, M.O. Building information modeling (BIM)-based modular integrated construction risk management—Critical survey and future needs. Comput. Ind. 2020 , 123 , 103327. [ Google Scholar ] [ CrossRef ]
• Pan, Y.; Zhang, L. Integrating BIM and AI for Smart Construction Management: Current Status and Future Directions. Arch. Comput. Methods Eng. 2023 , 30 , 1081–1110. [ Google Scholar ] [ CrossRef ]
• Fiandrino, C.; Anjomshoa, F.; Kantarci, B.; Kliazovich, D.; Bouvry, P.; Matthews, J.N. Sociability-Driven Framework for Data Acquisition in Mobile Crowdsensing over Fog Computing Platforms for Smart Cities. IEEE Trans. Sustain. Comput. 2017 , 2 , 345–358. [ Google Scholar ] [ CrossRef ]
• Caldas, L.R.; Silva, M.V.; Silva, V.P.; Carvalho, M.T.M.; Toledo, R.D. How Different Tools Contribute to Climate Change Mitigation in a Circular Building Environment? -A Systematic Literature Review. Sustainability 2022 , 14 , 3759. [ Google Scholar ] [ CrossRef ]
• Luengo-Oroz, M.; Bullock, J.; Pham, K.H.; Lam, C.S.N.; Luccioni, A. From Artificial Intelligence Bias to Inequality in the Time of COVID-19. IEEE Technol. Soc. Mag. 2021 , 40 , 71–79. [ Google Scholar ] [ CrossRef ]
• Luig, K.; Jansen, D.; Maietti, F.; Coltro, L.; Karadimas, D. Innovative Business Plans for H-BIM Application Related to Alternative Financing Opportunities for Cultural Heritage. In Proceedings of the Conference of the Marie Sklodowska-Curie Initial Training Network for Digital Cultural Heritage (ITN-DCH), Olimje, Slovenia, 23–25 May 2018. [ Google Scholar ]
• Bolognesi, C.; Vespasiani, M.; Zhang, Y. Holographic Visions For Architecture In A Park. In Proceedings of the 24th ISPRS Congress on Imaging Today, Foreseeing Tomorrow, Nice, France, 6–11 June 2022. [ Google Scholar ]
• Xie, M.; Qiu, Y.; Liang, Y.; Zhou, Y.; Liu, Z.; Zhang, G. Policies, applications, barriers and future trends of building information modeling technology for building sustainability and informatization in China. Energy Rep. 2022 , 8 , 7107–7126. [ Google Scholar ] [ CrossRef ]

Share and Cite

Liu, Z.; He, Y.; Demian, P.; Osmani, M. Immersive Technology and Building Information Modeling (BIM) for Sustainable Smart Cities. Buildings 2024 , 14 , 1765. https://doi.org/10.3390/buildings14061765

Liu Z, He Y, Demian P, Osmani M. Immersive Technology and Building Information Modeling (BIM) for Sustainable Smart Cities. Buildings . 2024; 14(6):1765. https://doi.org/10.3390/buildings14061765

Liu, Zhen, Yunrui He, Peter Demian, and Mohamed Osmani. 2024. "Immersive Technology and Building Information Modeling (BIM) for Sustainable Smart Cities" Buildings 14, no. 6: 1765. https://doi.org/10.3390/buildings14061765

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals