How recent is recent for good referencing?

how long a reserch paper can be considered a recent research works

Cited articles (i.e., references) in a research paper play a central role in demonstrating the necessity of the research and establishing the validity and significance of the research results.

Therefore, good referencing practices (e.g., citing relevant, critical, and recent research works on the topics) not only increase the quality of the research paper but also facilitate its peer review and availability to the right audience.

Citing or referencing recent articles in the research paper assures reviewers that an extensive literature review was undertaken while writing the paper and information in the paper is up to date. This builds trust between the authors of the paper and the reviewers, which may influence peer review reports.

How old is gold?

All being said, do we exactly know how old a research article can be before it gets the label of not being recent i.e., an old article not good for citing.

There is consensus among scientists and researchers that articles less than five years old are recent publications. However, it may vary from discipline to discipline. For example, researchers in fast-moving fields (e.g., nanotechnology or artificial intelligence) may feel 5-years is too old whereas those in biology may not have the same feeling.

How many recent references make a research paper contemporaneous?

Santini et al. (20018) suggested that if the most recent reference is more than 5 years or so, it can indicate that a full up to date review of the literature has not been undertaken.

However, the suggestion is a weak indicator of the comprehensiveness of the literature review done while writing a paper as it is based on the measure of only one reference.

To build a robust understanding of the matter, nXr team analyzed how old references of 69 research papers (published in three highly acclaimed journals: Nature, Science, and Cell) were.

why is it important for research to be recent

The graph clearly shows that 50% of references in the articles published in highly acclaimed journals are less than 6 years old. This indicates that well-written articles have the characteristic to cite more recent research papers.

How to get similar information for the references in the research paper you are writing?

No worries! When you cite using nXr reference manager and citation tool , nXr automatically creates a dashboard (accessible from your nXr.iLibrary) for the references in your research paper containing various data visualizations.

In one such visualization, you can see the publication year distribution of the references (see below). nXr also gives you an alert if 50% of the references are more than 5 years old so that you can check them.

why is it important for research to be recent

Logo for UNT Open Books

1 Chapter 1: The Importance of Research Methods and Becoming an Informed Consumer of Research

Case study : student apprehension regarding research methods.

Research Study

Understanding and Measuring Student Apprehension in Criminal Justice Research Methods Courses 1

Research Question

How do we measure disinterest, relevance argumentation, and math anxiety experienced by students enrolled in research methods courses?


It is said that “misery loves company,” so you are not alone in your apprehension and anxiety regarding your research methods course. The problem of student apprehension and anxiety related to taking a research methods course is not new and has been studied for over 25 years. Previously, such apprehension and anxiety appeared to be caused by math anxiety, especially as it applies to statistics. The authors of this article believe that student apprehension goes beyond math anxiety; that math anxiety is too simplistic of an explanation of student fear of research methods courses. Besides math anxiety, the researchers think that apprehension is caused by student indifference to the subject matter and irrelevance of the course because it does not apply to the “real world.” They state that student apprehension in research methods and statistics courses is due to three main factors:

Disinterest (D.);

Relevance Argumentation (RA.), and;

Math Anxiety (MA.).

Taken together, the reconceptualization is known as D.RA.MA., and the combination of these three factors constitutes the D.RA.MA. scale for research methods and statistics courses.

The researchers developed the D.RA.MA. scale by constructing survey questions to measure each factor in the scale (i.e., disinterest, relevance argumentation, and math anxiety). After they developed the survey, they tested it by distributing the survey to three criminal justice classes, totaling 80 students, from a midsized regional comprehensive university in the southern region of the United States. Higher scale scores demonstrate more disinterest, more relevance argumentation, or more math anxiety.

The D.RA.MA. scale consists of 20 survey questions. Ten questions were borrowed from an existing Math Anxiety scale developed by Betz 2 . The researchers then created five items to assess Disinterest and five items intended to measure Relevance Argumentation. The items for the D.RA.MA. scale are illustrated below.

Math Anxiety 3

I usually have been at ease in math classes.

Math does not scare me at all.

I am no good at math.

I don’t think that I could do advanced math.

Generally, I have been secure about attempting math.

For some reason, even though I study, math seems unusually hard for me.

Math has been my worst subject.

My mind goes blank and I am unable to think clearly when working in mathematics.

I think I could handle more difficult math.

I am not the type to do well in mathematics.

Relevance Argumentation 4

I will need research methods for my future work.

I view research methods as a subject that I will rarely use.

Research methods is not really useful for students who intend to work in Criminal Justice.

Knowing research methods will help me earn a living.

Research methods does not reflect the “real world.”

Research Disinterest 5

I am excited about taking research methods.

It would not bother me at all to take more research methods courses.

I expect a research methods class to be boring.

I don’t expect to learn much in research methods.

I really don’t care if I learn anything in research methods, as long as I get the requirement completed.

The Math Anxiety Scale responses for the 80 students ranged from 0 to 30 with a mean of 14, demonstrating a moderate level of math anxiety among the study participants. The responses for Relevance Argumentation ranged from 0 to 12 with a mean of 5.4 while those for Disinterest ranged from 1 to 15 with a mean of 7.0, demonstrating a moderate level of disinterest and relevance argumentation among students regarding research methods. Based on these findings, the study demonstrated that student apprehension regarding research methods courses goes beyond math anxiety and includes two additional factors; disinterest in the subject matter and irrelevance of research methods to the “real world.”

Limitations with the Study Procedure

This research study was designed to develop a broader measure of student apprehension in criminal justice research methods courses. Moving beyond just math anxiety, the researchers accomplished their objective by developing the D.RA.MA. scale; adding disinterest and relevance argumentation to the understanding of student apprehension regarding research methods. As is true for all research, this study is not without limitations. The biggest limitation of this study is the limited sample size. Only 80 students completed the survey. Although this is certainly a good start, similar research (i.e., replication) needs to be completed with larger student samples in different locations throughout the country before the actual quality of the D.RA.MA. scale can be determined.

Impact on Criminal Justice

The D.RA.MA. scale developed in this study identifies disinterest and relevance argumentation, in addition to math anxiety, as part of student apprehension and resistance to research methods. A variety of instructional strategies can be inferred from the D.RA.MA. survey. However, it is important for professors to recognize that no single approach will reduce research methods resistance and apprehension for all students. For example, discussing research methods in a popular culture framework may resonate with students and lead to engaged students who are more interested in the subject matter and identify with the relevance of research methods to criminal justice in general and the future careers of students, in particular. This approach may provide an effective means for combating student disinterest and relevance argumentation in criminal justice research methods courses. At a minimum, it is critical for professors to explain the relevance of research methods to the policies and practices of police, courts, and corrections. Students need to realize that research methods are essential tools for assessing agency policies and practices. Professors will always have D.RA.MA.-plagued students, but recognizing the problem and then developing effective strategies to connect with these students is the challenge all professors face. Experimenting with a multitude of teaching strategies to alleviate the math anxiety, relevance argumentation, and disinterest of criminal justice research methods students will result in more effective teaching and learning.

In This Chapter You Will Learn

What research is and why it is important to be an informed consumer of research

The sources of knowledge development and problems with each

How research methods can dispel myths about crime and the criminal justice system

The steps in the research process

How research has impacted criminal justice operations


As noted in the chapter opening case study, it is expected that you have some anxiety and apprehension about taking this criminal justice research methods course. But, you have taken a significant step toward success in this course by opening up your research methods book, so congratulations are in order. You might have opened this book for a number of reasons. Perhaps it is the first day of class and you are ready to get started on the course material. Perhaps you have a quiz or exam soon. Perhaps the book has been gathering dust on your shelf since the first day of class and you are not doing well in your research methods class and are looking for the book to help with course improvement. Perhaps you are taking a research methods class in the future and are seeing if all the chatter among students is true.

No matter how you got here, two things are probably true. First, you are taking this research methods course because it is a requirement for your major. The bottom line is that most of the students who read this text are required to take a research methods course. While you may think studying research methods is irrelevant to your career goals, unnecessary, overly academic, or perhaps even intimidating, you probably must finish this course in order to graduate. Second, you have heard negative comments about this course. The negative comments mention the difficulty of the course and the relevance of the course (e.g., “I am going to be a police officer, so why do I need to take a research methods course?”). If you are like most students we have experienced in our research methods courses in the past, you are not initially interested in this course and are concerned about whether you will do well in it.

If you are concerned about the course, realize that you are not alone because most students are anxious about taking a research methods course. Also realize that your professor is well aware of student anxiety and apprehension regarding research methods. So, relax and do not think about the entire course and the entire book. Take the course content one chapter, one week at a time. One of the advantages of taking a research methods course is that you learn about the process of research methods. Each chapter builds upon the previous chapters, illustrating and discussing more about the research process. This is certainly an advantage, but it is also critical that you understand the initial chapters in this book so you are not confused with the content discussed in later chapters. In addition to anxiety and apprehension over the course material, research methods can be boring if you only read and learn about it with no particular purpose in mind. Although examples are prevalent throughout the book, as you read this material, it is recommended that you think about the relevancy and application of the topics covered in this book to your specific criminal justice interests. As you continue to read the book, think about how you might use the information you are reading in your current position or your intended profession.

The goal of this research methods book is to develop you into an informed consumer of research. Most, if not all, of your fellow classmates will never conduct their own research studies. However, every one of you will be exposed to research findings in your professional and personal lives for the remainder of your lives. You are exposed to research findings in the media (e.g., television, newspapers, and online), in personal interaction with others (e.g., friends and family, doctors, and professors), as well as in class. You should challenge yourself for this semester to keep a journal and document exposure to research in your daily life outside of college whether through the nightly news, newspapers, magazine articles, Internet, personal conversations, or other means. At the end of the semester, you will be amazed at the amount of research you are exposed to in a short period of time. This book is focused on research exposure and assisting you to become an educated consumer of research by providing you the skills necessary to differentiate between good and not so good research. Why should you believe research findings if the study is faulty? Without being an educated consumer of research, you will not be able to differentiate between useful and not useful research. This book is designed to remedy this problem.

This book was written to make your first encounter with research methods relevant and successful while providing you the tools necessary to become an educated consumer of research. Therefore, this book is written with the assumption that students have not had a prior class on research methods. In addition, this book assumes that practical and evaluative knowledge of research methods is more useful than theoretical knowledge of the development of research methods and the relationship between theory and research. Since the focus of this book is on consumerism, not researcher training, practical and evaluative knowledge is more useful than theoretical knowledge.

It is also important to understand that the professors who design academic programs in criminal justice at the associate and bachelor level believe that an understanding of research methods is important for students. That is why, more than likely, this research methods course is a required course in your degree program. These professors understand that a solid understanding of research methods will enrich the qualifications of students for employment and performance in their criminal justice careers.

As previously stated, the basic goal of this book is to make students, as future and possibly even current practitioners in the criminal justice system, better informed and more capable consumers of the results of criminal justice research. This goal is based on the belief that an understanding of research methods allows criminal justice practitioners to be better able to make use of the results of research as it applies to their work-related duties. In fact, thousands of research questions are asked and answered each year in research involving criminal justice and criminological topics. In addition, thousands of articles are published, papers presented at conferences, and reports prepared that provide answers to these questions. The ability to understand research gives practitioners knowledge of the most current information in their respective fields and the ability to use this knowledge to improve the effectiveness of criminal justice agencies.

How Do We Know What We Know? Sources of Knowledge

The reality is the understanding of crime and criminal justice system operations by the public is frequently the product of misguided assumptions, distorted interpretations, outright myths, and hardened ideological positions. 6 This is a bold statement that basically contends that most people’s knowledge of crime and criminal justice is inaccurate. But, how do these inaccuracies occur? Most people have learned what they know about crime and criminal justice system operations through some other means besides scientific research results and findings. Some of that knowledge is based on personal experience and common sense. Much of it is based on the information and images supported by politicians, governmental agencies, and especially the media. This section will discuss the mechanisms used to understand crime and criminal justice operations by the public. It is important to note that although this section will focus on the failings of these knowledge sources, they each can be, and certainly are, accurate at times, and thus are valuable sources of knowledge.

Knowledge from Authority

We gain knowledge from parents, teachers, experts, and others who are in positions of authority in our lives. When we accept something as being correct and true just because someone in a position of authority says it is true, we are using what is referred to as authority knowledge as a means of knowing. Authorities often expend significant time and effort to learn something, and we can benefit from their experience and work.

However, relying on authority as a means of knowing has limitations. It is easy to overestimate the expertise of other people. A person’s expertise is typically limited to a few core areas of significant knowledge; a person is not an expert in all areas. More specifically, criminal justice professors are not experts on all topics related to criminal justice. One professor may be an expert on corrections but know little about policing. If this professor discusses topics in policing in which he is not an expert, we may still assume he is right when he may be wrong. Authority figures may speak on fields they know little about. They can be completely wrong but we may believe them because of their status as an expert. Furthermore, an expert in one area may try to use his authority in an unrelated area. Other times, we have no idea of how the experts arrived at their knowledge. We just know they are experts in the topic area.

As I am writing this, I recall an example of authority knowledge that was wrong during my police academy training in the late 1980s. My academy training was about four years after the U.S. Supreme Court decision in Tennessee v. Garner. 7 In this case, the Court limited the use of deadly force by police to defense of life situations and incidents where the suspect committed a violent offense. Prior to the decision, the police in several states could use deadly force on any fleeing suspect accused of a felony offense. One day, the academy class was practicing mock traffic stops. During one of my mock traffic stops, I received information that the vehicle I stopped was stolen. The driver and passenger exited the vehicle and fled on foot. I did not use deadly force (this was a training exercise so was not real) against the suspects and was chastised by my instructor who insisted that I should have shot the suspects as they were fleeing. Training instructors, just like professors, convey authority knowledge but, in this case, the instructor was wrong. I was not legally authorized to use deadly force in the traffic stop scenario despite the insistence of my instructor to the contrary.

Politicians are sometimes taken as a source of authority knowledge about the law, crime, and criminal justice issues. Since they enact laws that directly impact the operations of the criminal justice system, we may assume they are an authority on crime and criminal justice. More specifically, we may assume that politicians know best about how to reduce crime and increase the effectiveness of the criminal justice system. However, history is rife with laws that sounded good on paper but had no impact on crime. For example, there is little evidence that sex offender registration protects the public from sexual predators or acts as a deterrent to repeat sex offenders even though every state has a law requiring convicted sex offenders to register with local authorities. Perhaps politicians are not the criminal justice experts some perceive them to be.

History is also full of criminal justice authorities that we now see as being misinformed. For example, Cesare Lombroso is the father of the positivist school of criminology. He is most readily recognized for his idea that some individuals are born criminal. He stated that criminals have certain unique biological characteristics, including large protruding jaws, high foreheads, flattened noses, and asymmetrical faces, to name a few. 8 These characteristics were similar to those found in primitive humans. Therefore, Lombroso argued that some individuals were genetic “throwbacks” to a more primitive time and were less evolved than other people and thus, were more likely to be criminals. Lombroso’s research has been discredited because he failed to compare criminals with noncriminals. By studying only criminals, he found characteristics that were common to criminals. However, if Lombroso had studied a group of noncriminals, he would have discovered that these biological characteristics are just as prevalent among noncriminals. This example involves authority knowledge that is supported by research but the research methods used were flawed. The errors of Lombroso seem obvious now, but what do we know today through authority knowledge that is inaccurate or will be proven wrong in the future?

Knowledge from Tradition

In addition to authority knowledge, people often rely on tradition for knowledge. Tradition knowledge relies on the knowledge of the past. Individuals accept something as true because that is the way things have always been so it must be right. A good example of tradition knowledge is preventive/random patrol. Ever since vehicles were brought into the police patrol function, police administrators assumed that having patrol officers drive around randomly in the communities they serve, while they are not answering calls for service, would prevent crime. If you were a patrol officer in the early 1970s and asked your supervisor, “Why do I drive around randomly throughout my assigned area when I am not answering a call for service?” the answer would have been, “That is the way we have always done patrol and random patrol reduces crime through deterrence.” The Kansas City Preventive Patrol Experiment challenged the tradition knowledge that preventive/random patrol reduces crime. The results of the study made it clear that the traditional practice of preventive/random patrol had little to no impact on reducing crime. This allowed police departments to develop other patrol deployment strategies such as directed patrol and “hot spots” policing since preventive patrol was seen as ineffective. The development of effective patrol deployment strategies continues today.

Knowledge from Common Sense

We frequently rely on common sense knowledge for what we know about crime and the criminal justice system because it “just makes sense.” For example, it “just makes sense” that if we send juvenile delinquents on a field trip to prison where they will see first hand the prison environment as well as be yelled at by actual prisoners, they will refrain from future delinquency. That is exactly what the program Scared Straight, originally developed in the 1970s, is designed to do. Scared Straight programs are still in existence today and are even the premise for the television show Beyond Scared Straight on the A&E television network. As originally created, the program was designed to decrease juvenile delinquency by bringing at-risk and delinquent juveniles into prison where they would be “scared straight” by inmates serving life sentences. Participants in the program were talked to and yelled at by the inmates in an effort to scare them. It was believed that the fear felt by the participants would lead to a discontinuation of their delinquent behavior so that they would not end up in prison themselves. This sounds like a good idea. It makes sense, and the program was initially touted as a success due to anecdotal evidence based on a few delinquents who turned their lives around after participation in the program.

However, evaluations of the program and others like it showed that the program was in fact unsuccessful. In the initial evaluation of the Scared Straight program, Finckenauer used a classic experimental design (discussed in Chapter 5), to evaluate the original “Lifer’s Program” at Rahway State Prison in New Jersey where the program was initially developed. 13 Juveniles were randomly assigned to an experimental group that attended the Scared Straight program and a control group that did not participate in the program. Results of the evaluation were not positive. Post-test measures revealed that juveniles who were assigned to the experimental group and participated in the program were actually more seriously delinquent afterwards than those who did not participate in the program. Also using an experimental design with random assignment, Yarborough evaluated the “Juvenile Offenders Learn Truth” (JOLT) program at the State Prison of Southern Michigan at Jackson. 14 This program was similar to that of the “Lifer’s Program,” only with fewer obscenities used by inmates. Post-test measurements were taken at two intervals, three and six months after program completion. Again, results were not positive. Findings revealed no significant differences in delinquency between those juveniles who attended the program and those who did not. Other experiments conducted on Scared Straight- type programs further revealed their inability to deter juveniles from further delinquency. 15 Despite the common sense popularity of these programs, the evaluations showed that Scared Straight programs do not reduce delinquency and, in some instances, may actually increase delinquency. The programs may actually do more harm than good. I guess that begs the question, “Why do we still do these types of programs?”

Scared Straight programs and other widely held common sense beliefs about crime and the criminal justice system are questionable, based on the available research evidence. Common sense is important in our daily lives and is frequently correct, but, at times, it also contains inaccuracies, misinformation, and even prejudice.


Is It Safe to Put Felons on Probation?

Research Study 9

In the mid-1970s, the number of offenders on probation began to significantly increase. By the mid-1980s, probation was the most frequently used sentence in most states and its use was becoming more common for felons, whereas previously, probation was typically limited to misdemeanor crimes and offenses committed by juveniles. Increasing numbers of felony offenders were being placed on probation because judges had no other alternative forms of punishment. Prisons were already operating above capacity due to rising crime rates. Despite the increase in the use of probation in the 1980s, few empirical studies of probation (particularly its use with felony offenders) had been published. In the early 1980s, the Rand Corporation conducted an extensive study of probation to learn more about the offenders sentenced to probation and the effectiveness of probation as a criminal sanction. At the time the study began, over one-third of California’s probation population were convicted felons. 10 This was the first large-scale study of felony probation.

Is it safe to put felons on probation?

Data for the study were obtained from the California Board of Prison Terms (CBPT). The Board had been collecting comprehensive data on all offenders sentenced to prison since 1978 and on a sample of adult males from 17 counties who received probation. From these two data sources, researchers selected a sample of male offenders who had been convicted of the following crimes: robbery, assault, burglary, theft, forgery, and drug offenses. These crimes were selected because an offender could receive either prison or probation if convicted. Approximately 16,500 male felony offenders were included in the study. For each offender, researchers had access to their personal characteristics, information on their crimes, court proceedings, and disposition.

Two main research questions were answered in this study. First, what were the recidivism rates for felony offenders who received probation? When assessing recidivism rates, the study found that the majority of offenders sentenced to probation recidivated during the follow-up period, which averaged 31 months. Overall, 65% of the sample of probationers were re-arrested and 51 % were charged with and convicted of another offense. A total of 18% were convicted of a violent crime.

The second research question asked, what were the characteristics of the probationers who recidivated? Property offenders were more likely to recidivate compared to violent or drug offenders. Researchers also discovered that probationers tended to recidivate by committing the same crime that placed them on probation. Rand researchers included time to recidivism in their analysis and found that property and violent offenders recidivated sooner than drug offenders. The median time to the first filed charge was five months for property offenders and eight months for violent offenders.

The issue of whether or not the findings would generalize to other counties in California and to other states was raised. Data for the study came from probation and prison records from two counties in California. These two counties were not randomly selected, but were chosen because of their large probation populations and the willingness of departments to provide information. Further, the probation departments in these counties had experienced significant budget cuts. Supervision may have become compromised as a result and this could have explained why these counties had high rates of recidivism. Studies of probation recidivism in other states have found recidivism rates to be much lower, suggesting the Rand results may not have applied elsewhere. 11 Several studies examining the effectiveness of probation and the factors correlated with probation outcomes were published after 1985. Much of this research failed to produce results consistent with the Rand study.

The Rand study of felony probation received a considerable amount of attention within the field of corrections. According to one scholar, the study was acclaimed as “the most important criminological research to be reported since World War II.” 12 The National Institute of Justice disseminated the report to criminal justice agencies across the country and even highlighted the study in their monthly newsletter. Today, the study remains one of the most highly cited pieces of corrections research.

According to Rand researchers, these findings raised serious doubts about the effectiveness of probation for felony offenders. Most of the felons sentenced to probation recidivated and researchers were unable to develop an accurate prediction model to improve the courts’ decision-making. The continued use of probation as a sanction for felony offenders appeared to be putting the public at risk. However, without adequate prison space, the courts had no other alternatives besides probation when sentencing offenders.

The researchers made several recommendations to address the limitations of using probation for felony offenders. First, it was recommended that states formally acknowledge that the purpose of probation had changed. Probation was originally used as a means of furthering the goal of rehabilitation in the correctional system. As the United States moved away from that goal in the late 1960s, the expectations of probation changed. Probation was now used as a way to exercise “restrictive supervision” over more serious offenders. Second, probation departments needed to redefine the responsibilities of their probation officers. Probation officers were now expected to be surveillance officers instead of treatment personnel, which required specialized training. In addition, states needed to explore the possibility of broadening the legal authority of its probation officers by allowing them to act as law enforcement officers if necessary. Third, states were advised to adopt a formal client management system that included risk/need assessments of every client. Such a system would help establish uniform, consistent treatment of those on probation and would also help departments allocate their resources efficiently and effectively. Fourth, researchers encouraged states to develop alternative forms of community punishment that offered more public protection than regular probation, which led to the development and use of intensive supervision probation, house arrest, electronic monitoring, day reporting centers, and other intermediate punishments.

Knowledge from Personal Experience

If you personally see something or if it actually happens to you, then you are likely to accept it as true and gain knowledge from the experience. Gaining knowledge through actual experiences is known as personal experience knowledge, and it has a powerful and lasting impact on everyone. Personal experiences are essential building blocks of knowledge and of what we believe to be true. The problem with knowledge gained from personal experiences is that personal experiences can be unique and unreliable, which can distort reality and lead us to believe things that are actually false.

How can events that someone personally experienced be wrong? The events are not wrong. Instead, the knowledge gained from the experience is wrong. For example, the research consistently shows that a person’s demeanor significantly impacts the decision-making of police officers. During a traffic stop, if a person is rude, disrespectful, and uncooperative to the officer, then the driver is more likely to receive a traffic citation than a warning. That is what the research on police discretion shows. However, if a person was rude and uncooperative to a police officer during a traffic stop and was let go without a citation, the person will gain knowledge from this personal experience. The knowledge gained may include that being disrespectful during future traffic stops will get this person out of future tickets. Not likely. The event is not wrong. Instead, the knowledge gained from the experience is wrong because being disrespectful to the police usually leads to more enforcement action taken by the police, not less.

As a student in criminal justice, you have probably experienced something similar in interaction with friends, relatives, and neighbors. Your knowledge of criminal justice that you have developed in your criminal justice classes is trumped by one experience your friend, relative, or neighbor had with the criminal justice system. They believe they are right because they experienced it. However, there are four errors that occur in the knowledge gained from personal experiences: overgeneralization, selective observation, illogical reasoning, and resistance to change.

Overgeneralization happens when people conclude that what they have observed in one or a few cases is true for all cases. For example, you may see that a wealthy businesswomen in your community is acquitted of bribery and may conclude that “wealthy people, especially women, are never convicted in our criminal justice system,” which is an overgeneralization. It is common to draw conclusions about people and society from our personal interactions, but, in reality, our experiences are limited because we interact with just a small percentage of people in society.

The same is true for practitioners in the criminal justice system. Practitioners have a tendency to believe that because something was done a particular way in their agency, it is done that way in all agencies. That may not be true. Although there are certainly operational similarities across criminal justice agencies, there are also nuances that exist across the over 50,000 criminal justice agencies in the United States. Believing that just because it was that way in your agency, it must be that way in all agencies leads to overgeneralization.

Selective observation is choosing, either consciously or unconsciously, to pay attention to and remember events that support our personal preferences and beliefs. In fact, with selective observation, we will seek out evidence that confirms what we believe to be true and ignore the events that provide contradictory evidence. We are more likely to notice pieces of evidence that reinforce and support our ideology. As applied to the criminal justice system, when we are inclined to be critical of the criminal justice system, it is pretty easy to notice its every failing and ignore its successes. For example, if someone believes the police commonly use excessive force, the person is more likely to pay attention to and remember a police brutality allegation on the nightly news than a police pursuit that led to the apprehension of the suspect without incident on the same nightly news. As another example, if you believe treatment efforts on sex offenders are futile, you will pay attention to and remember each sex offender you hear about that recidivates but will pay little attention to any successes. It is easy to find instances that confirm our beliefs, but with selective observation, the complete picture is not being viewed. Therefore, if we only acknowledge the events that confirm our beliefs and ignore those that challenge them, we are falling victim to selective observation.

Besides selective observation, some of our observations may simply be wrong. Consider eyewitness identification. It is a common practice in the criminal justice system, but research has consistently demonstrated inaccuracies in eyewitness identification. The witness feels certain that the person viewed is the person who committed the offense, but sometimes the witness is wrong. Even when our senses of sight, hearing, taste, touch, and smell are fully operational, our minds have to interpret what we have sensed, which may lead to an inaccurate observation.


When Your Criminal Past Isn’t Yours 16

The business of background checks on prospective employees is increasing significantly. According to the Society for Human Resource Management, since the events of September 11, 2001, the percentage of companies that conduct criminal history checks during the hiring process has risen past 90%. Employers spend at least $2 billion a year to look into the pasts of their prospective employees. Problems with the business of background checks were identified through research that included a review of thousands of pages of court filings and interviews with dozens of court officials, data providers, lawyers, victims, and regulators.

The business of background checks is a system weakened by the conversion to digital files and compromised by the significant number of private companies that profit by amassing public records and selling them to employers. The private companies create a system in which a computer program scrapes the public files of court systems around the country to retrieve personal data. Basically, these are automated data-mining programs. Today, half the courts in the United States put criminal records on their public websites. So, the data are there for the taking, but the records that are retrieved typically are not checked for errors—errors that would be obvious to human eyes.

The errors can start with a mistake entered into the logs of a law enforcement agency or a court file. The biggest culprits, though, are companies that compile databases using public information. In some instances, their automated formulas misinterpret the information provided them. Other times, records wind up assigned to the wrong people with a common name. Furthermore, when a government agency erases a criminal conviction after a designated period of good behavior, many of the commercial databases don’t perform the updates required to purge offenses that have been removed from public record. It is clear that these errors can have substantial ramifications, including damaged reputations and loss of job opportunities.

Illogical reasoning occurs when someone jumps to premature conclusions or presents an argument that is based on invalid assumptions. Premature conclusions occur when we feel we have the answer based on a few pieces of evidence and do not need to seek additional information that may invalidate our conclusion. Think of a detective who, after examining only a few pieces of evidence, quickly narrows in on a murder suspect. It is common for a detective to assess the initial evidence and make an initial determination of who committed the murder. However, it is hoped that the detective will continue to sort through all the evidence for confirmation or rejection of his original conclusion regarding the murder suspect. Illogical reasoning by jumping to premature conclusions is common in everyday life. We look for evidence to confirm or reject our beliefs and stop when a small amount of evidence is present; we jump to conclusions. If a person states, “I know four people who have dropped out of high school, and each one of them ended up addicted to drugs, so all dropouts abuse drugs,” the person is jumping to conclusions.

Illogical reasoning also occurs when an argument, based on invalid assumptions, is presented. Let’s revisit the Scared Straight example previously discussed. Program developers assumed that brief exposure to the harsh realities of prison would deter juveniles from future delinquency. The Scared Straight program is an example of illogical reasoning. Four hours of exposure to prison life is not going to counteract years of delinquency and turn a delinquent into a nondelinquent. The program is based on a false assumption and fails to recognize the substantial risk factors present in the lives of most delinquents that must be mediated before the juvenile can live a crime-free lifestyle. A fear of prison, developed through brief exposure, is not enough to counteract the risk factors present in the lives of most delinquents. Although the Scared Straight program sounds good, it is illogical to assume that a brief experience with prison life will have a stronger impact on the decisions made by delinquents than peer support for delinquency, drug abuse, lack of education, poor parental supervision, and other factors that influence delinquency.

Resistance to change is the reluctance to change our beliefs in light of new, accurate, and valid information to the contrary. Resistance to change is common and it occurs for several reasons. First, even though our personal experience may be counter to our belief system, it is hard to admit we were wrong after we have taken a position on an issue. Even when the research evidence shows otherwise, people who work within programs may still believe they are effective. As previously stated, even though the research evidence shows otherwise, Scared Straight programs still exist and there is even a television show devoted to the program. Second, too much devotion to tradition and the argument that this is the way it has always been done inhibits change and hinders our ability to accept new directions and develop new knowledge. Third, uncritical agreement with authority inhibits change. Although authority knowledge is certainly an important means of gaining knowledge, we must critically evaluate the ideas, beliefs, and statements of those in positions of authority and be willing to challenge those statements where necessary. However, people often accept the beliefs of those in positions of authority without question, which hinders change.

Knowledge from Media Portrayals

Television shows, movies, websites, newspapers, and magazine articles are important sources of information. This is especially true for information about crime and the criminal justice system since most people have not had much contact with criminals or the criminal justice system. Instead of gaining knowledge about the criminal justice system through personal experience, most people learn about crime and the operations of the criminal justice system through media outlets. Since the primary goal of many of these media outlets is to entertain, they may not accurately reflect the reality of crime and criminal justice. Despite their inaccuracies, the media has a substantial impact on what people know about crime and the criminal justice system. Most people know what they know about crime and criminal justice through the media, and this knowledge even has an impact on criminal justice system operations.

An example of the potential impact of the media on the actual operations of the criminal justice system involves the CSI: Crime Scene Investigation television shows. The shows have been criticized for their unrealistic portrayal of the role of forensic science in solving criminal cases. Critics claim that CSI viewers accept what they see on the show as an accurate representation of how forensic science works. When summoned for jury duty, they bring with them unrealistic expectations of the forensic evidence they will see in trial. When the expected sophisticated forensic evidence is not presented in the real trial, the juror is more likely to vote to acquit the defendant. This phenomenon is known as the CSI Effect. Has the research shown that the CSI Effect exists and is impacting the criminal justice system? Most of the research shows that the CSI Effect does not exist and thus does not impact juror decision-making, but other research has shown that viewers of CSI have higher expectations related to evidence presented at trial. 17

There are several instances in which media attention on a particular topic created the idea that a major problem existed when it did not. An example is Halloween sadism. Halloween sadism is the practice of giving contaminated treats to children during trick or treating. 18 In 1985, Joel Best wrote an article entitled, “The Myth of the Halloween Sadist.” 19 His article reviewed press coverage of Halloween sadism in the leading papers in the three largest metropolitan areas ( New York Times, Los Angeles Times, and Chicago Tribune ) from 1958–1984. Although the belief in Halloween sadism is widespread, Best found few reported incidents and few reports of children being injured by Halloween sadism. Follow-ups on these reported incidents led to the conclusion that most of these reports were hoaxes. Best concluded, “I have been unable to find a substantiated report of a child being killed or seriously injured by a contaminated treat picked up in the course of trick or treating.” 20 Since 1985, Best has kept his research up to date and has come to the same conclusion. Halloween sadism is an urban legend; it is a story that is told as true, even though there is little or no evidence that the events in the story ever occurred.

Dispelling Myths: The Power of Research Methods

In the prior section, sources of knowledge were discussed along with the limitations of each. A researcher (e.g., criminologist), ideally, takes no knowledge claim for granted, but instead relies on research methods to discover the truth. In the attempt to generate new knowledge, a researcher is skeptical of knowledge that is generated by the sources discussed in the prior section, and this skepticism leads to the questioning of conventional thinking. Through this process, existing knowledge claims are discredited, modified, or substantiated. Research methods provide the researcher with the tools necessary to test current knowledge and discover new knowledge.

Although knowledge developed through research methods is by no means perfect and infallible, it is definitely a more systematic, structured, precise, and evidence-based process than the knowledge sources previously discussed. However, researchers should not dismiss all knowledge from the prior sources discussed, because, as mentioned, these sources of knowledge are sometimes accurate and certainly have their place in the development of knowledge. Researchers should guard against an elitist mind-set in which all knowledge, unless it is research-based knowledge, is dismissed.

To further discuss the importance of research methods in the development of knowledge, this section will discuss myths about crime and criminal justice. Myths are beliefs that are based on emotion rather than rigorous analysis. Take the myth of the Halloween sadist previously discussed. Many believe that there are real examples of children being harmed by razor blades, poison, or other nefarious objects placed in Halloween candy. This belief has changed the practices of many parents on Halloween; not allowing their children to trick-or-treat in their neighborhood and forbidding them from going to the doors of strangers. After careful analysis by Best, there is not a single, known example of children being seriously injured or killed by contaminated candy given by strangers. The Halloween sadist is a myth but it is still perpetuated today, and as the definition states, it is a belief based upon emotion rather than rigorous analysis. People accept myths as accurate knowledge of reality when, in fact, the knowledge is false.

The power of research is the ability to dispel myths. If someone were to assess the research literature on a myth or do their own research, she would find that the knowledge based on the myth is wrong. Perceived reality is contradicted by the facts developed through research. But that does not mean that the myth still doesn’t exist. It is important to keep in mind that the perpetuation and acceptance of myths by the public, politicians, and criminal justice personnel has contributed to the failure of criminal justice practices and policies designed to reduce crime and improve the operations of the criminal justice system. In this section, a detailed example of a myth about crime, police, courts, and corrections will be presented to demonstrate how the myth has been dispelled through research. In addition, several additional myths about crime, police, courts, and corrections will be briefly presented.

The Health Benefits of Alcohol Consumption 21

The press release from Oregon State University is titled “Beer Compound Shows Potent Promise in Prostate Cancer Battle.” The press release leads to several newspaper articles throughout the country written on the preventative nature of drinking beer on prostate cancer development with titles such as “Beer Protects Your Prostate” and “Beer May Help Men Ward Off Prostate Cancer.” By the titles alone, this sounds great; one of the main ingredients in beer appears to thwart prostate cancer.

The study that generated these headlines was conducted by a group of researchers at Oregon State University using cultured cells with purified compounds in a laboratory setting. The research showed that xanthohumol, a compound found in hops, slowed the growth of prostate cancer cells and also the growth of cells that cause enlarged prostates. But you would have to drink more than 17 pints of beer to consume a medically effective dose of xanthohumol, which is almost a case of beer. In addition, although the research is promising, further study is necessary to determine xanthohumol’s true impact on prostate cancer.

These are the types of headlines that people pay attention to and want to believe as true, even if disproven by later research. People want to believe that there are health benefits to alcohol consumption. You have probably heard about the health benefits of drinking red wine, but here is something you should consider. Recently, the University of Connecticut released a statement describing an extensive research misconduct investigation involving a member of its faculty. The investigation was sparked by an anonymous allegation of research irregularities. The comprehensive report of the investigation, which totals approximately 60,000 pages, concludes that the professor is guilty of 145 counts of fabrication and falsification of data. The professor had gained international notoriety for his research into the beneficial properties of resveratrol, which is found in red wine, especially its impact on aging. Obviously, this throws his research conclusions, that red wine has a beneficial impact on the aging process, into question.

Myths about Crime—Drug Users Are Violent

The myth of drug users as violent offenders continues to be perpetuated by media accounts of violent drug users. The public sees drug users as violent offenders who commit violent crimes to get money for drugs or who commit violent crimes while under the intoxicating properties of drugs. The public also recognizes the violent nature of the drug business with gangs and cartels using violence to protect their turf. In May 2012, extensive media attention was given to the case of the Miami man who ate the face of a homeless man for an agonizing 18 minutes until police shot and killed the suspect. The police believed that the suspect was high on the street drug known as “bath salts.” This horrific case definitely leaves the image in the public’s mind about the relationship between violence and drug use.

In recent years, media reports have focused on the relationship between methamphetamine use and violence; before then it was crack cocaine use and violence. 32 However, media portrayals regarding the violent tendencies of drug users date back to the 1930s and the release of Reefer Madness. In 1985, Goldstein suggested that drugs and violence could be related in three different ways:

1. violence could be the direct result of drug ingestion;

2. violence could be a product of the instability of drug market activity; and

3. violence could be the consequence of people having a compulsive need for drugs or money for drugs. 33

So, what does the research show? Studies have found that homicides related to crack cocaine were usually the product of the instability of drug market activity (i.e., buying and selling drugs can be a violent activity) and rarely the result of drug ingestion. 34 After an extensive review of research studies on alcohol, drugs, and violence, Parker and Auerhahn concluded, “Despite a number of published statements to the contrary, we find no significant evidence suggesting that drug use is associated with violence. There is substantial evidence to suggest that alcohol use is significantly associated with violence of all kinds.” 35 The reality is not everyone who uses drugs becomes violent and users who do become violent do not do so every time they use drugs; therefore, the relationship between violence and drug use is a myth.


Some additional myths about crime that research does not support include:

•Crime statistics accurately show what crimes are being committed and what crimes are most harmful. 22

•Most criminals—especially the dangerous ones—are mentally ill. 23

•White-collar crime is only about financial loss and does not hurt anyone. 24

•Serial murderers are middle-aged, white males. 25

•Criminals are significantly different from noncriminals. 26

•People are more likely to be a victim of violent crime committed by a stranger than by someone they know. 27

•Older adults are more likely to be victimized than people in any other age group. 28

•Sex offender registration protects the public from sexual predators. 29

•Juvenile crime rates are significantly increasing. 30

•Only the most violent juveniles are tried as adults. 31

Myths about Police—Female Police Officers Do Not Perform as Well as Males

Female police officers still face the myth that they cannot perform as well as male police officers. Throughout history, females have faced significant difficulties even becoming police officers. In the past, it was common for police agencies to require all police applicants to meet a minimum height requirement to be considered for employment. The minimum height requirement was 5′8″ for most agencies, which limited the ability of females to successfully meet the minimum standards to become a police officer. Even if women could meet the minimum height requirements, they were typically faced with a physical-abilities test that emphasized upper body strength (e.g., push-ups and bench presses). Women failed these tests more often than men, and thus were not eligible to be police officers. Minimum height requirements are no longer used in law enforcement, but the perception that female police officers are not as good as males still exists. Today, the myth that women cannot be effective police officers is based largely on the belief that the need to demonstrate superior physical strength is a daily, common occurrence in law enforcement along with the belief that police work is routinely dangerous, violent, and crime-related.

So, what does the research show? On occasion, it is useful for police officers to be able to overpower suspects by demonstrating superior physical strength, but those types of activities are rare in law enforcement. In addition, it is fairly rare for a police officer to have to deal with a dangerous and violent encounter or even an incident involving a crime. The Police Services Study conducted in the 1970s analyzed 26,418 calls for service in three metropolitan areas and found that only 19% of calls for service involve crime and only 2% of the total calls for service involve violent crime. 43 This research study was among the first to assess the types of calls for service received by police agencies.

Despite the belief that women do not make good police officers, consistent research findings show that women are extremely capable as police officers, and in some respects, outperform their male counterparts. 44 Research has demonstrated several advantages to the hiring, retention, and promotion of women in law enforcement. First, female officers are as competent as their male counterparts. Research does not show any consistent differences in how male and female patrol officers perform their duties. Second, female officers are less likely to use excessive force. Research has shown that female patrol officers are less likely to be involved in high-speed pursuits, incidents of deadly force, and the use of excessive force. Female officers are more capable at calming potentially violent situations through communication and also demonstrate heightened levels of caution. Third, female officers can help implement community-oriented policing. Studies have shown that female officers are more supportive of the community-policing philosophy than are their male counterparts. Fourth, female officers can improve law enforcement’s response to violence against women. Studies have shown that female officers are more patient and understanding in handling domestic violence calls, and female victims of domestic violence are more likely to provide positive evaluations of female officers than their male counterparts. 52


Some additional myths about the police that research does not support include:

•Police target minorities for traffic stops and arrests. 36

•Most crimes are solved through forensic science. 37

•COMPSTAT reduces crime. 38

•Intensive law enforcement efforts at the street level will lead to the control of illicit drug use and abuse. 39

•Police work primarily entails responding to crimes in progress or crimes that have just occurred. 40

•Police presence reduces crime. 41

•Detectives are most responsible for solving crimes and arresting offenders. 42

Myths about Courts—The Death Penalty Is Administered Fairly

According to a recent Gallup poll, 52% of Americans say the death penalty is applied fairly in the United States, the lowest mark in almost 40 years. 53 The issue of fairness and the death penalty typically concerns whether the punishment is equally imposed on offenders who are equally deserving based on legal factors (i.e., similar offense, similar prior criminal history, similar aggravating circumstances, and similar mitigating circumstances). 54 Unfairness can be shown if similarly situated offenders are more or less likely to receive death sentences based on age, gender, and race.

So, what does the research show? First, has research shown that a defendant’s age influences his or her chances of being sentenced to death? A study of about 5,000 homicides, controlling for legally relevant variables, found that defendants over the age of 25 were more than twice as likely to receive the death penalty in comparison to those 25 years of age or younger. 55

Second, has research shown that a defendant’s gender influences his or her chance of being sentenced to death? Capital punishment is almost exclusively reserved for male defendants. On December 31, 2010, there were 3,158 prisoners under a sentence of death in the United States: 58 were women, or 1.8%. 56 However, women account for 10–12% of all murders in the United States. 57 One research study found that male defendants were 2.6 times more likely than females to receive a death sentence after controlling for legally relevant factors. 58

Third, has research shown that a defendant’s race influences his or her chance of being sentenced to death? Most of the research on the biased nature of the death penalty has focused on racial inequities in the sentence. Although some research has shown that a defendant’s race has an impact on the likelihood of receiving a death sentence, a significant amount of research has shown that the race of the victim has the most substantial impact on death sentences. The research evidence clearly shows that offenders who murder white victims are more likely to receive a death sentence than offenders who murder black victims. 59 When assessing the race of both the victim and offender, the composition most likely to receive the death penalty is when a black offender murders a white victim. 60


Some additional myths about courts that research does not support include:

•Many criminals escape justice because of the exclusionary rule. 45

•Subjecting juvenile offenders to harsh punishments can reduce crime committed by juveniles. 46

•Public opinion is overwhelmingly in favor of imprisonment and harsh punishment for offenders. 47

•The death penalty brings closure and a sense of justice to the family and friends of murder victims. 48

•Insanity is a common verdict in criminal courts in the United States. 49

•Eyewitness identification is reliable evidence. 50

•Most people who commit crimes based on hatred, bias, or discrimination face hate crime charges and longer sentencing. 51

Myths about Corrections—Imprisonment Is the Most Severe Form of Punishment

It seems clear that besides the death penalty, the most severe punishment available in our criminal justice system is to lock up offenders in prison. On a continuum, it is perceived that sentence severity increases as one moves from fines, to probation, to intermediate sanctions such as boot camps, and finally, to incarceration in prison. The public and politicians support this perception as well.

So, what does the research show? What do criminals think is the most severe form of punishment? A growing body of research has assessed how convicted offenders perceive and experience the severity of sentences in our criminal justice system. 61 Research suggests that alternatives to incarceration in prison (i.e., probation and intermediate sanctions) are perceived by many offenders as more severe due to a greater risk of program failure (e.g., probation revocation). In comparison, serving prison time is easier. 62  

For example, one study found that about one-third of nonviolent offenders given the option of participating in an Intensive Supervision Probation (ISP) program, chose prison instead because the prospects of working every day and submitting to random drug tests was more punitive than serving time in prison. 73 Prisoners also stated that they would likely be caught violating probation conditions (i.e., high risk of program failure) and be sent to prison anyway. 74 In another research study involving survey responses from 415 inmates serving a brief prison sentence for a nonviolent crime, prison was considered the eighth most severe sanction, with only community service and probation seen as less punitive. Electronic monitoring (seventh), intensive supervision probation (sixth), halfway house (fifth), intermittent incarceration (fourth), day reporting (third), county jail (second), and boot camp (first) were all rated by inmates as more severe sanctions than prison. 75


Some additional myths about corrections that research does not support include:

•Punishing criminals reduces crime. 63

•Prisons are too lenient in their day-to-day operations (prisons as country clubs). 64

•Prisons can be self-supporting if only prisoners were forced to work. 65

•Private prisons are more cost effective than state-run prisons. 66

•Focus of community corrections is rehabilitation rather than punishment. 67

•Correctional rehabilitation does not work. 68

•Drug offenders are treated leniently by the criminal justice system. 69

•Most death row inmates will be executed eventually. 70

•If correctional sanctions are severe enough, people will think twice about committing crimes. 71

•Sexual violence against and exploitation of inmates of the same gender are primarily the result of lack of heterosexual opportunities. 72

What is Research and Why is It Important to be an Informed Consumer of Research?

We probably should have started the chapter with the question “What is research?” but we wanted to initially lay a foundation for the question with a discussion of the problems with how knowledge is developed and the power of research in discovering the truth. Research methods are tools that allow criminology and criminal justice researchers to systematically study crime and the criminal justice system. The study of research methods is the study of the basic rules, appropriate techniques, and relevant procedures for conducting research. Research methods provide the tools necessary to approach issues in criminal justice from a rigorous standpoint and challenge opinions based solely on nonscientific observations and experiences. Similarly, research is the scientific investigation of an issue, problem, or subject utilizing research methods. Research is a means of knowledge development that is designed to assist in discovering answers to research questions and leads to the creation of new questions.

How Is Knowledge Development through Research Different?

Previously, sources of knowledge development were discussed, including authority, tradition, common sense, personal experience, and media portrayals. The problems generated by each knowledge source were also discussed. Research is another source of knowledge development, but it is different than those previously discussed in several ways. First, research relies on logical and systematic methods and observations to answer questions. Researchers use systematic, well-established research practices to seek answers to their questions. The methods and observations are completed in such a way that others can inspect and assess the methods and observations and offer feedback and criticism. Researchers develop, refine, and report their understanding of crime and the criminal justice system more systematically than the public does through casual observation. Those who conduct scientific research employ much more rigorous methods to gather the information/knowledge they are seeking.

Second, in order to prove that a research finding is correct, a researcher must be able to replicate the finding using the same methods. Only through replication can we have confidence in our original finding. For researchers, it may be important to replicate findings many times over so that we are assured our original finding was not a coincidence or chance occurrence. The Minneapolis Domestic Violence Experiment is an example of this and will be discussed in detail in Chapter 5. In the experiment, the researchers found that arrests for domestic violence lead to fewer repeat incidences in comparison to separation of the people involved and mediation. Five replication studies were conducted and none were able to replicate the findings in the Minneapolis study. In fact, three of the replications found that those arrested for domestic violence had higher levels of continued domestic violence, so arrest did not have the deterrent effect found in the Minneapolis study.

Third, research is objective. Objectivity indicates a neutral and nonbiased perspective when conducting research. Although there are examples to the contrary, the researcher should not have a vested interest in what findings are discovered from the research. The researcher is expected to remain objective and report the findings of the study regardless of whether the findings support their personal opinion or agenda. In addition, research ensures objectivity by allowing others to examine and be critical of the methodology, findings, and results of research studies.

It should be clear that using research methods to answer questions about crime and the criminal justice system will greatly reduce the errors in the development of knowledge previously discussed. For example, research methods reduces the likelihood of overgeneralization by using systematic procedures for selecting individuals or groups to study that are representative of the individuals or groups that we wish to generalize. This is the topic of Chapter 3, which covers sampling procedures. In addition, research methods reduces the risk of selective observation by requiring that we measure and observe our research subjects systematically.

Being an Informed Consumer of Research

Criminal justice and criminological research is important for several reasons. First, it can provide better and more objective information. Second, it can promote better decision-making. Today, more than ever, we live in a world driven by data and in which there is an increasing dependence on the assessment of data when making decisions. As well as possible, research ensures that our decisions are based on data and not on an arbitrary or personal basis. Third, it allows for the objective assessment of programs. Fourth, it has often been the source of innovation within criminal justice agencies. Fifth, it can be directly relevant to criminal justice practice and have a significant impact on criminal justice operations.

Before we apply research results to practices in the criminal justice system, and before we even accept those research results as reasonable, we need to be able to know whether or not they are worthwhile. In other words, should we believe the results of the study? Research has its own limitations, so we need to evaluate research results and the methods used to produce them, and we do so through critical evaluation. Critical evaluation involves identifying both positive and negative aspects of the research study—both the good and the bad. Critical evaluation involves comparing the methodology used in the research with the standards established in research methods.

Through critical evaluation, consumers of research break studies down into their essential elements. What are the research questions and hypotheses? What were the independent and dependent variables? What research design was used? Was probability sampling used? What data-gathering procedures were employed? What type of data analysis was conducted and what conclusions were made? These are some of the questions that are asked by informed consumers of research. The evaluation of research ranges from the manner in which one obtains an idea to the ways in which one writes about the research results, and understanding each step in the research process is useful in our attempts to consume research conducted by others. Located between these two activities are issues concerning ethics, sampling, research design, data analyses, and interpretations.

The research design and procedures are typically the most critically evaluated aspects of research and will likewise receive the greatest amount of attention in this text. Informed consumers of research don’t just take the results of a research study at face value because the study is in an academic journal or written by someone with a Ph.D. Instead, informed consumers critically evaluate research. Taking what is learned throughout this text, critical evaluation of research is covered in Chapter 8, and upon completing this text, it is hoped that you will be an informed consumer of research and will put your research knowledge to use throughout your career.

Although many students will never undertake their own research, all will be governed by policies based upon research and exposed to research findings in their chosen professional positions. Most government agencies, including the criminal justice system, as well as private industry, routinely rely on data analysis. Criminal justice students employed with these agencies will be challenged if not prepared for quantitative tasks. Unfortunately, it is not unusual to find students as well as professionals in criminal justice who are unable to fully understand research reports and journal articles in their own field.

Beyond our criminal justice careers, we are all exposed to and use research to help us understand issues and to make personal decisions. For example, we know that cigarette smoking causes lung cancer and has other significant health impacts, so we don’t smoke. Your doctor tells you that your cholesterol is too high and you need to limit your red meat intake because research shows that consumption of red meat raises cholesterol; so, you quit eating red meat. That is why not all the examples in this text are criminal justice research examples. Some come from the medical field while others come from psychology and other disciplines. This is to remind you that you are probably exposed to much more research than you thought on day one of this class.

Overall, knowledge of research methods will allow you to more appropriately consider and consume information that is important to your career in criminal justice. It will help you better understand the process of asking and answering a question systematically and be a better consumer of the kind of information that you really need to be the best criminal justice professional you can. Once familiar with research methods, your anxiety about reviewing technical reports and research findings can be minimized. As discussed in the next section, research methods involve a process and once you understand the process, you can apply your knowledge to any research study, even those in other disciplines.

The Research Process

One of the nice things about studying research methods is it is about learning a process. Research methods can be seen as a sequential process with the first step being followed by the second step, and so on. There are certainly times when the order of the steps may be modified, but researchers typically follow the same process for each research study they complete regardless of the research topic (as depicted in Figure 2.1 in Chapter 2). Very simply, a research problem or question is identified, and a methodology is selected, developed, and implemented to answer the research question. This sequential process is one of the advantages of understanding research methods, because once you understand the process, you can apply that process to any research question that interests you. In addition, research methods are the same across disciplines. So, sampling is the same in business as it is in health education and as it is in criminal justice. Certainly the use of a particular method will be more common in one discipline in comparison to another, but the protocol for implementing the method to complete the research study is the same. For example, field research (discussed in Chapter 6) is used much more frequently in anthropology than in criminal justice. However, the research protocol to implement field research is the same whether you are studying an indigenous Indian tribe in South America in anthropology or a group of heroin users in St. Louis in criminal justice.

Some authors have presented the research process as a wheel or circle, with no specific beginning or end. Typically, the research process begins with the selection of a research problem and the development of research questions or hypotheses (discussed further in Chapter 2). It is common for the results of previous research to generate new research questions and hypotheses for the researcher. This suggests that research is cyclical, a vibrant and continuous process. When a research study answers one question, the result is often the generation of additional questions, which plunges the researcher right back into the research process to complete additional research to answer these new questions.

In this section, a brief overview of the research process will be presented. The chapters that follow address various aspects of the research process, but it is critical that you keep in mind the overall research process as you read this book, which is why is it presented here. Although you will probably not be expected to conduct a research study on your own, it is important for an educated consumer of research to understand the steps in the research process. The steps are presented in chronological order and appear neatly ordered. In practice, the researcher can go back and forth between the steps in the research process.

Step 1: Select a Topic and Conduct a Literature Review

The first step in the research process is typically the identification of a problem or topic that the researcher is interested in studying. Research topics can arise from a wide variety of sources, including the findings of a current study, a question that a criminal justice agency needs to have answered, or the result of intellectual curiosity. Once the researcher has identified a particular problem or topic, the researcher assesses the current state of the literature related to the problem or topic. The researcher will often spend a considerable amount of time in determining what the existing literature has to say about the topic. Has the topic already been studied to the point that the questions in which the researcher is interested have been sufficiently answered? If so, can the researcher approach the subject from a previously unexamined perspective? Many times, research topics have been previously explored but not brought to completion. If this is the case, it is certainly reasonable to examine the topic again. It is even appropriate to replicate a previous study to determine whether the findings reported in the prior research continue to be true in different settings with different participants. This step in the research process is also discussed in Chapter 2.

Step 2: Develop a Research Question

After a topic has been identified and a comprehensive literature review has been completed on the topic, the next step is the development of a research question or questions. The research question marks the beginning of your research study and is critical to the remaining steps in the research process. The research question determines the research plan and methodology that will be employed in the study, the data that will be collected, and the data analysis that will be performed. Basically, the remaining steps in the process are completed in order to answer the research question or questions established in this step. The development of research questions is discussed in more detail in Chapter 2.

Step 3: Develop a Hypothesis

After the research questions have been established, the next step is the formulation of hypotheses, which are statements about the expected relationship between two variables. For example, a hypothesis may state that there is no relationship between heavy metal music preference and violent delinquency. The two variables stated in the hypothesis are music preference and violent delinquency. Hypothesis development is discussed in more detail in Chapter 2.

Step 4: Operationalize Concepts

Operationalization involves the process of giving the concepts in your study a working definition and determining how each concept in your study will be measured. For example, in Step 3, the variables were music preference and violent delinquency. The process of operationalization involves determining how music preference and violent delinquency will be measured. Operationalization is further discussed in Chapter 2.

Step 5: Develop the Research Plan and Methodology

The next step is to develop the methodology that will be employed to answer the research questions and test the hypotheses. The research methodology is the blueprint for the study, which outlines how the research is to be conducted. The research questions will determine the appropriate methodology for the study. The research design selected should be driven by the research questions asked. In other words, the research questions dictate the methods used to answer them. The methodology is basically a research plan on how the research questions will be answered and will detail:

1. What group, subjects, or population will be studied and selected? Sampling will be discussed in Chapter 3.

2 . What research design will be used to collect data to answer the research questions? Various research designs will be covered in Chapters 4–7.

You need to have familiarity with all research designs so that you can become an educated consumer of research. A survey cannot answer all research questions, so knowing a lot about surveys but not other research designs will not serve you well as you assess research studies. There are several common designs used in criminal justice and criminology research. Brief descriptions of several common research designs are presented below, but each is discussed in detail in later chapters.

Survey research is one of the most common research designs employed in criminal justice research. It obtains data directly from research participants by asking them questions and is often conducted through self-administered questionnaires and personal interviews. For example, a professor might have her students complete a survey during class to understand the relationship between drug use and self-esteem. Survey research is discussed in Chapter 4.

Experimental designs are used when researchers are interested in determining whether a program, policy, practice, or intervention is effective. For example, a researcher may use an experimental design to determine if boot camps are effective at reducing juvenile delinquency. Experimental design is discussed in Chapter 5.

Field research involves researchers studying individuals or groups of individuals in their natural environment. The researcher is observing closely or acting as part of the group under study and is able to describe in depth not only the subject’s behaviors, but also consider the motivations that drive those behaviors. For example, if a researcher wanted to learn more about gangs and their activities, he may “hang out” with a gang in order to observe their behavior. Field research is discussed in Chapter 6.

A case study is an in-depth analysis of one or a few illustrative cases. This design allows the story behind an individual, a particular offender, to be told and then information from cases studies can be extrapolated to a larger group. Often these studies require the review and analysis of documents such as police reports and court records and interviews with the offender and others. For example, a researcher may explore the life history of a serial killer to try and understand why the offender killed. Case studies are discussed in Chapter 6.

Secondary data analysis occurs when researchers obtain and reanalyze data that was originally collected for a different purpose. This can include reanalyzing data collected from a prior research study, using criminal justice agency records to answer a research question, or historical research. For example, a researcher using secondary data analysis may analyze inmate files from a nearby prison to understand the relationship between custody level assignment and disciplinary violations inside prison. Secondary data analysis is discussed in Chapter 7.

Content analysis requires the assessment of content contained in mass communication outlets such as newspapers, television, magazines, and the like. In this research design, documents, publications, or presentations are reviewed and analyzed. For example, a researcher utilizing content analysis might review true crime books involving murder to see how the characteristics of the offender and victim in the true crime books match reality as depicted in the FBI’s Supplemental Homicide Reports. Content analysis is discussed in Chapter 7.

Despite the options these designs offer, other research designs are available and will be discussed later in the text. Ultimately, the design used will depend on the nature of the study and the research questions asked.

Step 6: Execute the Research Plan and Collect Data

The next step in the research process is the collection of the data based on the research design developed. For example, if a survey is developed to study the relationship between gang membership and violent delinquency, the distribution and collection of surveys from a group of high school students would occur in this step. Data collection is discussed in several chapters throughout this text.

Step 7: Analyze Data

After the data have been collected, the next phase in the research process involves analyzing the data through various and appropriate statistical techniques. The most common means for data analysis today is through the use of a computer and statistically oriented software. Data analysis and statistics are discussed in Chapter 9.

Step 8: Report Findings, Results, and Limitations

Reporting and interpreting the results of the study make up the final step in the research process. The findings and results of the study can be communicated through reports, journals, books, or computer presentations. At this step, the results are reported and the research questions are answered. In addition, an assessment is made regarding the support or lack of support for the hypotheses tested. It is also at this stage that the researcher can pose additional research questions that may now need to be answered as a result of the research study. In addition, the limitations of the study, as well as the impact those limitations may have on the results of the study, will be described by the researcher. All research has limitations, so it is incumbent on the researcher to identify those limitations for the reader. The process of assessing the quality of research will be discussed in Chapter 8.

Research in Action: Impacting Criminal Justice Operations

Research in the criminal justice system has had significant impacts on its operations. The following sections provide an example of research that has significantly impacted each of the three main components of the criminal justice system: police, courts, and corrections. The purpose of this section is to demonstrate that research has aided the positive development and progression of the criminal justice system.

Police Research Example 76

The efforts of criminal justice researchers in policing have been important and have created the initial and critical foundation necessary for the further development of effective and productive law enforcement. One seminal study asked: How important is it for the police to respond quickly when a citizen calls? The importance of rapid response was conveyed in a 1973 National Commission on Productivity Report despite the fact that there was very little empirical evidence upon which to base this assumption. In fact, the Commission stated “there is no definitive relationship between response time and deterrence, but professional judgment and logic do suggest that the two are related in a strong enough manner to make more rapid response important.” 77 Basically the Commission members were stating that we don’t have any research evidence that response times are important, but we “know” that they are. Police departments allocated substantial resources to the patrol function and deployed officers in an effort to improve response time through the use of the 9-1-1 telephone number, computer-assisted dispatch, and beat assignment systems. Officers were typically assigned to a patrol beat. When the officers were not answering calls for service, they remained in their assigned beats so they could immediately respond to an emergency.

The data for the project were collected as part of a larger experiment on preventive patrol carried out in Kansas City, Missouri, between October 1972 and September 1973. 78 To determine the impact of response time, researchers speculated that the following variables would be influenced by response time: 1) the outcome of the response, 2) citizen satisfaction with response time, and 3) citizen satisfaction with the responding officer. Several data sources were used in the study. First, surveys were completed after all citizen-initiated calls (excluding automobile accidents) that involved contact with a police officer. The survey instrument consisted of questions to assess the length of time to respond to a call and the outcome of the call (i.e., arrest). Over 1,100 surveys were completed. Second, a follow-up survey was mailed to citizens whom the police had contacted during their response. These surveys asked questions to assess citizen satisfaction with response time and outcome. Over 425 of these surveys were returned.

The data collected during the study showed that response time did not determine whether or not the police made an arrest or recovered stolen property. This was the most surprising finding from the study because it challenged one of the basic underlying principles of police patrol. Researchers attributed the lack of significance to the fact that most citizens waited before calling the police. Rapid response simply did not matter in situations where citizens delayed in reporting the crime.

Rapid response time was not only believed to be important in determining the outcome of a response (i.e., more likely to lead to an arrest), it was also considered an important predictor of citizen satisfaction. Data from the study showed that when the police arrived sooner than expected, citizens were more satisfied with response time. However, subsequent research has shown that citizens are also satisfied with a delayed response as long as the dispatcher sets a reasonable expectation for when the patrol officer will arrive. Response time was also the best predictor of how satisfied a citizen was with the responding officer. It was further revealed that citizens became dissatisfied with the police when they were not informed of the outcome (i.e., someone was arrested). Again, these findings indicate the need for dispatchers and patrol officers to communicate with complainants regarding when they should expect an officer to arrive and the outcome of the call.

Based on the results of the response time study, the researchers concluded that rapid response was not as important as police administrators had thought. Response time was not related to an officer’s ability to make an arrest or recover stolen property. Results from the response time study challenged traditional beliefs about the allocation of patrol in our communities. Based on tradition knowledge, as previously discussed, rapidly responding to calls for service is what the police had always done since they started using patrol vehicles. In addition, common sense, as previously discussed, played a role in the practice of rapid response to calls for service; it just made sense that if a patrol officer arrives sooner, she will be more likely to make an arrest.

Prior to the research, police departments operated under the assumption that rapid response was a crucial factor in the ability of an officer to solve a crime and an important predictor of citizen satisfaction. In response to the research on rapid response, many police departments changed the way they responded to calls for service. Many departments adopted a differential police response approach. Differential police response protocols allow police departments to prioritize calls and rapidly dispatch an officer only when an immediate response is needed (i.e., crimes in progress). For crimes in progress, rapid response is critical and may reduce the injuries sustained by the victim as well, but these emergency calls usually account for less than 2% of all 9-1-1 calls for police service. For nonemergency calls, an officer is either dispatched at a later time when the officer is available or a report is taken over the phone or through some other means. Differential police response has been shown to save departments money and give patrol officers more time to engage in community-oriented and proactive policing activities. The benefits for a department are not at the expense of the public. In fact, a study by Robert Worden found a high degree of citizen satisfaction with differential police response. 79

Courts Research Example 80

Research on the courts component of the criminal justice system, while far from complete, has produced direct effects on the operations of the criminal justice system. The study reviewed in this section asked the following research question: Are jurors able to understand different legal rules for establishing a defendant’s criminal responsibility? The study described below explored the issue of criminal responsibility as it applies to the insanity defense in the United States. For several years, the M ’ Naghten rule was the legal rule applied in all courts of the United States. Under M ’ Naghten, criminal responsibility was absent when the offender did not understand the nature of his actions due to failure to distinguish “right” from “wrong.” This is known as the “right/wrong test” for criminal responsibility. The case of Durham v. United States was heard in the U.S. Court of Appeals for the District of Columbia and offered an alternative test for criminal responsibility and insanity. The legal rule emerging from Durham was that criminal responsibility was absent if the offense was a product of mental disease or defect. This ruling provided psychiatrists with a more important role at trial because of the requirement that the behavior be linked to a mental disorder that only a psychiatrist could officially determine.

At the time of Simon’s 1967 study, most courts across the country still followed the M ’ Naghten rule. Questions arose, however, regarding whether juries differed in their understanding of M ’ Naghten versus Durham and, in turn, whether this resulted in differences in their ability to make informed decisions regarding criminal responsibility in cases involving the insanity defense. The study was designed to determine the effect of different legal rules on jurors’ decision-making in cases where the defense was insanity. There was a question of whether there was a difference between the rules to the extent that jurors understood each rule and could capably apply it.

Simon conducted an experimental study on jury deliberations in cases where the only defense was insanity. 81 Utilizing a mock jury approach, Simon took the transcripts of two actual trials with one reflecting the use of the M ’ Naghten rule and the other the Durham rule. Both cases were renamed and the transcripts were edited to constitute a trial of 60–90 minutes in length. These edited transcripts were then recorded, with University of Chicago Law School faculty as the attorneys, judges, and witnesses involved in each case. Groups of 12 jurors listened to each trial with instruction provided at the end regarding the particular rule of law ( M ’ Naghten or Durham) for determining criminal responsibility. Each juror submitted a written statement with his or her initial decision on the case before jury deliberations, and the juries’ final decisions after deliberation were also reported.

Simon found significant differences in the verdicts across the two groups ( M ’ Naghten rule applied and Durham rule applied) even when the case was the same. For the M ’ Naghten version of the case, the psychiatrists stated that the defendant was mentally ill yet knew right from wrong during the crime. These statements/instructions should have led to a guilty verdict on the part of the mock jury. As expected, the M ’ Naghten juries delivered guilty verdicts in 19 of the 20 trials, with one hung jury. For the Durham version of the case, the psychiatrists stated that the crime resulted from the defendant’s mental illness, which should have lead to acquittal. However, the defendant was acquitted in only five of the 26 Durham trials. Twenty-six groups of 12 jurors were exposed to the Durham version of the trial and the case was the same each time. Simon interpreted these results as suggesting that jurors were unambiguous in their interpretations and applications of M ’ Naghten (due to the consistency in guilty verdicts), but they were less clear on the elements of Durham and how to apply it (reflected by the mix of guilty, not guilty, and hung verdicts). 82

After Simon’s study, most states rejected the Durham test. Recall her finding that the Durham rule produced inconsistent verdicts. She interpreted this finding as Durham being no better than providing no guidance to jurors on how to decide the issue of insanity. The observation helped to fuel arguments against the use of Durham, which, in turn, contributed to its demise as a legal rule. Today, only New Hampshire uses a version of the Durham rule in insanity cases.


The Punishment Cost of Being Young, Black, and Male

Steffensmeier, Ulmer, and Kramer 83 hypothesized that African Americans overall were not likely to be treated more harshly than white defendants by the courts because it was only particular subgroups of minority defendants that fit with court actors’ stereotypes of “more dangerous” offenders. In particular, they argued that younger African American males not only fulfilled this stereotype more than any other age, race, and gender combination, they were also more likely to be perceived by judges as being able to handle incarceration better than other subgroups.

In order to test their hypotheses, the researchers examined sentencing data from Pennsylvania spanning four years (1989–1992). Almost 139,000 cases were examined. The sentences they examined included whether a convicted defendant was incarcerated in prison or jail, and the length of incarceration in prison or jail. The researchers found that offense severity and prior record were the most important predictors of whether a convicted defendant was incarcerated and the length of incarceration. The authors found that the highest likelihood of incarceration and the longest sentences for males were distributed to African Americans aged 18–29 years. Their analysis of females revealed that white females were much less likely than African American females to be incarcerated, regardless of the age group examined. Taken altogether, the analysis revealed that African American males aged 18–29 years maintained the highest odds of incarceration and the longest sentences relative to any other race, sex, and age group.

Overall, this research showed that judges focused primarily on legal factors (offense severity and prior record) when determining the sentences of convicted offenders. These are the factors we expect judges to consider when making sentencing decisions. However, the research also found that judges base their decisions in part on extralegal factors, particularly the interaction of a defendant’s age, race, and gender. This research expanded our knowledge beyond the impact of singular factors on sentencing to expose the interaction effects of several variables (race, gender, and age). Court personnel are aware of these interaction effects based on this study, and others that followed, as well as their personal experiences in the criminal justice system. Identification and recognition of inequities in our justice system (in this case that young, African American males are punished more severely in our justice system) is the first step in mitigating this inequity.

Corrections Research Example 84

Although the research in corrections is far from complete, it has contributed greatly to the development of innovative programs and the professional development of correctional personnel. The contributions of academic and policy-oriented research can be seen across the whole range of correctional functions from pretrial services through probation, institutional corrections, and parole.

Rehabilitation remained the goal of our correctional system until the early 1970s, when the efficacy of rehabilitation was questioned. Violent crime was on the rise, and many politicians placed the blame on the criminal justice system. Some believed the system was too lenient on offenders. Interest in researching the effectiveness of correctional treatment remained low until 1974 when an article written by Robert Martinson and published in Public Interest titled “What Works? Questions and Answers about Prison Reform” generated enormous political and public attention to the effectiveness of correctional treatment. 85

Over a six-month period, Martinson and his colleagues reviewed all of the existing literature on correctional treatment published in English from 1945 to 1967. Each of the articles was evaluated according to traditional standards of social science research. Only studies that utilized an experimental design, included a sufficient sample size, and could be replicated were selected for review. A total of 231 studies examining a variety of different types of treatment were chosen, including educational and vocational training, individual and group counseling, therapeutic milieus, medical treatment, differences in length and type of incarceration, and community corrections. All of the treatment studies included at least one measure of offender recidivism, such as whether or not offenders were rearrested or violated their parole. The recidivism measures were used to examine the success or failure of a program in terms of reducing crime.

After reviewing all 231 studies, Martinson reported that there was no consistent evidence that correctional treatment reduced recidivism. Specifically, he wrote, “with few and isolated exceptions, the rehabilitative efforts that have been reported so far have had no appreciable effect on recidivism.” 86 Martinson further indicated that the lack of empirical support for correctional treatment could be a consequence of poorly implemented programs. If the quality of the programs were improved, the results may have proved more favorable, but this conclusion was for the most part ignored by the media and policy-makers.

Martinson’s report became commonly referred to as “nothing works” and was subsequently used as the definitive study detailing the failures of rehabilitation. The article had implications beyond questioning whether or not specific types of correctional treatment reduced recidivism. The entire philosophy of rehabilitation was now in doubt because of Martinson’s conclusion that “our present strategies … cannot overcome, or even appreciably reduce, the powerful tendencies of offenders to continue in criminal behavior.” 87

Martinson’s article provided policy makers the evidence to justify spending cuts on rehabilitative programs. Furthermore, it allowed politicians to respond to growing concerns about crime with punitive, get-tough strategies. States began implementing strict mandatory sentences that resulted in more criminals being sent to prison and for longer periods of time. Over the next several years, Martinson’s article was used over and over to support abandoning efforts to treat offenders until rehabilitation became virtually nonexistent in our correctional system.

Chapter Summary

This chapter began with a discussion of sources of knowledge development and the problems with each. To depict the importance of research methods in knowledge development, myths about crime and the criminal justice system were reviewed along with research studies that have dispelled myths. As the introductory chapter in this text, this chapter also provided an overview of the steps in the research process from selecting a topic and conducting a literature review at the beginning of a research study to reporting findings, results, and limitations at the end of the study. Examples of actual research studies in the areas of police, courts, and corrections were also provided in this chapter to demonstrate the research process in action and to illustrate how research has significantly impacted practices within the criminal justice system. In addition, this chapter demonstrated the critical importance of becoming an informed consumer of research in both your personal and professional lives.

Critical Thinking Questions

1. What are the primary sources of knowledge development, and what are the problems with each?

2. How is knowledge developed through research methods different from other sources of knowledge?

3. What myths about crime and criminal justice have been dispelled through research? Give an example of a research study that dispelled a myth.

4. Why is it important to be an informed consumer of research?

5. What are the steps in the research process, and what activities occur at each step?

authority knowledge: Knowledge developed when we accept something as being correct and true just because someone in a position of authority says it is true

case study: An in-depth analysis of one or a few illustrative cases

common sense knowledge: Knowledge developed when the information “just makes sense”

content analysis: A method requiring the analyzing of content contained in mass communication outlets such as newspapers, television, magazines, and the like

CSI Effect: Due to the unrealistic portrayal of the role of forensic science in solving criminal cases in television shows, jurors are more likely to vote to acquit a defendant when the expected sophisticated forensic evidence is not presented

differential police response: Methods that allow police departments to prioritize calls and rapidly dispatch an officer only when an immediate response is needed (i.e., crimes in progress)

experimental designs: Used when researchers are interested in determining whether a program, policy, practice, or intervention is effective

field research: Research that involves researchers studying individuals or groups of individuals in their natural environment

Halloween sadism: The practice of giving contaminated treats to children during trick or treating

hypotheses: Statements about the expected relationship between two concepts

illogical reasoning: Occurs when someone jumps to premature conclusions or presents an argument that is based on invalid assumptions

myths: Beliefs that are based on emotion rather than rigorous analysis

operationalization: The process of giving a concept a working definition; determining how each concept in your study will be measured

overgeneralization: Occurs when people conclude that what they have observed in one or a few cases is true for all cases

personal experience knowledge: Knowledge developed through actual experiences

research: The scientific investigation of an issue, problem, or subject utilizing research methods

research methods: The tools that allow criminology and criminal justice researchers to systematically study crime and the criminal justice system and include the basic rules, appropriate techniques, and relevant procedures for conducting research

resistance to change: The reluctance to change our beliefs in light of new, accurate, and valid information to the contrary

secondary data analysis: Occurs when researchers obtain and reanalyze data that were originally collected for a different purpose

selective observation: Choosing, either consciously or unconsciously, to pay attention to and remember events that support our personal preferences and beliefs

survey research: Obtaining data directly from research participants by asking them questions, often conducted through self-administered questionnaires and personal interviews

tradition knowledge: Knowledge developed when we accept something as true because that is the way things have always been, so it must be right

variables: Concepts that have been given a working definition and can take on different values

1 Briggs, Lisa T., Stephen E. Brown, Robert B. Gardner, and Robert L. Davidson. (2009). “D.RA.MA: An extended conceptualization of student anxiety in criminal justice research methods courses.” Journal of Criminal Justice Education 20 (3), 217–226.

2 Betz, N. E. (1978). “Prevalence, distribution, and correlates of math anxiety in college students. Journal of Counseling Psychology 25 (5), 441–448.

3 Briggs, et al., 2009, p. 221.

4 Ibid, p. 221.

5 Ibid, p. 221.

6 Kappeler, Victor E., and Gary W. Potter. (2005). The mythology of crime and criminal justice. Prospect Heights, IL: Waveland.

7 Tennessee v. Gamer, 471 U.S. 1 (1985).

8 Lombroso-Ferrero, Gina. (1911). Criminal man, according to the classification of Cesare Lombroso. New York: Putnam.

9 This study was included in Amy B. Thistlethwaite and John D. Wooldredge. (2010). Forty studies that changed criminal justice: Explorations into the history of criminal justice research. Upper Saddle River, NJ: Prentice Hall.

10 Petersilia, J., S. Turner, J. Kahan, and J. Peterson. (1985). Granting felons probation: Public risks and alternatives. Santa Monica, CA: Rand.

11 Vito, G. (1986). “Felony probation and recidivism: Replication and response.” Federal Probation 50, 17–25.

12 Conrad, J. (1985). “Research and development in corrections.” Federal Probation 49, 69–71.

13 Finckenauer, James O. (1982). Scared straight! and the panacea phenomenon. Englewood Cliffs, NJ: Prentice Hall.

14 Yarborough, J.C. (1979). Evaluation of JOLT (Juvenile Offenders Learn Truth) as a deterrence program. Lansing, MI: Michigan Department of Corrections.

15 Petrosino, Anthony, Carolyn Turpin-Petrosino, and James O. Finckenauer. (2000). “Well-meaning programs can have harmful effects! Lessons from experiments of programs such as Scared Straight,” Crime & Delinquency 46, 354–379.

16 Robertson, Jordan. “I’m being punished for living right”: Background check system is haunted by errors. December 20, 2011. /ap-impact-criminal-past-isnt-182335059.html. Retrieved on December 29, 2011.

17 Shelton, D. E. (2008). “The ‘CSI Effect’: Does it really exist?” NIJ Journal 259 [NCJ 221501].

18 Best, Joel. (2011). “Halloween sadism: The evidence.” 19716/726/Halloween%20sadism.revised%20thru%20201l.pdf?sequence=6. Retrieved on May 7, 2012.

19 Best, Joel. (1985, November). “The myth of the Halloween sadist. Psychology Today 19 (11), p. 14.

21 “Beer compound shows potent promise in prostate cancer battle.” Press release from Oregon State University May 30, 2006. may/beer-compound-shows-potent-promise-prostate-cancer-battle. Retrieved on January 6, 2012; Colgate, Emily C., Cristobal L. Miranda, Jan F. Stevens, Tammy M. Bray, and Emily Ho. (2007). “Xanthohumol, a prenylflavonoid derived from hops induces apoptosis and inhibits NF-kappaB activation in prostate epithelial cells,” Cancer Letters 246, 201–209; “Health benefits of red wine exaggerated” /nutrition/health-benefits-red-wine-exaggerated. Retrieved on January 14, 2012; “Scientific journals notified following research misconduct investigation.” January 11, 2012. -notified-following-research-misconduct-investigation/. Retrieved on January 14, 2012.

22 Pepinsky, Hal. “The myth that crime and criminality can be measured.” 3–11 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

23 Bullock, Jennifer L., and Bruce A. Arrigo. “The myth that mental illness causes crime.” 12–19 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

24 Friedrichs, David O. “The myth that white-collar crime is only about financial loss.” 20–28 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

25 Kuhns III, Joseph B., and Charisse T. M. Coston. “The myth that serial murderers are disproportionately white males.” 37–44 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

26 Longmire, Dennis R., Jacqueline Buffington-Vollum, and Scott Vollum. “The myth of positive differentiation in the classification of dangerous offenders.” 123–131 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

27 Masters, Ruth E., Lori Beth Way, Phyllis B. Gerstenfeld, Bernadette T. Muscat, Michael Hooper, John P. J. Dussich, Lester Pincu, and Candice A. Skrapec. (2013). CJ realities and challenges, 2nd ed. New York: McGraw-Hill.

32 Brownstein, Henry H. “The myth of drug users as violent offenders.” 45–53 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

33 Goldstein, P. (1985). “The drugs/violence nexus: A tripartite conceptual framework.” Journal of Drug Issues 15, 493–506.

34 Goldstein, P, H. Brownstein, and P. Ryan. (1992). “Drug-related homicide in New York City: 1984 and 1988.” Crime & Delinquency 38, 459–476.

35 Parker, R., and K. Auerhahn. (1998). “Alcohol, drugs, and violence.” Annual Review of Sociology 24, 291–311, p. 291.

36 Buerger, Michael. “The myth of racial profiling.” 97–103 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

37 Cordner, Gary, and Kathryn E. Scarborough. “The myth that science solves crimes.” 104–110 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

38 Willis, James J., Stephen D. Mastrofski, and David Weisburd. “The myth that COMPSTAT reduces crime and transforms police organizations.” 111–119 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

39 Masters, et al., 2013.

43 Scott, Eric J. (1981). Calls for service: Citizen demand and initial police response. Washington, DC: Government Printing Office.

44 Lersch, Kim. “The myth of policewomen on patrol.” 89–96 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

45 Janikowski, Richard. “The myth that the exclusionary rule allows many criminals to escape justice.” 132–139 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

46 Bishop, Donna M. “The myth that harsh punishments reduce juvenile crime.” 140–148 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

47 Immarigeon, Russ. “The myth that public attitudes are punitive.” 149–157 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

48 Acker, James R. “The myth of closure and capital punishment.” 167–175 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

49 Masters, et al., 2013.

52 Lersch, 2006.

53 Newport, Frank. “In U.S., support for death penalty falls to 39-year low.” October 13, 2011. .com/poll/150089/support-death-penalty-falls-year-low.aspx. Retrieved on April 16, 2012.

54 Applegate, Brandon. “The myth that the death penalty is administered fairly.” 158–166 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

55 Williams, M. R., and J. E. Holcomb. (2001). “Racial disparity and death sentences in Ohio.” Journal of Criminal Justice 29, 207–218.

56 Snell, Tracy L. (2011, December). Capital punishment, 2010—statistical tables. Washington, DC: Bureau of Justice Statistics.

57 Applegate, 2006.

58 Williams and Holcomb, 2001.

59 Applegate, 2006.

61 Wood, Peter B. “The myth that imprisonment is the most severe form of punishment.” 192–200 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

63 Michalowski, Raymond. “The myth that punishment reduces crime.” 179–191 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

64 McShane, Marilyn, Frank P. Williams III, and Beth Pelz. “The myth of prisons as country clubs.” 201–208 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

65 Parker, Mary. “The myth that prisons can be self-supporting.” 209–213 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

66 Blakely, Curtis, and John Ortiz Smykla. “Correctional privatization and the myth of inherent efficiency.” 214–220 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

67 Jones, G. Mark. “The myth that the focus of community corrections is rehabilitation.” 221–226 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

68 Cullen, Francis T., and Paula Smith. “The myth that correctional rehabilitation does not work.” 227–238 in Bohm, Robert M., and Jeffrey T. Walker. (2006). Demystifying crime and criminal justice. Los Angeles: Roxbury.

69 Masters, et al., 2013.

73 Petersilia, Joan. (1990). “When probation becomes more dreaded than prison. Federal Probation 54, 23–27.

75 Wood, P. B., and H. G. Grasmick. (1999). “Toward the development of punishment equivalencies: Male and female inmates rate the severity of alternative sanctions compared to prison.” Justice Quarterly 16, 19–50.

76 Example is excerpted from Amy B. Thistlethwaite and John D. Wooldredge. (2010). Forty studies that changed criminal justice: Explorations into the history of criminal justice research. Upper Saddle River, NJ: Prentice Hall. This is an excellent book that demonstrates the impact research has had on criminal justice operations.

77 National Commission on Productivity. (1973). Opportunities for improving productivity in police services. Washington, DC: United States Government Printing Office, p. 19.

78 Pate, T., A. Ferrara, R. Bowers, and J. Lorence. (1976). Police response time: Its determinants and effects. Washington, DC: Police Foundation.

79 Worden, R. (1993). “Toward equity and efficiency in law enforcement: Differential police response. American Journal of Police 12, 1–32.

80 Example is excerpted from Amy B. Thistlethwaite and John D. Wooldredge. (2010). Forty studies that changed criminal justice: Explorations into the history of criminal justice research. Upper Saddle River, NJ: Prentice Hall.

81 Simon, R. (1967). The jury and the defense of insanity. Boston: Little, Brown.

83 Steffensmeier, D., J. Ulmer, & J. Kramer. (1998). “The interaction of race, gender, and age in criminal sentencing: The punishment cost of being young, black, and male. Criminology 36, 763–797.

84 Example is excerpted from Amy B. Thistlethwaite and John D. Wooldredge. (2010). Forty studies that changed criminal justice: Explorations into the history of criminal justice research. Upper Saddle River, NJ: Prentice Hall.

85 Martinson, R. (1974). “What works? Questions and answers about prison reform.” The Public Interest 10, 22–54.

86 Ibid, p. 25.

87 Ibid, p. 49.

Applied Research Methods in Criminal Justice and Criminology by University of North Texas is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License , except where otherwise noted.

Share This Book

2.1 Why Is Research Important?

Learning objectives.

By the end of this section, you will be able to:

  • Explain how scientific research addresses questions about behavior
  • Discuss how scientific research guides public policy
  • Appreciate how scientific research can be important in making personal decisions

Scientific research is a critical tool for successfully navigating our complex world. Without it, we would be forced to rely solely on intuition, other people’s authority, and blind luck. While many of us feel confident in our abilities to decipher and interact with the world around us, history is filled with examples of how very wrong we can be when we fail to recognize the need for evidence in supporting claims. At various times in history, we would have been certain that the sun revolved around a flat earth, that the earth’s continents did not move, and that mental illness was caused by possession ( Figure 2.2 ). It is through systematic scientific research that we divest ourselves of our preconceived notions and superstitions and gain an objective understanding of ourselves and our world.

The goal of all scientists is to better understand the world around them. Psychologists focus their attention on understanding behavior, as well as the cognitive (mental) and physiological (body) processes that underlie behavior. In contrast to other methods that people use to understand the behavior of others, such as intuition and personal experience, the hallmark of scientific research is that there is evidence to support a claim. Scientific knowledge is empirical : It is grounded in objective, tangible evidence that can be observed time and time again, regardless of who is observing.

While behavior is observable, the mind is not. If someone is crying, we can see behavior. However, the reason for the behavior is more difficult to determine. Is the person crying due to being sad, in pain, or happy? Sometimes we can learn the reason for someone’s behavior by simply asking a question, like “Why are you crying?” However, there are situations in which an individual is either uncomfortable or unwilling to answer the question honestly, or is incapable of answering. For example, infants would not be able to explain why they are crying. In such circumstances, the psychologist must be creative in finding ways to better understand behavior. This chapter explores how scientific knowledge is generated, and how important that knowledge is in forming decisions in our personal lives and in the public domain.

Use of Research Information

Trying to determine which theories are and are not accepted by the scientific community can be difficult, especially in an area of research as broad as psychology. More than ever before, we have an incredible amount of information at our fingertips, and a simple internet search on any given research topic might result in a number of contradictory studies. In these cases, we are witnessing the scientific community going through the process of reaching a consensus, and it could be quite some time before a consensus emerges. For example, the explosion in our use of technology has led researchers to question whether this ultimately helps or hinders us. The use and implementation of technology in educational settings has become widespread over the last few decades. Researchers are coming to different conclusions regarding the use of technology. To illustrate this point, a study investigating a smartphone app targeting surgery residents (graduate students in surgery training) found that the use of this app can increase student engagement and raise test scores (Shaw & Tan, 2015). Conversely, another study found that the use of technology in undergraduate student populations had negative impacts on sleep, communication, and time management skills (Massimini & Peterson, 2009). Until sufficient amounts of research have been conducted, there will be no clear consensus on the effects that technology has on a student's acquisition of knowledge, study skills, and mental health.

In the meantime, we should strive to think critically about the information we encounter by exercising a degree of healthy skepticism. When someone makes a claim, we should examine the claim from a number of different perspectives: what is the expertise of the person making the claim, what might they gain if the claim is valid, does the claim seem justified given the evidence, and what do other researchers think of the claim? This is especially important when we consider how much information in advertising campaigns and on the internet claims to be based on “scientific evidence” when in actuality it is a belief or perspective of just a few individuals trying to sell a product or draw attention to their perspectives.

We should be informed consumers of the information made available to us because decisions based on this information have significant consequences. One such consequence can be seen in politics and public policy. Imagine that you have been elected as the governor of your state. One of your responsibilities is to manage the state budget and determine how to best spend your constituents’ tax dollars. As the new governor, you need to decide whether to continue funding early intervention programs. These programs are designed to help children who come from low-income backgrounds, have special needs, or face other disadvantages. These programs may involve providing a wide variety of services to maximize the children's development and position them for optimal levels of success in school and later in life (Blann, 2005). While such programs sound appealing, you would want to be sure that they also proved effective before investing additional money in these programs. Fortunately, psychologists and other scientists have conducted vast amounts of research on such programs and, in general, the programs are found to be effective (Neil & Christensen, 2009; Peters-Scheffer, Didden, Korzilius, & Sturmey, 2011). While not all programs are equally effective, and the short-term effects of many such programs are more pronounced, there is reason to believe that many of these programs produce long-term benefits for participants (Barnett, 2011). If you are committed to being a good steward of taxpayer money, you would want to look at research. Which programs are most effective? What characteristics of these programs make them effective? Which programs promote the best outcomes? After examining the research, you would be best equipped to make decisions about which programs to fund.

Link to Learning

Watch this video about early childhood program effectiveness to learn how scientists evaluate effectiveness and how best to invest money into programs that are most effective.

Ultimately, it is not just politicians who can benefit from using research in guiding their decisions. We all might look to research from time to time when making decisions in our lives. Imagine that your sister, Maria, expresses concern about her two-year-old child, Umberto. Umberto does not speak as much or as clearly as the other children in his daycare or others in the family. Umberto's pediatrician undertakes some screening and recommends an evaluation by a speech pathologist, but does not refer Maria to any other specialists. Maria is concerned that Umberto's speech delays are signs of a developmental disorder, but Umberto's pediatrician does not; she sees indications of differences in Umberto's jaw and facial muscles. Hearing this, you do some internet searches, but you are overwhelmed by the breadth of information and the wide array of sources. You see blog posts, top-ten lists, advertisements from healthcare providers, and recommendations from several advocacy organizations. Why are there so many sites? Which are based in research, and which are not?

In the end, research is what makes the difference between facts and opinions. Facts are observable realities, and opinions are personal judgments, conclusions, or attitudes that may or may not be accurate. In the scientific community, facts can be established only using evidence collected through empirical research.


Psychological research has a long history involving important figures from diverse backgrounds. While the introductory chapter discussed several researchers who made significant contributions to the discipline, there are many more individuals who deserve attention in considering how psychology has advanced as a science through their work ( Figure 2.3 ). For instance, Margaret Floy Washburn (1871–1939) was the first woman to earn a PhD in psychology. Her research focused on animal behavior and cognition (Margaret Floy Washburn, PhD, n.d.). Mary Whiton Calkins (1863–1930) was a preeminent first-generation American psychologist who opposed the behaviorist movement, conducted significant research into memory, and established one of the earliest experimental psychology labs in the United States (Mary Whiton Calkins, n.d.).

Francis Sumner (1895–1954) was the first African American to receive a PhD in psychology in 1920. His dissertation focused on issues related to psychoanalysis. Sumner also had research interests in racial bias and educational justice. Sumner was one of the founders of Howard University’s department of psychology, and because of his accomplishments, he is sometimes referred to as the “Father of Black Psychology.” Thirteen years later, Inez Beverly Prosser (1895–1934) became the first African American woman to receive a PhD in psychology. Prosser’s research highlighted issues related to education in segregated versus integrated schools, and ultimately, her work was very influential in the hallmark Brown v. Board of Education Supreme Court ruling that segregation of public schools was unconstitutional (Ethnicity and Health in America Series: Featured Psychologists, n.d.).

Although the establishment of psychology’s scientific roots occurred first in Europe and the United States, it did not take much time until researchers from around the world began to establish their own laboratories and research programs. For example, some of the first experimental psychology laboratories in South America were founded by Horatio Piñero (1869–1919) at two institutions in Buenos Aires, Argentina (Godoy & Brussino, 2010). In India, Gunamudian David Boaz (1908–1965) and Narendra Nath Sen Gupta (1889–1944) established the first independent departments of psychology at the University of Madras and the University of Calcutta, respectively. These developments provided an opportunity for Indian researchers to make important contributions to the field (Gunamudian David Boaz, n.d.; Narendra Nath Sen Gupta, n.d.).

When the American Psychological Association (APA) was first founded in 1892, all of the members were White males (Women and Minorities in Psychology, n.d.). However, by 1905, Mary Whiton Calkins was elected as the first female president of the APA, and by 1946, nearly one-quarter of American psychologists were female. Psychology became a popular degree option for students enrolled in the nation’s historically Black higher education institutions, increasing the number of Black Americans who went on to become psychologists. Given demographic shifts occurring in the United States and increased access to higher educational opportunities among historically underrepresented populations, there is reason to hope that the diversity of the field will increasingly match the larger population, and that the research contributions made by the psychologists of the future will better serve people of all backgrounds (Women and Minorities in Psychology, n.d.).

The Process of Scientific Research

Scientific knowledge is advanced through a process known as the scientific method . Basically, ideas (in the form of theories and hypotheses) are tested against the real world (in the form of empirical observations), and those empirical observations lead to more ideas that are tested against the real world, and so on. In this sense, the scientific process is circular. The types of reasoning within the circle are called deductive and inductive. In deductive reasoning , ideas are tested in the real world; in inductive reasoning , real-world observations lead to new ideas ( Figure 2.4 ). These processes are inseparable, like inhaling and exhaling, but different research approaches place different emphasis on the deductive and inductive aspects.

In the scientific context, deductive reasoning begins with a generalization—one hypothesis—that is then used to reach logical conclusions about the real world. If the hypothesis is correct, then the logical conclusions reached through deductive reasoning should also be correct. A deductive reasoning argument might go something like this: All living things require energy to survive (this would be your hypothesis). Ducks are living things. Therefore, ducks require energy to survive (logical conclusion). In this example, the hypothesis is correct; therefore, the conclusion is correct as well. Sometimes, however, an incorrect hypothesis may lead to a logical but incorrect conclusion. Consider this argument: all ducks are born with the ability to see. Quackers is a duck. Therefore, Quackers was born with the ability to see. Scientists use deductive reasoning to empirically test their hypotheses. Returning to the example of the ducks, researchers might design a study to test the hypothesis that if all living things require energy to survive, then ducks will be found to require energy to survive.

Deductive reasoning starts with a generalization that is tested against real-world observations; however, inductive reasoning moves in the opposite direction. Inductive reasoning uses empirical observations to construct broad generalizations. Unlike deductive reasoning, conclusions drawn from inductive reasoning may or may not be correct, regardless of the observations on which they are based. For instance, you may notice that your favorite fruits—apples, bananas, and oranges—all grow on trees; therefore, you assume that all fruit must grow on trees. This would be an example of inductive reasoning, and, clearly, the existence of strawberries, blueberries, and kiwi demonstrate that this generalization is not correct despite it being based on a number of direct observations. Scientists use inductive reasoning to formulate theories, which in turn generate hypotheses that are tested with deductive reasoning. In the end, science involves both deductive and inductive processes.

For example, case studies, which you will read about in the next section, are heavily weighted on the side of empirical observations. Thus, case studies are closely associated with inductive processes as researchers gather massive amounts of observations and seek interesting patterns (new ideas) in the data. Experimental research, on the other hand, puts great emphasis on deductive reasoning.

We’ve stated that theories and hypotheses are ideas, but what sort of ideas are they, exactly? A theory is a well-developed set of ideas that propose an explanation for observed phenomena. Theories are repeatedly checked against the world, but they tend to be too complex to be tested all at once; instead, researchers create hypotheses to test specific aspects of a theory.

A hypothesis is a testable prediction about how the world will behave if our idea is correct, and it is often worded as an if-then statement (e.g., if I study all night, I will get a passing grade on the test). The hypothesis is extremely important because it bridges the gap between the realm of ideas and the real world. As specific hypotheses are tested, theories are modified and refined to reflect and incorporate the result of these tests Figure 2.5 .

To see how this process works, let’s consider a specific theory and a hypothesis that might be generated from that theory. As you’ll learn in a later chapter, the James-Lange theory of emotion asserts that emotional experience relies on the physiological arousal associated with the emotional state. If you walked out of your home and discovered a very aggressive snake waiting on your doorstep, your heart would begin to race and your stomach churn. According to the James-Lange theory, these physiological changes would result in your feeling of fear. A hypothesis that could be derived from this theory might be that a person who is unaware of the physiological arousal that the sight of the snake elicits will not feel fear.

A scientific hypothesis is also falsifiable , or capable of being shown to be incorrect. Recall from the introductory chapter that Sigmund Freud had lots of interesting ideas to explain various human behaviors ( Figure 2.6 ). However, a major criticism of Freud’s theories is that many of his ideas are not falsifiable; for example, it is impossible to imagine empirical observations that would disprove the existence of the id, the ego, and the superego—the three elements of personality described in Freud’s theories. Despite this, Freud’s theories are widely taught in introductory psychology texts because of their historical significance for personality psychology and psychotherapy, and these remain the root of all modern forms of therapy.

In contrast, the James-Lange theory does generate falsifiable hypotheses, such as the one described above. Some individuals who suffer significant injuries to their spinal columns are unable to feel the bodily changes that often accompany emotional experiences. Therefore, we could test the hypothesis by determining how emotional experiences differ between individuals who have the ability to detect these changes in their physiological arousal and those who do not. In fact, this research has been conducted and while the emotional experiences of people deprived of an awareness of their physiological arousal may be less intense, they still experience emotion (Chwalisz, Diener, & Gallagher, 1988).

Scientific research’s dependence on falsifiability allows for great confidence in the information that it produces. Typically, by the time information is accepted by the scientific community, it has been tested repeatedly.

As an Amazon Associate we earn from qualifying purchases.

This book may not be used in the training of large language models or otherwise be ingested into large language models or generative AI offerings without OpenStax's permission.

Want to cite, share, or modify this book? This book uses the Creative Commons Attribution License and you must attribute OpenStax.

Access for free at
  • Authors: Rose M. Spielman, William J. Jenkins, Marilyn D. Lovett
  • Publisher/website: OpenStax
  • Book title: Psychology 2e
  • Publication date: Apr 22, 2020
  • Location: Houston, Texas
  • Book URL:
  • Section URL:

© Jan 6, 2024 OpenStax. Textbook content produced by OpenStax is licensed under a Creative Commons Attribution License . The OpenStax name, OpenStax logo, OpenStax book covers, OpenStax CNX name, and OpenStax CNX logo are not subject to the Creative Commons license and may not be reproduced without the prior and express written consent of Rice University.

The BMJ logo

When is the evidence too old?

A few weeks ago, when submitting an abstract to a nursing conference, I was suddenly faced with a dilemma about age. Not my own age, but the age of evidence I was using to support my work. One key element of the submission criteria was to provide five research citations to support the abstract, and all citations were to be less than ten years old.  This requirement left me stumped for a while. The research I wanted to cite was more than ten years old, yet it was excellent research within a very small body of work on the topic. Suddenly I struggled to meet the criteria and almost gave up on the submission, thinking my abstract would not tick all of the boxes if I used research now deemed to be ‘out of date’. I suddenly thought about all of the work I had published more than ten years ago – all that hard work past its use-by date.

Way back in the mid 1990s, a colleague and I started to have conversations with Australian nurses about the importance of evidence based practice (EBP) for the future of Australian nursing.  The movement away from the comfort of ‘ritual and routine’ to the uncertainty of EBP was challenging. At the time we described EBP according to the principle that “all interventions should be based on the best currently available scientific evidence” 1 . We had embraced the ideas of authors such as Ian Chalmers 2 and were keen to educate nurses and nursing students about “practices that had been clearly shown to work and question practices for which no evidence exists and discard those which have been shown to do harm” 1 It was very much about the importance of using the most ‘robust’ and ‘reliable’ evidence that we had available to guide us in clinical decision making, taking into account individual patients at the centre of care. It was also about teaching nurses and nursing students about how to ask the right questions, where to look for answers and how to recognize when you have found the right answer to support individualized patient care.

Definitions of evidence-based practice are quite varied and I have heard nurses talk about using “current best evidence” while others use the “most current evidence”. These are quite different approaches, with the latter statement suggesting that more recent is best. This is sometimes reinforced in nursing education, where students are graded according to the use of recent research, with limitations placed on the age of resources used to support their work. However, I wonder if we are losing something in this translation about the meaning of ‘best evidence’ to support care. When does the published evidence get too old and where do we draw the line and stop reading research from our past?

Personally I have always expected my students to use up to date research when supporting their recommendations for care. However, I have also encouraged them to look back to see where the new research has come from and to acknowledge the foundation it has been built on.  I am always keen to hear about the latest developments in healthcare and work to support the readers of EBN who need and want to know about what is new and important in the health care literature. Keeping up to date with new evidence is critically important for change. But I wonder how we strike a balance between absorbing recent research and taking into account robust research that preceded its publication by more than a decade?

So, let’s think about these ideas for a minute. If we put our blinkers on and ignore important research from the recently ‘outdated’ literature from the 1990s (when I first became interested in doing research), we could miss some important foundational work that still influences practice today. The two references I have used below, both from the 1990s, would not be included in the discussion at all. If we only consider literature that is recent, and value that more highly than if it is robust, then we will be missing important evidence to inform practice. Researchers could start asking the same research questions over and over (I have seen some of this already in nursing literature) and even feel pressured to repeat previous studies all over again to check if the findings still hold true in the contemporary world. Perhaps that is something to watch for in the future.

It is important to keep up to date with current research findings, new innovations in care, recent trends in patient problems, trends in patient outcomes and changes in the social, political and system context of the care we provide. But it is also important to look back as we move forward, thinking about the strength of the evidence as well as its age.

Allison Shorten RN RM PhD

Yale University School of Nursing


  • Shorten A. & Wallace MC. ‘Evidence-based practice – The future is clear’. Australian Nurses Journal, 1996, Vol. 4, No. 6, pp. 22-24.
  • Chalmers I. The Cochrane collaboration: Preparing, maintaining, and disseminating systematic reviews of the effects of health care, Annals New York Academy of Science, 1993, Vol. 703, pp. 156-165.

Comment and Opinion | Open Debate

The views and opinions expressed on this site are solely those of the original authors. They do not necessarily represent the views of BMJ and should not be used to replace medical advice. Please see our full website terms and conditions .

All BMJ blog posts are posted under a CC-BY-NC licence

BMJ Journals

 view all topics  > Science and Decision Making

Based on Science

For stronger evidence, look for multiple studies

why is it important for research to be recent

  • Silje Marie Svartefoss   ORCID: 1   nAff4 ,
  • Jens Jungblut 2 ,
  • Dag W. Aksnes 1 ,
  • Kristoffer Kolltveit 2 &
  • Thed van Leeuwen 3  

5 Altmetric

In this article, we study the motivation and performance of researchers. More specifically, we investigate what motivates researchers across different research fields and countries and how this motivation influences their research performance. The basis for our study is a large-N survey of economists, cardiologists, and physicists in Denmark, Norway, Sweden, the Netherlands, and the UK. The analysis shows that researchers are primarily motivated by scientific curiosity and practical application and less so by career considerations. There are limited differences across fields and countries, suggesting that the mix of motivational aspects has a common academic core less influenced by disciplinary standards or different national environments. Linking motivational factors to research performance, through bibliometric data on publication productivity and citation impact, our data show that those driven by practical application aspects of motivation have a higher probability for high productivity. Being driven by career considerations also increases productivity but only to a certain extent before it starts having a detrimental effect.

Similar content being viewed by others

why is it important for research to be recent

Are the strategic research agendas of researchers in the social sciences determinants of research productivity?

Research incentives and research output.

why is it important for research to be recent

Exploring the determinants of research performance for early-career researchers: a literature review

Avoid common mistakes on your manuscript.


Motivation and abilities are known to be as important factors in explaining employees’ job performance of employees (Van Iddekinge et al. 2018 ), and in the vast scientific literature on motivation, it is common to differentiate between intrinsic and extrinsic motivation factors (Ryan and Deci 2000 ). In this context, path-breaking individuals are said to often be intrinsically motivated (Jindal-Snape and Snape 2006 ; Thomas and Nedeva 2012 ; Vallerand et al. 1992 ), and it has been found that the importance of these of types of motivations differs across occupations and career stages (Duarte and Lopes 2018 ).

In this article, we address the issue of motivation for one specific occupation, namely: researchers working at universities. Specifically, we investigate what motivates researchers across fields and countries (RQ1) and how this motivation is linked to their research performance (RQ2). The question of why people are motivated to do their jobs is interesting to address in an academic context, where work is usually harder to control, and individuals tend to have a lot of much freedom in structuring their work. Moreover, there have been indications that academics possess an especially high level of motivation for their tasks that is not driven by a search for external rewards but by an intrinsic satisfaction from academic work (Evans and Meyer 2003 ; Leslie 2002 ). At the same time, elements of researchers’ performance are measurable through indicators of their publication activity: their productivity through the number of outputs they produce and the impact of their research through the number of citations their publications receive (Aksnes and Sivertsen 2019 ; Wilsdon et al. 2015 ).

Elevating research performance is high on the agenda of many research organisations (Hazelkorn 2015 ). How such performance may be linked to individuals’ motivational aspects has received little attention. Thus, a better understanding of this interrelation may be relevant for developing institutional strategies to foster environments that promote high-quality research and research productivity.

Previous qualitative research has shown that scientists are mainly intrinsically motivated (Jindal-Snape and Snape 2006 ). Other survey-based contributions suggest that there can be differences in motivations across disciplines (Atta-Owusu and Fitjar 2021 ; Lam 2011 ). Furthermore, the performance of individual scientists has been shown to be highly skewed in terms of publication productivity and citation rates (Larivière et al. 2010 ; Ruiz-Castillo and Costas 2014 ). There is a large body of literature explaining these differences. Some focus on national and institutional funding schemes (Hammarfelt and de Rijcke 2015 ; Melguizo and Strober 2007 ) and others on the research environment, such as the presence of research groups and international collaboration (Jeong et al. 2014 ), while many studies address the role of academic rank, age, and gender (see e.g. Baccini et al. 2014 ; Rørstad and Aksnes 2015 ). Until recently, less emphasis has been placed on the impact of researchers’ motivation. Some studies have found that different types of motivations drive high levels of research performance (see e.g. Horodnic and Zaiţ 2015 ; Ryan and Berbegal-Mirabent 2016 ). However, researchers are only starting to understand how this internal drive relates to research performance.

While some of the prior research on the impact of motivation depends on self-reported research performance evaluations (Ryan 2014 ), the present article combines survey responses with actual bibliometric data. To investigate variation in research motivation across scientific fields and countries, we draw on a large-N survey of economists, cardiologists, and physicists in Denmark, Norway, Sweden, the Netherlands, and the UK. To investigate how this motivation is linked to their research performance, we map the survey respondents’ publication and citation data from the Web of Science (WoS).

This article is organised as follows. First, we present relevant literature on research performance and motivation. Next, the scientific fields and countries are then presented before elaborating on our methodology. In the empirical analysis, we investigate variations in motivation across fields, gender, age, and academic position and then relate motivation to publications and citations as our two measures of research performance. In the concluding section, we discuss our findings and implications for national decision-makers and individual researchers.

Motivation and research performance

As noted above, the concepts of intrinsic and extrinsic motivation play an important role in the literature on motivation and performance. Here, intrinsic motivation refers to doing something for its inherent satisfaction rather than for some separable consequence. Extrinsic motivation refers to doing something because it leads to a separable outcome (Ryan and Deci 2000 ).

Some studies have found that scientists are mainly intrinsically motivated (Jindal-Snape and Snape 2006 ; Lounsbury et al. 2012 ). Research interests, curiosity, and a desire to contribute to new knowledge are examples of such motivational factors. Intrinsic motives have also been shown to be crucial when people select research as a career choice (Roach and Sauermann 2010 ). Nevertheless, scientists are also motivated by extrinsic factors. Several European countries have adopted performance-based research funding systems (Zacharewicz et al. 2019 ). In these systems, researchers do not receive direct financial bonuses when they publish, although such practices may occur at local levels (Stephan et al. 2017 ). Therefore, extrinsic motivation for such researchers may include salary increases, peer recognitions, promotion, or expanded access to research resources (Lam 2011 ). According to Tien and Blackburn ( 1996 ), both types of motivations operate simultaneously, and their importance vary and may depend on the individual’s circumstances, personal situation, and values.

The extent to which different kinds of motivations play a role in scientists’ performance has been investigated in several studies. In these studies, bibliometric indicators based on the number of publications are typically used as outcome measures. Such indicators play a critical role in various contexts in the research system (Wilsdon et al. 2015 ), although it has also been pointed out that individuals can have different motivations to publish (Hangel and Schmidt-Pfister 2017 ).

Based on a survey of Romanian economics and business administration academics combined with bibliometric data, Horodnic and Zait ( 2015 ) found that intrinsic motivation was positively correlated with research productivity, while extrinsic motivation was negatively correlated. Their interpretations of the results are that researchers motivated by scientific interest are more productive, while researchers motivated by extrinsic forces will shift their focus to more financially profitable activities. Similarly, based on the observation that professors continue to publish even after they have been promoted to full professor, Finkelstein ( 1984 ) concluded that intrinsic rather than extrinsic motivational factors have a decisive role regarding the productivity of academics.

Drawing on a survey of 405 research scientists working in biological, chemical, and biomedical research departments in UK universities, Ryan ( 2014 ) found that (self-reported) variations in research performance can be explained by instrumental motivation based on financial incentives and internal motivation based on the individual’s view of themselves (traits, competencies, and values). In the study, instrumental motivation was found to have a negative impact on research performance: As the desire for financial rewards increase, the level of research performance decreases. In other words, researchers mainly motivated by money will be less productive and effective in their research. Contrarily, internal motivation was found to have a positive impact on research performance. This was explained by highlighting that researchers motivated by their self-concept set internal standards that become a reference point that reinforces perceptions of competency in their environments.

Nevertheless, it has also been argued that intrinsic and extrinsic motivations for publishing are intertwined (Ma 2019 ). According to Tien and Blackburn ( 1996 ), research productivity is neither purely intrinsically nor purely extrinsically motivated. Publication activity is often a result of research, which may be intrinsically motivated or motivated by extrinsic factors such as a wish for promotion, where the number of publications is often a part of the assessment (Cruz-Castro and Sanz-Menendez 2021 ; Tien 2000 , 2008 ).

The negative relationship between external/instrumental motivation and performance and the positive relationship between internal/self-concept motivation and performance are underlined by Ryan and Berbegal-Mirabent ( 2016 ). Drawing on a fuzzy set qualitative comparative analysis of a random sampling of 300 of the original respondents from Ryan ( 2014 ), they find that scientists working towards the standards and values they identify with, combined with a lack of concern for instrumental rewards, contribute to higher levels of research performance.

Based on the above, this article will address two research questions concerning different forms of motivation and the relationship between motivation and research performance.

How does the motivation of researchers vary across fields and countries?

How do different types of motivations affect research performance?

In this study, the roles of three different motivational factors are analysed. These are scientific curiosity, practical and societal applications, and career progress. The study aims to assess the role of these specific motivational factors and not the intrinsic-extrinsic distinction more generally. Of the three factors, scientific curiosity most strongly relates to intrinsic motivation; practical and societal applications also entail strong intrinsic aspects. On the other hand, career progress is linked to extrinsic motivation.

In addition to variation in researchers’ motivations by field and country, we consider differences in relation to age, position and gender. Additionally, when investigating how motivation relates to scientific performance we control for the influence of age, gender, country and funding. These are dimensions where differences might be found in motivational factors given that scientific performance, particularly publication productivity, has been shown to differ along these dimensions (Rørstad and Aksnes 2015 ).

Research context: three fields, five countries

To address the research question about potential differences across fields and countries, the study is based on a sample consisting of researchers in three different fields (cardiology, economics, and physics) and five countries (Denmark, Norway, Sweden, the Netherlands, and the UK). Below, we describe this research context in greater detail.

The fields represent three different domains of science: medicine, social sciences, and the natural sciences, where different motivational factors may be at play. This means that the fields cover three main areas of scientific investigations: the understanding of the world, the functioning of the human body, and societies and their functions. The societal role and mission of the fields also differ. While a primary aim of cardiology research and practice is to reduce the burden of cardiovascular disease, physics research may drive technology advancements, which impacts society. Economics research may contribute to more effective use of limited resources and the management of people, businesses, markets, and governments. In addition, the fields also differ in publication patterns (Piro et al. 2013 ). The average number of publications per researcher is generally higher in cardiology and physics than in economics (Piro et al. 2013 ). Moreover, cardiologists and physicists mainly publish in international scientific journals (Moed 2005 ; Van Leeuwen 2013 ). In economics, researchers also tend to publish books, chapters, and articles in national languages, in addition to international journal articles (Aksnes and Sivertsen 2019 ; van Leeuwen et al. 2016 ).

We sampled the countries with a twofold aim. On the one hand, we wanted to have countries that are comparable so that differences in the development of the science systems, working conditions, or funding availability would not be too large. On the other hand, we also wanted to assure variation among the countries regarding these relevant framework conditions to ensure that our findings are not driven by a specific contextual condition.

The five countries in the study are all located in the northwestern part of Europe, with science systems that are foremost funded by block grant funding from the national governments (unlike, for example, the US, where research grants by national funding agencies are the most important funding mechanism) (Lepori et al. 2023 ).

In all five countries, the missions of the universities are composed of a blend of education, research, and outreach. Furthermore, the science systems in Norway, Denmark, Sweden, and the Netherlands have a relatively strong orientation towards the Anglo-Saxon world in the sense that publishing in the national language still exists, but publishing in English in internationally oriented journals in which English is the language of publications is the norm (Kulczycki et al. 2018 ). These framework conditions ensure that those working in the five countries have somewhat similar missions to fulfil in their professions while also belonging to a common mainly Anglophone science system.

However, in Norway, Denmark, Sweden, and the Netherlands, research findings in some social sciences, law, and the humanities are still oriented on publishing in various languages. Hence, we avoided selecting the humanities field for this study due to a potential issue with cross-country comparability (Sivertsen 2019 ; Sivertsen and Van Leeuwen 2014 ; Van Leeuwen 2013 ).

Finally, the chosen countries vary regarding their level of university autonomy. When combining the scores for organisational, financial, staffing, and academic autonomy presented in the latest University Autonomy in Europe Scorecard presented by the European University Association (EUA), the UK, the Netherlands, and Denmark have higher levels of autonomy compared to Norway and Sweden, with Swedish universities having less autonomy than their Norwegian counterparts (Pruvot et al. 2023 ). This variation is relevant for our study, as it ensures that our findings are not driven by response from a higher education system with especially high or low autonomy, which can influence the motivation and satisfaction of academics working in it (Daumiller et al. 2020 ).

Data and methods

The data used in this article are a combination of survey data and bibliometric data retrieved from the WoS. The WoS database was chosen for this study due to its comprehensive coverage of research literature across all disciplines, encompassing the three specific research areas under analysis. Additionally, the WoS database is well-suited for bibliometric analyses, offering citation counts essential for this study.

Two approaches were used to identify the sample for the survey. Initially, a bibliometric analysis of the WoS using journal categories (‘Cardiac & cardiovascular systems’, ‘Economics’, and ‘Physics’) enabled the identification of key institutions with a minimum number of publications within these journal categories. Following this, relevant organisational units and researchers within these units were identified through available information on the units’ webpages. Included were employees in relevant academic positions (tenured academic personnel, post-docs, and researchers, but not PhD students, adjunct positions, guest researchers, or administrative and technical personnel).

Second, based on the WoS data, people were added to this initial sample if they had a minimum number of publications within the field and belonged to any of the selected institutions, regardless of unit affiliation. For economics, the minimum was five publications within the selected period (2011–2016). For cardiology and physics, where the individual publication productivity is higher, the minimum was 10 publications within the same period. The selection of the minimum publication criteria was based on an analysis of publication outputs in these fields between 2011 and 2016. The thresholds were applied to include individuals who are more actively engaged in research while excluding those with more peripheral involvement. The higher thresholds for cardiology and physics reflect the greater frequency of publications (and co-authorship) observed in these fields.

The benefit of this dual-approach strategy to sampling is that we obtain a more comprehensive sample: the full scope of researchers within a unit and the full scope of researchers that publish within the relevant fields. Overall, 59% of the sample were identified through staff lists and 41% through the second step involving WoS data.

The survey data were collected through an online questionnaire first sent out in October 2017 and closed in December 2018. In this period, several reminders were sent to increase the response rate. Overall, the survey had a response rate of 26.1% ( N  = 2,587 replies). There were only minor variations in response rates between scientific fields; the variations were larger between countries. Tables  1 and 2 provide an overview of the response rate by country and field.

Operationalisation of motivation

Motivation was measured by a question in the survey asking respondents what motivates or inspires them to conduct research, of which three dimensions are analysed in the present paper. The two first answer categories were related to intrinsic motivation (‘Curiosity/scientific discovery/understanding the world’ and ‘Application/practical aims/creating a better society’). The third answer category was more related to extrinsic motivation (‘Progress in my career [e.g. tenure/permanent position, higher salary, more interesting/independent work]’). Appendix Table A1 displays the distribution of respondents and the mean value and standard deviation for each item.

These three different aspects of motivation do not measure the same phenomenon but seem to capture different aspects of motivation (see Pearson’s correlation coefficients in Appendix Table A2 ). There is no correlation between curiosity/scientific discovery, career progress, and practical application. However, there is a weak but significant positive correlation between career progress and practical application. These findings indicate that those motivated by career considerations to some degrees also are motivated by practical application.

In addition to investigating how researchers’ motivation varies by field and country, we consider the differences in relation to age, position and gender as well. Field of science differentiates between economics, cardiology, physics, and other fields. The country variables differentiate between the five countries. Age is a nine-category variable. The position variable differentiates between full professors, associate professors, and assistant professors. The gender variable has two categories (male or female). For descriptive statistics on these additional variables, see Appendix Table A3 .

Publication productivity and citation impact

To analyse the respondents’ bibliometric performance, the Centre for Science and Technology Studies (CWTS) in-house WoS database was used. We identified the publication output of each respondent during 2011–2017 (limited to regular articles, reviews, and letters). For 16% of the respondents, no publications were identified in the database. These individuals had apparently not published in international journals covered by the database. However, in some cases, the lack of publications may be due to identification problems (e.g. change of names). Therefore, we decided not to include the latter respondents in the analysis.

Two main performance measures were calculated: publication productivity and citation impact. As an indicator of productivity, we counted the number of publications for each individual (as author or co-author) during the period. To analyse the citation impact, a composite measure using three different indicators was used: total number of citations (total citations counts for all articles they have contributed to during the period, counting citations up to and including 2017), normalised citation score (MNCS), and proportion of publications among the 10% most cited articles in their fields (Waltman and Schreiber 2013 ). Here, the MNCS is an indicator for which the citation count of each article is normalised by subject, article type, and year, where 1.00 corresponds to the world average (Waltman et al. 2011 ). Based on these data, averages for the total publication output of each respondent were calculated. By using three different indicators, we can avoid biases or limitations attached to each of them. For example, using the MNCS, a respondent with only one publication would appear as a high impact researcher if this article was highly cited. However, when considering the additional indicator, total citation counts, this individual would usually perform less well.

The bibliometric scores were skewedly distributed among the respondents. Rather than using the absolute numbers, in this paper, we have classified the respondents into three groups according to their scores on the indicators. Here, we have used percentile rank classes (tertiles). Percentile statistics are increasingly applied in bibliometrics (Bornmann et al. 2013 ; Waltman and Schreiber 2013 ) due to the presence of outliers and long tails, which characterise both productivity and citation distributions.

As the fields analysed have different publication patterns, the respondents within each field were ranked according to their scores on the indicators, and their percentile rank was determined. For the productivity measure, this means that there are three groups that are equal in terms of number of individuals included: 1: Low productivity (the group with the lowest publication numbers, 0–33 percentile), 2: Medium productivity (33–67 percentile), and 3: High productivity (67–100 percentile). For the citation impact measure, we conducted a similar percentile analysis for each of the three composite indicators. Then everyone was assigned to one of the three percentile groups based on their average score: 1: Low citation impact (the group with lowest citation impact, 0–33 percentile), 2: Medium citation impact (33–67 percentile), and 3: High citation impact (67–100 percentile), cf. Table  3 . Although it might be argued that the application of tertile groups rather than absolute numbers leads to a loss of information, the advantage is that the results are not influenced by extreme values and may be easier to interpret.

Via this approach, we can analyse the two important dimensions of the respondents’ performance. However, it should be noted that the WoS database does not cover the publication output of the fields equally. Generally, physics and cardiology are very well covered, while the coverage of economics is somewhat lower due to different publication practices (Aksnes and Sivertsen 2019 ). This problem is accounted for in our study by ranking the respondents in each field separately, as described above. In addition, not all respondents may have been active researchers during the entire 2011–2017 period, which we have not adjusted for. Despite these limitations, the analysis provides interesting information on the bibliometric performance of the respondents at an aggregated level.

Regression analysis

To analyse the relationship between motivation and performance, we apply multinomial logistic regression rather then ordered logistic regression because we assume that the odds for respondents belonging in each category of the dependent variables are not equal (Hilbe 2017 ). The implication of this choice of model is that the model tests the probability of respondents being in one category compared to another (Hilbe 2017 ). This means that a reference or baseline category must be selected for each of the dependent variables (productivity and citation impact). Furthermore, the coefficient estimates show how the probability of being in one of the other categories decreases or increases compared to being in the reference category.

For this analysis, we selected the medium performers as the reference or baseline category for both our dependent variables. This enables us to evaluate how the independent variables affect the probability of being in the low performers group compared to the medium performers and the high performers compared to the medium performers.

To evaluate model fit, we started with a baseline model where only types of motivations were included as independent variables. Subsequently, the additional variables were introduced into the model, and based on measures for model fit (Pseudo R 2 , -2LL, and Akaike Information Criterion (AIC)), we concluded that the model with all additional variables included provides the best fit to the data for both the dependent variables (see Appendix Tables A5 and A6 ). Additional control variables include age, gender, country, and funding. We include these variables as controls to obtain robust effects of motivation and not effects driven by other underlying factors. The type of funding was measured by variables where the respondent answered the following question: ‘How has your research been funded the last five years?’ The funding variable initially consisted of four categories: ‘No source’, ‘Minor source’, ‘Moderate source’, and ‘Major source’. In this analysis, we have combined ‘No source’ and ‘Minor source’ into one category (0) and ‘Moderate source’ and ‘Major source’ into another category (1). Descriptive statistics for the funding variables are available in Appendix Table A4 . We do not control for the influence of field due to how the scientific performance variables are operationalised, the field normalisation implies that there are no variations across fields. We also do not control for position, as this variable is highly correlated with age, and we are therefore unable to include these two variables in the same model.

The motivation of researchers

In the empirical analysis, we first investigate variation in motivation and then relate it to publications and citations as our two measures of research performance.

As Fig.  1 shows, the respondents are mainly driven by curiosity and the wish to make scientific discoveries. This is by far the most important motivation. Practical application is also an important source of motivation, while making career progress is not identified as being very important.

figure 1

Motivation of researchers– percentage

As Table  4 shows, at the level of fields, there are no large differences, and the motivational profiles are relatively similar. However, physicists tend to view practical application as somewhat less important than cardiologists and economists. Moreover, career progress is emphasised most by economists. Furthermore, as table 5 shows, there are some differences in motivation between countries. For curiosity/scientific discovery and practical application, the variations across countries are minor, but researchers in Denmark tend to view career progress as somewhat more important than researchers in the other countries.

Furthermore, as table 6 shows, women seem to view practical application and career progress as a more important motivation than men; these differences are also significant. Similar gender disparities have also been reported in a previous study (Zhang et al. 2021 ).

There are also some differences in motivation across the additional variables worth mentioning, as Table  7 shows. Unsurprisingly, perhaps, there is a significant moderate negative correlation between age, position, and career progress. This means that the importance of career progress as a motivation seems to decrease with increased age or a move up the position hierarchy.

In the second part of the analysis, we relate motivation to research performance. We first investigate publications and productivity using the percentile groups. Here, we present the results we use using predicted probabilities because they are more easily interpretable than coefficient estimates. For the model with productivity percentile groups as the dependent variable, the estimates for career progress were negative when comparing the medium productivity group to the high productivity group and the medium productivity group to the low productivity group. This result indicates that the probability of being in the high and low productivity groups decreases compared to the medium productivity group as the value of career progress increases, which may point towards a curvilinear relationship between the variables. A similar pattern was also found in the model with the citation impact group as the dependent variable, although it was not as apparent.

As a result of this apparent curvilinear relationship, we included quadric terms for career progress in both models, and these were significant. Likelihood ratio tests also show that the models with quadric terms included have a significant better fit to the data. Furthermore, the AIC was also lower for these models compared to the initial models where quadric terms were not included (see Appendix Tables A5 – A7 ). Consequently, we base our results on these models, which can be found in Appendix Table A7 . Due to a low number of respondents in the low categories of the scientific curiosity/discovery variable, we also combined the first three values into one to include it as a variable in the regression analysis, which results in a reduced three-value variable for scientific curiosity/discovery.

Results– productivity percentile group

Using the productivity percentile group as the dependent variable, we find that the motivational aspects of practical application and career progress have a significant effect on the probability of being in the low, medium, or high productivity group but not curiosity/scientific discovery. In Figs.  2 and 3 , each line represents the probability of being in each group across the scale of each motivational aspect.

figure 2

Predicted probability for being in each of the productivity groups according to the value on the ‘practical application’ variable

figure 3

Predicted probability of being in the low and high productivity groups according to the value on the ‘progress in my career’ variable

Figure  2 shows that at low values of application, there are no significant differences between the probability of being in either of the groups. However, from around value 3 of application, the differences between the probability of being in each group increases, and these are also significant. As a result, we concluded that high scores on practical application is related to increased probability of being in the high productivity group.

In Fig.  3 , we excluded the medium productivity group from the figure because there are no significant differences between this group and the high and low productivity group. Nevertheless, we found significant differences between the low productivity and the high productivity group. Since we added a quadric term for career progress, the two lines in Fig.  3 have a curvilinear shape. Figure  3 shows that there are only significant differences between the probability of being in the low or high productivity group at mid and high values of career progress. In addition, the probability of being in the high productivity group is at its highest value at mid values of career progress. This indicates that being motivated by career progress increases the probability of being in the high productivity group but only up to a certain point before it begins to have a negative effect on the probability of being in this group.

We also included age and gender as variables in the model, and Figs.  4 and 5 show the results. Figure  4 shows that age especially impacts the probability of being in the high productivity and low productivity groups. The lowest age category (< 30–34 years) has the highest probability for being in the low productivity group, while from the mid age category (50 years and above), the probability is highest for being in the high productivity group. This means that increased age is related to an increased probability of high productivity. The variable controlling for the effect of funding also showed some significant results (see Appendix Table A7 ). The most relevant finding is that receiving competitive grants from external public sources had a very strong and significant positive effect on being in the high productivity group and a medium-sized significant negative effect on being in the low productivity group. This shows that receiving external funding in the form of competitive grants has a strong effect on productivity.

figure 4

Predicted probability of being in each of the productivity groups according to age

Figure  5 shows that there is a difference between male and female respondents. For females, there are no differences in the probability of being in either of the groups, while males have a higher probability of being in the high productivity group compared to the medium and low productivity groups.

figure 5

Results– citation impact group

For the citation impact group as the dependent variable, we found that career progress has a significant effect on the probability of being in the low citation impact group or the high citation group but not curiosity/scientific discovery or practical application. Figure  6 shows how the probability of being in the high citation impact group increases as the value on career progress increases and is higher than that of being in the low citation impact group, but only up to a certain point. This indicates that career progress increases the probability of being in the high citation impact group to some degree but that too high values are not beneficial for high citation impact. However, it should also be noted that the effect of career progress is weak and that it is difficult to conclude on how very low or very high values of career progress affect the probability of being in the two groups.

figure 6

Predicted probability for being in each of the citation impact groups according to the value on the ‘progress in my career’ variable

We also included age and gender as variables in the model, and we found a similar pattern as in the model with productivity percentile group as the dependent variable. However, the relationship between the variables is weaker in this model with the citation impact group as the dependent variable. Figure  7 shows that the probability of being in the high citation impact group increases with age, but there is no significant difference between the probability of being in the high citation impact group and the medium citation impact group. We only see significant differences when each of these groups is compared to the low citation impact group. In addition, the increase in probability is more moderate in this model.

figure 7

Predicted probability of being in each of the citation impact groups according to age

Figure  8 shows that there are differences between male and female respondents. Male respondents have a significant higher probability of being in the medium or high citation impact group compared to the low citation impact group, but there is no significant difference in the probability between the high and medium citation impact groups. For female respondents, there are no significant differences. Similarly, for age, the effect also seems to be more moderate in this model compared to the model with productivity percentile groups as the dependent variable. In addition, the effect of funding sources is more moderate on citation impact compared to productivity (see Appendix Table A7 ). Competitive grants from external public sources still have the most relevant effect, but the effect size and level of significance is lower than for the model where productivity groups are the dependent variable. Respondents who received a large amount of external funding through competitive grants are more likely to be highly cited, but the effect size is much smaller, and the result is only significant at p  < 0.1. Those who do not receive much funding from this source are more likely to be in the low impact group. Here, the effect size is large, and the coefficient is highly significant.

figure 8

Predicted probability for being in each of the citation impact groups according to gender

Concluding discussion

This article aimed to explore researchers’ motivations and investigate the impact of motivation on research performance. By addressing these issues across several fields and countries, we provided new evidence on the motivation and performance of researchers.

Most researchers in our large-N survey found curiosity/scientific discovery to be a crucial motivational factor, with practical application being the second most supported aspect. Only a smaller number of respondents saw career progress as an important inspiration to conduct their research. This supports the notion that researchers are mainly motivated by core aspects of academic work such as curiosity, discoveries, and practical application of their knowledge and less so by personal gains (see Evans and Meyer 2003 ). Therefore, our results align with earlier research on motivation. In their interview study of scientists working at a government research institute in the UK, Jindal-Snape and Snape ( 2006 ) found that the scientists were typically motivated by the ability to conduct high quality, curiosity-driven research and de-motivated by the lack of feedback from management, difficulty in collaborating with colleagues, and constant review and change. Salaries, incentive schemes, and prospects for promotion were not considered a motivator for most scientists. Kivistö and colleagues ( 2017 ) also observed similar patterns in more recent survey data from Finnish academics.

As noted in the introduction, the issue of motivation has often been analysed in the literature using the intrinsic-extrinsic distinction. In our study, we have not applied these concepts directly. However, it is clear that the curiosity/scientific discovery item should be considered a type of intrinsic motivation, as it involves performing the activity for its inherent satisfaction. Moreover, the practical application item should probably be considered mainly intrinsic, as it involves creating a better society (for others) without primarily focusing on gains for oneself. The career progress item explicitly mentions personal gains such as position and higher salary and is, therefore, a type of extrinsic motivation. This means that our results support the notion that there are very strong elements of intrinsic motivation among researchers (Jindal-Snape and Snape 2006 ).

When analysing the three aspects of motivation, we found some differences. Physicists tend to view practical application as less important than researchers in the two other fields, while career progress was most emphasised by economists. Regarding country differences, our data suggest that career progress is most important for researchers in Denmark. Nevertheless, given the limited effect sizes, the overall picture is that motivational factors seem to be relatively similar regarding disciplinary and country dimensions.

Regarding gender aspects of motivation, our data show that women seem to view practical application and career progress as more important than men. One explanation for this could be the continued gender differences in academic careers, which tend to disadvantage women, thus creating a greater incentive for female scholars to focus on and be motivated by career progress aspects (Huang et al. 2020 ; Lerchenmueller and Sorenson 2018 ). Unsurprisingly, respondents’ age and academic position influenced the importance of different aspects of motivation, especially regarding career progress. Here, increased age and moving up the positional hierarchy are linked to a decrease in importance. This highlights that older academics and those in more senior positions drew more motivation from other sources that are not directly linked to their personal career gains. This can probably be explained by the academic career ladder plateauing at a certain point in time, as there are often no additional titles and very limited recognition beyond becoming a full professor. Finally, the type of funding that scholars received also had an influence on their productivity and, to a certain extent, citation impact.

Overall, there is little support that researchers across various fields and countries are very different when it comes to their motivation for conducting research. Rather, there seems to be a strong common core of academic motivation that varies mainly by gender and age/position. Rather than talking about researchers’ motivation per se, our study, therefore, suggests that one should talk about motivation across gender, at different stages of the career, and, to a certain degree, in different fields. Thus, motivation seems to be a multi-faceted construct, and the importance of different aspects of motivation vary between different groups.

In the second step of our analysis, we linked motivation to performance. Here, we focused on both scientific productivity and citation impact. Regarding the former, our data show that both practical application and career progress have a significant effect on productivity. The relationship between practical application aspects and productivity is linear, meaning that those who indicate that this aspect of motivation is very important to them have a higher probability of being in the high productivity group. The relationship between career aspects of motivation and productivity is curve linear, and we found only significant differences between the high and low productivity groups at mid and high values of the motivation scale. This indicates that being more motivated by career progress increases productivity but only to a certain extent before it starts having a detrimental effect. A common assumption has been that intrinsic motivation has a positive and instrumental effect and extrinsic motivation has a negative effect on the performance of scientists (Peng and Gao 2019 ; Ryan and Berbegal-Mirabent 2016 ). Our results do not generally support this, as motives related to career progress are positively linked with productivity only to a certain point. Possibly, this can be explained by the fact that the number of publications is often especially important in the context of recruitment and promotion (Langfeldt et al. 2021 ; Reymert et al. 2021 ). Thus, it will be beneficial from a scientific career perspective to have many publications when trying to get hired or promoted.

Regarding citation impact, our analysis highlights that only the career aspects of motivation have a significant effect. Similar to the results regarding productivity, being more motivated by career progress increases the probability of being in the high citation impact group, but only to a certain value when the difference stops being significant. It needs to be pointed out that the effect strength is weaker than in the analysis that focused on productivity. Thus, these results should be treated with greater caution.

Overall, our results shed light on some important aspects regarding the motivation of academics and how this translates into research performance. Regarding our first research question, it seems to be the case that there is not one type of motivation but rather different contextual mixes of motivational aspects that are strongly driven by gender and the academic position/age. We found only limited effects of research fields and even less pronounced country effects, suggesting that while situational, the mix of motivational aspects also has a common academic core that is less influenced by different national environments or disciplinary standards. Regarding our second research question, our results challenge the common assumption that intrinsic motivation has a positive effect and extrinsic motivation has a negative effect on the performance of scientists. Instead, we show that motives related to career are positively linked to productivity at least to a certain point. Our analysis regarding citation patterns achieved similar results. Combined with the finding regarding the importance of current academic position and age for specific patterns of motivation, it could be argued that the fact that the number of publications is often used as a measurement in recruitment and promotion makes academics that are more driven by career aspects publish more, as this is perceived as a necessary condition for success.

Our study has a clear focus on the research side of academic work. However, most academics do both teaching and research, which raises the question of how far our results can also inform our knowledge regarding the motivation for teaching. On the one hand, previous studies have highlighted that intrinsic motivation is also of high importance for the quality of teaching (see e.g. Wilkesmann and Lauer 2020 ), which fits well with our findings. At the same time, the literature also highlights persistent goal conflicts of academics (see e.g. Daumiller et al. 2020 ), given that extra time devoted to teaching often comes at the costs of publications and research. Given that other findings in the literature show that research performance continues to be of higher importance than teaching in academic hiring processes (Reymert et al. 2021 ), the interplay between research performance, teaching performance, and different types of motivation is most likely more complicated and demands further investigation.

While offering several relevant insights, our study still comes with certain limitations that must be considered. First, motivation is a complex construct. Thus, there are many ways one could operationalise it, and not one specific understanding so far seems to have emerged as best practice. Therefore, our approach to operationalisation and measurement should be seen as an addition to this broader field of measurement approaches, and we do not claim that this is the only sensible way of doing it. Second, we rely on self-reported survey data to measure the different aspects of motivation in our study. This means that aspects such as social desirability could influence how far academics claim to be motivated by certain aspects. For example, claiming to be mainly motivated by personal career gains may be considered a dubious motive among academics.

With respect to the bibliometric analyses, it is important to realise that we have lumped researchers into categories, thereby ‘smoothening’ the individual performances into group performances under the various variables. This has an effect that some extraordinary scores might have become invisible in our study, which might have been interesting to analyse separately, throwing light on the relationships we studied. However, breaking the material down to the lower level of analysis of individual researchers also comes with a limitation, namely that at the level of the individual academic, bibliometrics tend to become quite sensitive for the underlying numbers, which in itself is then hampered by the coverage of the database used, the publishing cultures in various countries and fields, and the age and position of the individuals. Therefore, the level of the individual academic has not been analysed in our study, how interesting and promising outcomes might have been. even though we acknowledge that such a study could yield interesting results.

Finally, our sample is drawn from northwestern European countries and a limited set of disciplines. We would argue that we have sufficient variation in countries and disciplines to make the results relevant for a broader audience context. While our results show rather small country or discipline differences, we are aware that there might be country- or discipline-specific effects that we cannot capture due to the sampling approach we used. Moreover, as we had to balance sufficient variation in framework conditions with the comparability of cases, the geographical generalisation of our results has limitations.

This article investigated what motivates researchers across different research fields and countries and how this motivation influences their research performance. The analysis showed that the researchers are mainly motivated by scientific curiosity and practical application and less so by career considerations. Furthermore, the analysis shows that researchers driven by practical application aspects of motivation have a higher probability of high productivity. Being driven by career considerations also increases productivity but only to a certain extent before it starts having a detrimental effect.

The article is based on a large-N survey of economists, cardiologists, and physicists in Denmark, Norway, Sweden, the Netherlands, and the UK. Building on this study, future research should expand the scope and study the relationship between motivation and productivity as well as citation impact in a broader disciplinary and geographical context. In addition, we encourage studies that develop and validate our measurement and operationalisation of aspects of researchers’ motivation.

Finally, a long-term panel study design that follows respondents throughout their academic careers and investigates how far their motivational patterns shift over time would allow for more fine-grained analysis and thereby a richer understanding of the important relationship between motivation and performance in academia.

Data availability

The data set for this study is available from the corresponding author upon reasonable request.

Aksnes DW, Sivertsen G (2019) A criteria-based assessment of the coverage of Scopus and web of Science. J Data Inform Sci 4(1):1–21.

Article   Google Scholar  

Atta-Owusu K, Fitjar RD (2021) What motivates academics for external engagement? Exploring the effects of motivational drivers and organizational fairness. Sci Public Policy. . November, scab075

Baccini A, Barabesi L, Cioni M, Pisani C (2014) Crossing the hurdle: the determinants of individual. Sci Perform Scientometrics 101(3):2035–2062.

Bornmann L, Leydesdorff L, Mutz R (2013) The use of percentiles and percentile rank classes in the analysis of bibliometric data: opportunities and limits. J Informetrics 7(1):158–165.

Cruz-Castro L, Sanz-Menendez L (2021) What should be rewarded? Gender and evaluation criteria for tenure and promotion. J Informetrics 15(3):1–22.

Daumiller M, Stupnisky R, Janke S (2020) Motivation of higher education faculty: theoretical approaches, empirical evidence, and future directions. Int J Educational Res 99:101502.

Duarte H, Lopes D (2018) Career stages and occupations impacts on workers motivations. Int J Manpow 39(5):746–763.

Evans IM, Meyer LH (2003) Motivating the professoriate: why sticks and carrots are only for donkeys. High Educ Manage Policy 15(3):151–167.

Finkelstein MJ (1984) The American academic profession: a synthesis of social scientific inquiry since World War II. Ohio State University, Columbus

Google Scholar  

Hammarfelt B, de Rijcke S (2015) Accountability in context: effects of research evaluation systems on publication practices, disciplinary norms, and individual working routines in the Faculty of arts at Uppsala University. Res Evaluation 24(1):63–77.

Hangel N, Schmidt-Pfister D (2017) Why do you publish? On the tensions between generating scientific knowledge and publication pressure. Aslib J Inform Manage 69(5):529–544.

Hazelkorn E (2015) Rankings and the reshaping of higher education: the battle for world-class excellence. Palgrave McMillan, Basingstoke

Book   Google Scholar  

Hilbe JM (2017) Logistic regression models. Taylor & Francis Ltd, London

Horodnic IA, Zaiţ A (2015) Motivation and research productivity in a university system undergoing transition. Res Evaluation 24(3):282–292

Huang J, Gates AJ, Sinatra R, Barabási A-L (2020) Historical comparison of gender inequality in scientific careers across countries and disciplines. Proceedings of the National Academy of Sciences 117(9):4609–4616.

Jeong S, Choi JY, Kim J-Y (2014) On the drivers of international collaboration: the impact of informal communication, motivation, and research resources. Sci Public Policy 41(4):520–531.

Jindal-Snape D, Snape JB (2006) Motivation of scientists in a government research institute: scientists’ perceptions and the role of management. Manag Decis 44(10):1325–1343.

Kivistö J, Pekkola E, Lyytinen A (2017) The influence of performance-based management on teaching and research performance of Finnish senior academics. Tert Educ Manag 23(3):260–275.

Kulczycki E, Engels TCE, Pölönen J, Bruun K, Dušková M, Guns R et al (2018) Publication patterns in the social sciences and humanities: evidence from eight European countries. Scientometrics 116(1):463–486.

Lam A (2011) What motivates academic scientists to engage in research commercialization: gold, ribbon or puzzle? Res Policy 40(10):1354–1368.

Langfeldt L, Reymert I, Aksnes DW (2021) The role of metrics in peer assessments. Res Evaluation 30(1):112–126.

Larivière V, Macaluso B, Archambault É, Gingras Y (2010) Which scientific elites? On the concentration of research funds, publications and citations. Res Evaluation 19(1):45–53.

Lepori B, Jongbloed B, Hicks D (2023) Introduction to the handbook of public funding of research: understanding vertical and horizontal complexities. In: Lepori B, Hicks BJ D (eds) Handbook of public funding of research. Edward Elgar Publishing, Cheltenham, pp 1–19

Chapter   Google Scholar  

Lerchenmueller MJ, Sorenson O (2018) The gender gap in early career transitions in the life sciences. Res Policy 47(6):1007–1017.

Leslie DW (2002) Resolving the dispute: teaching is academe’s core value. J High Educ 73(1):49–73

Lounsbury JW, Foster N, Patel H, Carmody P, Gibson LW, Stairs DR (2012) An investigation of the personality traits of scientists versus nonscientists and their relationship with career satisfaction: relationship of personality traits and career satisfaction of scientists and nonscientists. R&D Manage 42(1):47–59.

Ma L (2019) Money, morale, and motivation: a study of the output-based research support scheme. Univ Coll Dublin Res Evaluation 28(4):304–312.

Melguizo T, Strober MH (2007) Faculty salaries and the maximization of prestige. Res High Educt 48(6):633–668

Moed HF (2005) Citation analysis in research evaluation. Springer, Dordrecht

Netherlands Observatory of Science (NOWT) (2012) Report to the Dutch Ministry of Science, Education and Culture (OC&W). Den Haag 1998

Peng J-E, Gao XA (2019) Understanding TEFL academics’ research motivation and its relations with research productivity. SAGE Open 9(3):215824401986629.

Piro FN, Aksnes DW, Rørstad K (2013) A macro analysis of productivity differences across fields: challenges in the measurement of scientific publishing. J Am Soc Inform Sci Technol 64(2):307–320.

Pruvot EB, Estermann T, Popkhadze N (2023) University autonomy in Europe IV. The scorecard 2023. Retrieved from Brussels. autonomy scorecard.pdf

Reymert I, Jungblut J, Borlaug SB (2021) Are evaluative cultures national or global? A cross-national study on evaluative cultures in academic recruitment processes in Europe. High Educ 82(5):823–843.

Roach M, Sauermann H (2010) A taste for science? PhD scientists’ academic orientation and self-selection into research careers in industry. Res Policy 39(3):422–434.

Rørstad K, Aksnes DW (2015) Publication rate expressed by age, gender and academic position– A large-scale analysis of Norwegian academic staff. J Informetrics 9(2):317–333.

Ruiz-Castillo J, Costas R (2014) The skewness of scientific productivity. J Informetrics 8(4):917–934.

Ryan JC (2014) The work motivation of research scientists and its effect on research performance: work motivation of research scientists. R&D Manage 44(4):355–369.

Ryan JC, Berbegal-Mirabent J (2016) Motivational recipes and research performance: a fuzzy set analysis of the motivational profile of high-performing research scientists. J Bus Res 69(11):5299–5304.

Ryan RM, Deci EL (2000) Intrinsic and extrinsic motivations: classic definitions and new directions. Contemp Educ Psychol 25(1):54–67.

Sivertsen G (2019) Understanding and evaluating research and scholarly publishing in the social sciences and humanities (SSH). Data Inform Manage 3(2):61–71.

Sivertsen G, Van Leeuwen T (2014) Scholarly publication patterns in the social sciences and humanities and their relationship with research assessment

Stephan P, Veugelers R, Wang J (2017) Reviewers are blinkered by bibliometrics. Nature 544(7651):411–412.

Thomas D, Nedeva M (2012) Characterizing researchers to study research funding agency impacts: the case of the European Research Council’s starting grants. Res Evaluation 21(4):257–269.

Tien FF (2000) To what degree does the desire for promotion motivate faculty to perform research? Testing the expectancy theory. Res High Educt 41(6):723–752.

Tien FF (2008) What kind of faculty are motivated to perform research by the desire for promotion? High Educ 55(1):17–32.

Tien FF, Blackburn RT (1996) Faculty rank system, research motivation, and faculty research productivity: measure refinement and theory testing. J High Educ 67(1):2.

Vallerand RJ, Pelletier LG, Blais MR, Briere NM, Senecal C, Vallieres EF (1992) The academic motivation scale: a measure of intrinsic, extrinsic, and amotivation in education. Educ Psychol Meas 52(4):1003–1017.

Van Iddekinge CH, Aguinis H, Mackey JD, DeOrtentiis PS (2018) A meta-analysis of the interactive, additive, and relative effects of cognitive ability and motivation on performance. J Manag 44(1):249–279.

Van Leeuwen T (2013) Bibliometric research evaluations, Web of Science and the social sciences and humanities: A problematic relationship? Bibliometrie - Praxis Und Forschung, September, Bd. 2(2013).

Van Leeuwen T, van Wijk E, Wouters PF (2016) Bibliometric analysis of output and impact based on CRIS data: a case study on the registered output of a Dutch university. Scientometrics 106(1):1–16.

Waltman L, Schreiber M (2013) On the calculation of percentile-based bibliometric indicators. J Am Soc Inform Sci Technol 64(2):372–379.

Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AFJ (2011) Towards a new crown indicator: an empirical analysis. Scientometrics 87(3):467–481.

Wilkesmann U, Lauer S (2020) The influence of teaching motivation and new public management on academic teaching. Stud High Educ 45(2):434–451.

Wilsdon J, Allen L, Belfiore E, Campbell P, Curry S, Hill S, Jones R et al (2015) The metric tide: report of the independent review of the role of metrics in research assessment and management.

Zacharewicz T, Lepori B, Reale E, Jonkers K (2019) Performance-based research funding in EU member states—A comparative assessment. Sci Public Policy 46(1):105–115.

Zhang L, Sivertsen G, Du H, Huang Y, Glänzel W (2021) Gender differences in the aims and impacts of research. Scientometrics 126(11):8861–8886.

Download references


We are thankful to the R-QUEST team for input and comments to the paper.

The authors disclosed the receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Research Council Norway (RCN) [grant number 256223] (R-QUEST).

Open access funding provided by University of Oslo (incl Oslo University Hospital)

Author information

Silje Marie Svartefoss

Present address: TIK Centre for Technology, Innovation and Culture, University of Oslo, 0317, Oslo, Norway

Authors and Affiliations

Nordic Institute for Studies in Innovation, Research and Education (NIFU), Økernveien 9, 0608, Oslo, Norway

Silje Marie Svartefoss & Dag W. Aksnes

Department of Political Science, University of Oslo, 0315, Oslo, Norway

Jens Jungblut & Kristoffer Kolltveit

Centre for Science and Technology Studies (CWTS), Leiden University, 2311, Leiden, The Netherlands

Thed van Leeuwen

You can also search for this author in PubMed   Google Scholar


All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Silje Marie Svartefoss, Jens Jungblut, Dag W. Aksnes, Kristoffer Kolltveit, and Thed van Leeuwen. The first draft of the manuscript was written by all authors in collaboration, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Silje Marie Svartefoss .

Ethics declarations

Competing interests.

The authors have no competing interests to declare that are relevant to the content of this article.

Informed consent

was retrieved from the participants in this study.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit .

Reprints and permissions

About this article

Svartefoss, S.M., Jungblut, J., Aksnes, D.W. et al. Explaining research performance: investigating the importance of motivation. SN Soc Sci 4 , 105 (2024).

Download citation

Received : 14 December 2023

Accepted : 15 April 2024

Published : 23 May 2024


Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Performance
  • Productivity
  • Find a journal
  • Publish with us
  • Track your research

Developing NICE guidelines: the manual

NICE process and methods [PMG20] Published: 31 October 2014 Last updated: 29 May 2024

  • Tools and resources
  • 1 Introduction
  • 2 The scope
  • 3 Decision-making committees
  • 4 Developing review questions and planning the evidence review

5 Identifying the evidence: literature searching and evidence submission

  • 6 Reviewing evidence
  • 7 Incorporating economic evaluation
  • 8 Linking to other guidance
  • 9 Interpreting the evidence and writing the guideline
  • 10 The validation process for draft guidelines, and dealing with stakeholder comments
  • 11 Finalising and publishing the guideline recommendations
  • 12 Support for putting the guideline recommendations into practice
  • 13 Ensuring that published guidelines are current and accurate
  • 14 Updating guideline recommendations
  • 15 Appendices
  • Update information

NICE process and methods

5.1 introduction, 5.2 searches during guideline recommendation scoping and surveillance, 5.3 searches during guideline recommendation development, 5.4 health inequalities and equality and diversity, 5.5 quality assurance, 5.6 documenting the search, 5.7 re-running searches, 5.8 calls for evidence from stakeholders, 5.9 references and further reading.

The systematic identification of evidence is an essential step in developing NICE guideline recommendations.

This chapter sets out how evidence is identified at each stage of the guideline development cycle. It provides details of the systematic literature searching methods used to identify the best available evidence for NICE guidelines. It also provides details of associated information management processes including quality assurance (peer review), re‑running searches, and documenting the search process.

Our searching methods are informed by the chapter on searching & selecting studies in the Cochrane Handbook for Systematic Reviews of Interventions and the Campbell Collaboration's searching for studies guide . The Summarized Research in Information Retrieval for HTA (SuRe Info) resource also provides research-based advice on information retrieval for systematic reviews.

Our literature searches are designed to be systematic, transparent, and reproducible, and minimise dissemination bias. Dissemination bias may affect the results of reviews and includes publication bias and database bias.

We use search methods that balance recall and precision. When the need to reduce the number of studies requires pragmatic search approaches that may increase the risk of missing relevant studies, the context and trade-offs are discussed and agreed within the development team and made explicit in the reported search methods.

A flexible approach to identifying evidence is adopted, guided by the subject of the review question (see the chapter on developing review questions and planning the evidence review ), type of evidence sought, and the resource constraints of the evidence review. Often an evidence review will be an update of our earlier work, therefore the approach can be informed by previous searches and surveillance reviews (see the chapter on ensuring that published guidelines are current and accurate ).

Scoping searches

Scoping searches are top-level searches to support scope development. The purpose of the searches is to investigate the current evidence around the topic, and to identify any areas where an evidence review may be beneficial and any research gaps. The results of the searches are used to draft the scope of the upcoming guideline or update and to inform the discussions at scoping workshops (if held). Scoping searches do not aim to be exhaustive.

In some cases, scoping searches are not required when it is more efficient to use the surveillance review (see the chapter on the scope ).

The sources searched at scoping stage will vary according to the topic, type of review questions the guideline or update will seek to address, and type of evidence sought. Each scoping search is tailored using combinations of the following types of information:

NICE guidance and guidance from other organisations

policy and legislation guides

key systematic reviews and epidemiological reviews

economic evaluations

current practice data, including costs and resource use and any safety concerns

views and experiences of people using services, their family members or carers, or the public

other real-world health and social care data (for example audits, surveys, registries, electronic health records, patient-generated health data), if appropriate

summaries of interventions that may be appropriate, including any national safety advice

statistics (for example on epidemiology, natural history of the condition, service configuration or national prevalence data).

All scoping searches are fully documented and if new issues are identified at a scoping workshop, the search is updated. A range of possible sources considered for scoping searches is provided in the appendix on suggested sources for scoping .

Health inequalities searches

The purpose of these searches is to identify evidence to help inform the scope, health inequalities briefing, or the equality and health inequalities assessment (EHIA). They help identify key issues relevant to health inequalities on the topic, for example covering protected characteristics, groups experiencing or at risk of inequalities, or wider determinants of health.

The searches involve finding key data sources, such as routinely available national databases, audits or published reports by charities, non-governmental bodies, or government organisations.

Surveillance searches

Surveillance determines whether published recommendations remain current. The searches are tailored to the evidence required. This may include searches for new or updated policies, legislation, guidance from other organisations, or ongoing studies in the area covered by the evidence review.

If required, published evidence is identified by searching a range of bibliographic databases relevant to the topic. Surveillance searches generally use the same core set of databases used during the development of the original evidence review. A list of sources is given in the appendix on sources for evidence reviews .

The search approach and sources will vary between topics and may include:

population and intervention searches

focused searches for specific question areas

forward and backward citation searching.

Searches usually focus on randomised controlled trials and systematic reviews, although other study types will be considered where appropriate, for example for diagnostic questions.

The search period starts at either the end of the search for the last update of a guideline evidence review, or at the last search date for any previous surveillance check. Where appropriate, living evidence surveillance could be set up to continuously monitor the publication of new evidence over a period of time until impact reaches the threshold for actions. For more information on NICE guideline recommendation surveillance, see the chapter on ensuring that guideline recommendations are current and accurate and appendix on surveillance - interim principles for monitoring approaches of guideline recommendations .

Search protocols

Search protocols form part of the wider guideline review protocol (see the appendix on the review protocol template ). They pre‑define how the evidence is identified and provide a basis for developing the search strategies.

Once the final scope is agreed, the information specialist develops the search protocols and agrees them with the development team before the evidence search begins.

A search protocol includes the following elements:

approach to the search strategy, tailored to the review question and eligibility criteria

sources to be searched

plans to use any additional or alternative search techniques , when known at the protocol development stage, and the reasons for their use

details of any limits to be applied to the search

references to any key papers used to inform the search approach.

Searches are done on a mix of bibliographic databases, websites and other sources, depending on the subject of the review question and the type of evidence sought.

For most searches there are key sources that are prioritised, and other potentially relevant sources that can be considered. It is important to ensure adequate coverage of the relevant literature and to search a range of sources. However, there are practical limits to the number of sources that can be searched in the standard time available for an evidence review.

The selection of sources varies according to the requirements of the review question.

Clinical intervention sources

For reviews of the effectiveness of clinical interventions the following sources are prioritised for searching:

the Cochrane Central Register of Controlled Trials (CENTRAL)

the Cochrane Database of Systematic Reviews (CDSR)

Clinical safety sources

In addition to the sources searched for clinical interventions, the following should be prioritised for clinical safety review questions:

MHRA drug safety updates

National patient safety alerts .

Antimicrobial resistance sources

For reviews of antimicrobial resistance, the following sources should be prioritised:

UK Health Security Agency's English surveillance programme for antimicrobial utilisation and resistance (ESPAUR) report

UK Health Security Agency's antimicrobial resistance local indicators .

Cost-effectiveness sources

For reviews of cost effectiveness, economic databases are used in combination with general bibliographic databases, such as MEDLINE and Embase (see appendix G on sources for economic reviews ).

Economic evaluations of social care interventions may be published in journals that are not identified through standard searches. Targeted searches based on references of key articles and contacting authors can be considered to identify relevant papers.

Topic-specific sources

Some topics we cover may require the use of topic-specific sources. Examples include:

PsycINFO (psychology and psychiatry)

CINAHL (nursing and allied health professions)

ASSIA (Applied Social Sciences Index and Abstracts)

HealthTalk , and other sources to identify the views and experiences of people using services, carers and the public

social policy and practice

social care online

sociological abstracts

transport database

Greenfile (environmental literature)

HMIC (Health Management Information Consortium).

Searching for model inputs

Evidence searches may be needed to inform design-oriented conceptual models. Examples include precise searches to find representative NHS costs for an intervention or finding out the proportion of people offered an intervention who take up the offer.

Some model inputs, such as costs, use national sources such as national list prices or national audit data. In some cases, it may be more appropriate to identify costs from the academic literature. Further advice on methods to identify model inputs are also informed by Paisley (2016) and Kaltenhaler et al. (2011). See also the chapter on incorporating economic evaluation .

Real-world data

Information specialists can identify sources of real-world data (such as electronic health records, registries, and audits) for data analysts to explore further. The Health Data Research Innovation Gateway can be used to identify datasets. The NICE real-world evidence framework (2022) has additional guidance on searching for and selecting real-world data sources.

Grey literature

For some review questions, for example, where significant evidence is likely to be published in non-journal sources and there is a paucity of evidence in published journal sources, it may be appropriate to search for grey literature . Useful sources of grey literature include:

HMIC (Health Management Information Consortium)

TRIP database

Canadian Agency for Drugs and Technology in Health (CADTH) Grey Matters resource .

Committee members may also be able to suggest additional appropriate sources for grey literature.

A list containing potential relevant sources is provided in the appendix on sources for evidence reviews .

Developing search strategies

The approach to devising and structuring search strategies is informed by the review protocol. The PICO (population, intervention, comparator and outcome) or SPICE (setting, perspective, intervention, comparison, evaluation) frameworks may be used to structure a search strategy for intervention review questions. For other types of review questions, alternative frameworks may be more suitable.

It is sometimes more efficient to conduct a single search for multiple review questions, rather than conducting a separate search for each question.

Some topics may not easily lend themselves to PICO- or SPICE-type frameworks. In these cases, it may be better to combine multiple, shorter searches rather than attempting to capture the entire topic using a single search. This is often referred to as multi-stranded searching.

In some instances, for example where the terminology around a topic is diffuse or ill defined, it may be difficult to specify the most appropriate search terms in advance. In these cases, an iterative approach to searching can be used.

In an iterative approach, searching is done in several stages, with each search considering the evidence that has already been retrieved (for example, see Booth et al. 2020 ). Searching in stages allows the reviewers to review the most relevant, high-quality information first and then make decisions for identifying additional evidence if needed.

Decisions to use iterative approaches are agreed by the development team and staff with responsibility for quality assurance because it can affect timelines.

Updating previous work

Where high-quality review-level evidence is available on a topic, the review team may choose to update or expand this previous work rather than duplicating the existing findings. In these cases, the original review searches are re-run and expanded to account for any differences in scope and inclusion criteria between the original review and the update.

Cost-effectiveness searches

There are several methods that can be used to identify economic evaluations:

All relevant review questions can be covered by a single search using the population search terms, combined with a search filter, to identify economic evidence.

The search strategies for individual review questions can be combined with search filters to identify economic evidence. If using this approach, it may be necessary to adapt strategies for some databases to ensure adequate sensitivity.

Economic evidence can be manually sifted while screening evidence from a general literature search (so no separate searches are required).

The rationale for the selected approach is recorded in the search protocol.

Where searches are needed to populate an economic model, these are usually done separately.

Identifying search terms

Search terms usually consist of a combination of subject headings and free‑text terms from the titles and abstracts of relevant references.

When identifying subject headings, variations in thesaurus and indexing terms for each database should be considered, for example MeSH (Medical Subject Headings) in MEDLINE and Emtree in Embase. Not all databases have indexing terms and some contain records that have not yet been indexed.

Free‑text terms may include synonyms, acronyms and abbreviations, spelling variants, old and new terminology, brand and generic medicine names, and lay and medical terminology.

For updates, previous search terms, including those from surveillance searches, are reviewed and used to inform new search terms. New or changed terms are identified, as well as any changes to indexing terms. This also applies when an existing review, for example a Cochrane review, is being updated to answer a review question.

Key studies can be a useful source of search terms, as can reports, guidelines, topic-specific websites, committee members and topic experts.

Some websites and databases have limited search functionality. It may be necessary to use fewer search terms or do multiple searches of the same resource with different search term combinations.

It may be helpful to use frequency analysis or text mining to develop the search-term strategy. Tools such as PubReMiner and Medline Ranker can help, either by highlighting search terms that might not otherwise be apparent, or by flagging terms of high value when exhaustive synonym searching is unfeasible or inadvisable.

Search limits

The application of limits to search strategies will reflect the eligibility criteria in the review protocol. Typically, English language limits, date limits, and the exclusion of conference abstracts and animal studies are usually done as a matter of routine.

Search filters

A search filter is a string of search terms with known (validated) performance. When a particular study design is required for a review question, relevant search filters are usually applied to literature search strategies.

Other search filters relating to age, setting, geography, and health inequalities are also applied as relevant. The most comprehensive list of available search filters is the search filter resource of the InterTASC Information Specialists' SubGroup . This resource also includes critical appraisal tools, which are used for filter selection.

Economics-related filters

A variety of search filters of relevance to cost effectiveness are available. These include filters for economic evaluations, quality of life data, and cost-utilities data. It may be necessary to use more than 1 filter to identify relevant data. In addition, it may be appropriate to add geographic search filters, such as those for the UK or Organisation for Economic Co-operation and Development (OECD) countries, to retrieve economic studies relevant to the UK or OECD (Ayiku et al. 2017, 2019, 2021).

Use of machine learning-based classifiers

Machine learning-based classification software has been developed for some study types (for example the Cochrane RCT classifier, Thomas et al. 2020 ). These classifiers apply a probability weighting to each bibliographical reference within a set of search results. The weighting relates to the reference's likelihood to be a particular study type, based on a model created from analysis of known, relevant papers. The weightings can then be used to either order references for screening or be used with a fixed cut-off value to divide a list of references into those more likely to be included, and those that can be excluded without manual screening.

We support the use of machine classifiers if their performance characteristics are known, and if they improve efficiency in the search and screening process. However, caution is needed when using classifiers, because they may not be as effective if used on data that is different to the type of data for which they were originally developed. For example, the Cochrane RCT classifier is reported to have over 99% recall for health studies but showed "unacceptably low" recall for educational research ( Stansfield et al. 2022 ).

Priority screening, a type of machine classifier that orders references for manual sifting based on previous sifting decisions, is considered in the chapter on reviewing evidence .

Additional search techniques

Additional search techniques are used alongside database searching when it is known, or reasonably likely, that relevant evidence is not indexed in bibliographic databases, or when it will be difficult to retrieve relevant evidence from databases in a way that adequately balances recall and precision. Additional search techniques include forward and backward citation searching, journal hand-searches and contacting experts and stakeholders.

Existing reviews may provide an additional source of primary studies, with reference lists being used as an indirect method of identifying primary research.

Various tools, including Citationchaser and Web of Science, are available to speed up the process of citation searching. These may not be as comprehensive as manual reference list checking (due to limitations of the underlying data sources), but the trade-off in terms of speed is generally acceptable.

All search techniques should follow the same principles of transparency, rigour and reproducibility as other search methods.

If possible, additional search techniques should be considered at the outset and documented in the search protocol. They should also be documented in the supporting appendices for the final evidence review.

All searches aim to be inclusive. This may mean not specifying any population groups.

Searches should avoid inadvertently excluding relevant groups. For example, if the population group is older people, a search for older people should pick up subpopulations such as disabled older people.

Additional search strategies may be needed to target evidence about people with protected characteristics or people experiencing or at risk from other inequalities.

Searches may need to be developed iteratively to ensure coverage of the health inequalities issues or evidence on the impacts of an intervention on equality.

Appropriate terminology for the search should be used, considering how language has evolved.

Quality assuring the literature search is an important step in developing guideline recommendations. Studies have shown that errors do occur.

For each search (including economic searches), the initial MEDLINE search strategy is quality assured by a second information specialist. A standardised checklist, based on the PRESS peer review of electronic search strategies: 2015 guideline statement , is used to ensure clarity and consistency when quality assuring search strategies.

The information specialist carrying out the quality assurance process also considers how appropriate the overall search approach is to the parameters of the evidence review (for example, the time available to carry out the review). The quality assurance comments are recorded and the information specialist who conducted the search should respond to the comments and revise the search strategy as needed.

Search strategy translations across the remaining databases are also checked by a second information specialist to ensure that the strategies have been adapted appropriately, in accordance with the interfaces and search functionality of the sources used.

Details of the evidence search are included as appendices to the individual evidence reviews. They are published for consultation alongside the draft evidence review and included in the final version.

Records are kept of the searches undertaken during guideline recommendation development for all review questions to ensure that the process for identifying the evidence is transparent and reproducible.

We use the PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews to inform search reporting. The search documentation is an audit trail that allows the reader to understand both the technical aspect of what was done (such as which sources were searched; what platform was used and on what date; any deviations from the original search protocol) and the underlying rationale for the search approach where this may not be immediately apparent.

Documenting the search begins with creating the search protocol (see the section on search protocols ). If using an iterative or emergent stepped approach, initial search strategies, key decision points and the reasons for subsequent search steps are clearly documented in the search protocol and final evidence review. When using a proprietary search engine such as Google, whose underlying algorithm adapts to different users, the search is reported in a way that should allow the reader to understand what was done.

Searches undertaken to identify evidence for each review question (including economics searches) may be re-run before consultation or before publication. For example, searches are re‑run if the evidence changes quickly, there is reason to believe that substantial new evidence exists, or the development time is longer than usual.

A decision to re‑run searches is taken by the development team and staff with responsibility for quality assurance.

If undertaken, searches are re‑run at least 6 to 8 weeks before the final committee meeting before consultation.

If evidence is identified after the last cut‑off date for searching but before publication, a judgement on its impact is made by the development team and staff with responsibility for quality assurance. In exceptional circumstances, this evidence can be considered if its impact is judged as potentially substantial.

In some topic areas or for some review questions, staff with responsibility for quality assurance, the development team or the committee may believe that there is relevant evidence in addition to that identified by the searches. In these situations, the development team may invite stakeholders, and possibly also other relevant organisations or individuals with a significant role or interest (see expert witnesses in the section on other attendees at committee meetings in the chapter on decision-making committees ), to submit evidence. A call for evidence is issued directly to registered stakeholders on the NICE website. Examples and details of process are included in the appendix on call for evidence and expert witnesses . Confidential information should be kept to an absolute minimum.

Ayiku L, Levay P, Hudson T et al. (2017) The medline UK filter: development and validation of a geographic search filter to retrieve research about the UK from OVID medline. Health Information and Libraries Journal 34(3): 200–216

Ayiku L, Levay P, Hudson T et al. (2019) The Embase UK filter: validation of a geographic search filter to retrieve research about the UK from OVID Embase. Health Information and Libraries Journal 36(2): 121–133

Ayiku L, Hudson T, Williams C et al. (2021) The NICE OECD countries' geographic search filters: Part 2-validation of the MEDLINE and Embase (Ovid) filters . Journal of the Medical Library Association 109(4): 583–9

Booth A, Briscoe S, Wright JM (2020) The "realist search": a systematic review of current practice and reporting . Research Synthesis Methods 11: 14–35

Canadian Agency for Drugs and Technologies in Health (2019) Grey Matters: a practical tool for searching health-related grey literature [online; accessed 24 July 2023]

Glanville J, Lefebvre C, Wright K (editors) (2008, updated 2017) The InterTASC Information Specialists' Subgroup Search Filters Resource [online; accessed 24 July 2023]

Kaltenthaler E, Tappenden P, Paisley S (2011) NICE DSU Technical support document 13: identifying and reviewing evidence to inform the conceptualisation and population of cost-effectiveness models [online; accessed 24 July 2023]

Kugley S, Wade A, Thomas J et al. (2017) Searching for studies: a guide to information retrieval for Campbell systematic reviews . Oslo: The Campbell Collaboration

Lefebvre C, Glanville J, Briscoe S et al. Chapter 4: Searching for and selecting studies . In: Higgins JPT, Thomas J, Cumpston M et al. (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021

McGowan J, Sampson M, Salzwedel DM et al. (2016) PRESS Peer Review of Electronic Search Strategies: 2015 guideline statement . Journal of Clinical Epidemiology 75: 40–6

National Institute for Health and Care Excellence (2022) NICE real-world evidence framework [online; accessed 24 July 2023]

Paisley S (2016) Identification of key parameters in decision-analytic models of cost-effectiveness: a description of sources and a recommended minimum search requirement. Pharmacoeconomics 34: 597–8

Rethlefsen M, Kirtley S, Waffenschmidt S et al. (2021) PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews . Systematic Reviews 10: 39

Stansfield C, Stokes G, Thoman J (2022) Applying machine classifiers to update searches: analysis from two case studies . Research Synthesis Methods 13: 121–33

Summarized research for Information Retrieval in HTA (SuRe Info) [online; accessed 24 July 2023]

Reliability and validity: Importance in Medical Research


  • 1 Al-Nafees Medical College,Isra University, Islamabad, Pakistan.
  • 2 Fauji Foundation Hospital, Foundation University Medical College, Islamabad, Pakistan.
  • PMID: 34974579
  • DOI: 10.47391/JPMA.06-861

Reliability and validity are among the most important and fundamental domains in the assessment of any measuring methodology for data-collection in a good research. Validity is about what an instrument measures and how well it does so, whereas reliability concerns the truthfulness in the data obtained and the degree to which any measuring tool controls random error. The current narrative review was planned to discuss the importance of reliability and validity of data-collection or measurement techniques used in research. It describes and explores comprehensively the reliability and validity of research instruments and also discusses different forms of reliability and validity with concise examples. An attempt has been taken to give a brief literature review regarding the significance of reliability and validity in medical sciences.

Keywords: Validity, Reliability, Medical research, Methodology, Assessment, Research tools..

Publication types

  • Biomedical Research*
  • Reproducibility of Results

Cookies on the NHS England website

We’ve put some small files called cookies on your device to make our site work.

We’d also like to use analytics cookies. These send information about how our site is used to a service called Google Analytics. We use this information to improve our site.

Let us know if this is OK. We’ll use a cookie to save your choice. You can  read more about our cookies before you choose.

Change my preferences I'm OK with analytics cookies

Maximising the benefits of research: Guidance for integrated care systems

England has a vibrant research and development ecosystem, with well-developed research infrastructure and research expertise within our health and care workforce. The value of research in transforming health and care is significant; additionally, staff satisfaction, recruitment and retention is higher among staff who are involved in research. The inception of integrated care systems (ICSs) provides the opportunity for systems to embed research within health and care for the benefit of our population. Supporting this opportunity, a clear research thread runs through ICS strategies and plans, from joint strategic needs assessments and joint health and wellbeing strategies , integrated care strategies , joint forwards plans , integrated care board (ICB) annual reports and the assessment by NHS England of the discharge of duties by ICBs.

The Health and Care Act 2022 (the 2022 Act) sets new legal duties on ICBs around the facilitation and promotion of research in matters relevant to the health service, and the use in the health service of evidence obtained from research. NHS England will assess ICBs for their discharge of these duties. The ICS design framework sets the expectation that in arranging provision of health services, ICBs will facilitate their partners in the health and care system to work together, combining expertise and resources to foster and deploy research and innovations. This guidance supports ICBs in fulfilling their research duties.

ICSs are encouraged to develop a research strategy that aligns to or could be included in their integrated care strategy. This strategy will enable the unification of research across ICS partners, and be consistently embedded to:

  • identify and address local research priorities and needs, and work collaboratively to address national research priorities
  • improve the quality of health and care and outcomes for all through the evidence generated by research
  • increase the quality, quantity and breadth of research undertaken locally
  • extend and expand research in settings such as primary care, community care, mental health services, public health and social care
  • drive the use of research evidence for quality improvement and evidence-based practice
  • influence the national research agenda to better meet local priorities and needs
  • improve co-ordination and standardisation within and between localities for the set up and delivery of research
  • harness the patient and economic benefits of commercial contract research
  • co-ordinate and develop the research workforce across all settings.

1. Introduction

This guidance sets out what good research practice looks like. It supports integrated care systems (ICSs) to maximise the value of their duties around research for the benefit of their population’s health and care and, through co-ordination across ICSs, for national and international impact. It supports integrated care boards (ICBs), integrated care partnerships (ICPs) and their partners to develop a research strategy that aligns to or can be incorporated into their integrated care strategy, and helps them and their workforce to build on existing research initiatives and activities across health and social care to improve sector-wide performance and best practice

  • explains the ICB legal duties and other requirements around research and the use of evidence from research, and that research is included in forward planning and reporting
  • encourages system leaders to develop a footprint-wide research strategy that aligns to local and national research priorities, develops and supports their workforce, takes the opportunities offered by commercial research and includes plans to embed research in their system’s governance and leadership
  • identifies best practice examples and other resources that ICBs may find useful as they develop their research strategies.

This guidance provides comprehensive information for use by:

  • those with senior responsibility, including at board level, for research strategy development and/or operationalising research
  • managers responsible for developing joint strategic needs assessments, integrated care strategies, joint health and wellbeing strategies, joint forward plans, other linked strategies, or reporting on ICB activities
  • research managers
  • research and development/innovation leads
  • heads of services
  • knowledge and library specialists.

It may also be useful to individuals involved in research, education, and partner organisations such as local authorities, social care services, the voluntary, community and social enterprise sector (VCSE) and other providers of healthcare services.

NHS England provides guidance on embedding research in the NHS and secure data environments, and the Office for Life Sciences (OLS ) champions research, innovation and the use of technology to transform health and care service. Other sources of guidance, support and information are signposted in this guidance to support ICSs in aligning to national visions, strategies and plans around research.

1.1 Definition of research

NHS England uses the UK Policy Framework for Health and Social Care Research definition of research:

“… the attempt to derive generalisable or transferable new knowledge to answer or refine relevant questions with scientifically sound methods. This excludes audits of practice and service evaluation. It includes activities that are carried out in preparation for or as a consequence of the interventional part of the research, such as screening potential participants for eligibility, obtaining participants’ consent and publishing results. It also includes non-interventional health and social care research (that is, projects that do not involve any change in standard treatment, care, or other services), projects that aim to generate hypotheses, methodological research and descriptive research”.

This broad definition encompasses the range of types of research:

  • clinical trials and other clinical investigations into the safety and effectiveness of medicines, devices and health technologies
  • public health research
  • observational studies
  • discovery science and experimental medicine
  • translational research in which results from basic research are developed into results that directly benefit people
  • applied research
  • research to support policy-making and commissioning
  • social care research and research in social care settings
  • research into NHS services and care pathways.

1.2 Why research is important

The UK is a world leader for research and invention in healthcare, with around 25% of the world’s top 100 prescription medicines being discovered and developed in the UK ( The impact of collaboration: The value of UK medical research to EU science and health ). Research in the health and care system is important because it underpins all advances in health and care and is the basis for evidence-based practice. Engaging clinicians and healthcare organisations in research is associated with improvements in delivery of healthcare ( Does the engagement of clinicians and organisations in research improve healthcare performance: a three-stage review) . To benefit service users and the public, the NHS and local government, and achieve return on investment, it is vital that research is disseminated, shared and translated into practice.

The National Institute for Health and Care Research (NIHR) is funded by the Department of Health and Social Care (DHSC) to transform research in the health and social care system, including through support for NHS research. Research led to the first proven treatments for Covid, for example the use of dexamethasone, estimated to have saved over a million lives worldwide . This success was in part due to how research is undertaken in the unique environment of the NHS, innovative trial designs, the support provided by the NIHR, frontline staff enabling research, and the awareness and readiness of the public to support research. We need to learn from these and other successes, and translate this across all health and care settings. ICSs will play a vital role in enabling research to be embedded in evolving patient pathways across their footprints.

Example: PRINCIPLE trial – finding treatments for Covid recovery at home

The Platform Randomised Trial of Treatment in the Community for Epidemic and Pandemic Illnesses (PRINCIPLE) was a UK-wide, clinical study to find Covid treatments for recovery at home without the need to attend hospital. The study was open to all with ongoing Covid symptoms, registration was easy, and the trial was run entirely remotely by delivering ‘participant packs’ to people’s homes. It was one of the first trials in the world to show that azithromycin and doxycycline did not benefit patients with Covid and to identify the effectiveness of a commonly used drug – inhaled budesonide –in reducing time to recovery.

The PRINCIPLE study team demonstrated the integral role that primary, secondary and ambulatory care staff can play in the delivery of studies. Local collaborators were trained in good clinical practice to allow them to assess and confirm the eligibility of potential participants, and were commended specifically for their use of patient data to contact people soon after they received a positive test result. It is this network of local staff contributing to research within their healthcare setting that has enabled over 10,000 people to be recruited onto this study so far – one of the largest at home Covid treatment studies worldwide.

This is an example of a study design that incorporates the vital contributions of healthcare providers across the system.

Policy-makers and commissioners need evidence to support their decision-making around the delivery and system-wide transformation of health and care services, including how health inequalities will be reduced.

There is also evidence that:

  • staff involved in research have greater job satisfaction and staff turnover is lower in research active trusts ( Academic factors in medical recruitment: evidence to support improvements in medical recruitment and retention by improving the academic content in medical posts)
  • research active hospitals have lower mortality rates, and not just among research participants ( Research activity and the association with mortality )
  • 83% of people believe that health research is very important ( Survey of the general public: attitudes towards health research)
  • healthcare performance improvements have been seen from the creation of academic research placements ( Experiences of hospital allied health professionals in collaborative student research projects: a qualitative study )
  • clinical academic research, and in particular the practice changes resulting from it, is associated with improved patient and carer experiences ( A qualitative systematic review and thematic synthesis exploring the impacts of clinical academic activity by healthcare professionals outside medicine ).

Key to having research embedded in health and care is having staff who can understand, undertake, use and generate new research, and share actionable research finding as part of a pro-research culture. Education and training are therefore critical for research to be sustainably embedded within health and care, and for people to develop careers in research and support it in their clinical or care roles.

DHSC, NHS England, the devolved administrations, NIHR and other partners expect to publish a clinical research workforce strategy in 2023/24 to help the UK realise the national clinical research vision outlined in Saving and Improving Lives: The Future of UK Clinical Research Delivery and deliver the Life Sciences Vision to see research embedded in the NHS as part of health and care pathways.

Research will support ICSs to deliver on their four key aims:

Improving outcomes

The NHS 2023/34 priorities and operational planning guidance emphasises the importance of research in improving patient care, outcomes and experience.

Research evidence will inform commissioning decisions to improve experience and outcomes. Research activities should align with the local health priorities identified through local joint strategic needs assessments, and may be best designed and delivered by collaborating with partners. Research priorities may be best addressed by collaborating with partners nationally to design and deliver research.

Tackling inequalities

Research can give a better understanding of local populations and the wider determinants of health, and with this the steps to maintain health and narrow health inequalities.

Enhancing productivity

The development of ICSs creates the opportunity to consider research delivery within the ICS and across ICS boundaries, increasing flexibility of workforce or recruitment while reducing bureaucracy and improving research productivity and value for money.

Supporting social and economic development

An active research ecosystem working in a co-ordinated way and to national standards brings revenue and jobs to regions. The NIHR Clinical Research Network (CRN) supports service users, the public and health and care organisations across England to participate in high-quality research. The 2019 impact and value report detailed the significant income and cost savings that commercial research generates for NHS trusts. Between 2016/17 and 2018/19 the NHS received on average £9,000 per patient recruited to a commercial clinical trial and saved over £5,800 in drug costs for each of these patients. This equates to income of £355 million and cost savings of £26.8 million in 2018/19.

In 2021 150 members of the Association of Medical Research Charities funded £1.55 billion of medical research, including the salaries of 20,000 researchers. Every £1 million spent by charities on medical research in the UK contributes £1.83 million to the economy.

Example: Research that cut problematic prescribing and generated cost savings in general practice – a local health priority

Analysis of routine patient data identified the need for strategies targeting clinicians and patients to curb rising opioid prescribing. From this, the Campaign to Reduce Opioid Prescription (CROP) was launched in 2016, urging GPs across West Yorkshire to ‘think-twice’ before prescribing opioids. This promoted the NICE guidance on chronic pain , which recommends reducing the use of opioids because there is little or no evidence that they make any difference to people’s quality of life, pain or psychological distress, but they can cause harm, including possible addiction.

Over a year 15,000 fewer people were prescribed opioids (a 5.63% relative reduction), a net saving to the NHS of £700,000. The biggest reduction was in people aged over 75, who are at higher risk of opioid-related falls and death, and there was no compensatory rise in the prescribing of other painkillers or referrals to musculoskeletal services.

The CROP campaign, led by researchers at the University of Leeds, has subsequently been rolled out across all ICBs in Yorkshire and the Humber, and the North East and North Cumbria ICB, and the 1,045 practices to which it has been delivered are reporting results similar to the above.

Foy R, Leaman B, McCrorie C, Petty D, House A, Bennett M, et al (2016) Prescribed opioids in primary care: cross-sectional and longitudinal analyses of influence of patient and practice characteristics | BMJ Open 69(5).

Alderson SL, Faragher TM, Willis TA, Carder P, Johnson S, Foy R (2021) The effects of an evidence- and theory-informed feedback intervention on opioid prescribing for non-cancer pain in primary care: A controlled interrupted time series analysis. PLOS Med .

2. ICS, ICP and ICB responsibilities and requirements

ICBs have legal duties and other requirements that relate to research. These are additional to the duties and responsibilities of individual providers within ICS footprints. This section sets out what these duties mean in practical terms and gives examples of how to meet them.

2.1 Legal duties relating to research in the Health and Care Act 2022

Part 1 of the 2022 Act includes specific legal duties for ICBs and NHS England in respect of research. In the Explanatory Notes to the 2022 Act, government sets out how ICBs could discharge their research duty.

Duty to facilitate or otherwise promote research

The ICB duty builds on the previous clinical commissioning group (CCG) duty to promote research, by requiring each ICB, in the exercise of its functions, to facilitate or otherwise promote research on matters relevant to the health service. This duty is intended to include a range of activities to enable research. Section 3 of this guidance outlines ways in which ICBs can do this.

The NHS Constitution also makes clear that patients should be enabled to take part in research: “the NHS pledges … to inform you of research studies in which you may be eligible to participate”.

The Provider Selection Regime (PSR) will be a new set of rules for arranging healthcare services in England, introduced by regulations made under the 2022 Act. The research component should be referred to once the PSR is published.

Duty to facilitate or otherwise promote the use in the health service of evidence obtained from research

This duty similarly builds on the CCG requirement to promote the use of evidence. ICBs must, in the exercise of their functions, facilitate or otherwise promote the use in the health service of evidence obtained from research. For example, ICBs should facilitate or otherwise promote the use of evidence in care, clinical and commissioning decisions.

Duty for ICSs to include research in their joint forward plans and annual reports

Joint forward plans are five-year plans developed by ICBs and their partner NHS trusts and foundation trusts. Systems are encouraged to use the joint forward plan as a shared delivery plan for the integrated care strategy and joint health and wellbeing strategy, aligned to the NHS’s universal commitments. The plan must explain how the ICB will discharge its duties around research, and the ICB must report on the discharge of its research duties in its annual report. These inclusions will raise the profile of research at board level and help embed research as a business-as-usual activity.

The joint forward plan and NHS Oversight Framework guidance set the minimum requirements for what needs to be included in plans and reports.

NHS England duty to include how each ICB is carrying out its duties relating to research in its annual performance assessment of each ICB

NHS England has a new legal duty to annually assess the performance of each ICB and publish a summary of its findings. For 2022/23 NHS England will complete a narrative assessment, identifying areas of good and/or outstanding performance, areas for improvement and any areas that are particularly challenged, drawing on national expertise as required and having regard to relevant guidance. This assessment will include a section considering how effectively the ICB has discharged its duties to facilitate or otherwise promote research and the use of evidence obtained from research.

This, alongside the implementation of the NHS Long Term Plan commitment to develop research metrics for NHS providers, will increase transparency across the system and enable more targeted support for research. Research metrics from NHS England, the Care Quality Commission (CQC) and NIHR will enable the monitoring of progress over time, and are under development with sector colleagues, including providers.

2.2 Legal requirement to work with people and communities

Working with people and communities is a requirement of ICBs, and statutory guidance is available to support them and their partner providers meet this legal duty. A co-ordinated approach across healthcare delivery and research will make it more likely that research reflects what matters to people and communities.

This will also help ICBs to fulfil their legal duty in the 2022 Act to reduce health inequalities in access to health services and the outcomes achieved. Section 3.9 includes links to resources to help guide engagement with underserved communities around research.

The Public Sector Equality Duty also applies and requires equality of opportunities between persons who share a relevant protected characteristic and persons who do not.

2.3 Research governance

While research can address local priorities, it typically operates across ICS boundaries and at national and international levels. Health and social care research is governed by a range of laws, policies, and international, national and professional standards.

The Health Research Authority (HRA ) is responsible for ensuring such regulation is co-ordinated and standardised across the UK to make it easier to do research that people can trust. The HRA is an executive non-departmental public body created by the Care Act 2014 to protect and promote the interests of patients and the public in health and social care research, including by co-ordinating and standardising the practice of research regulation. Local authorities and the NHS are obliged to have regard to its guidance on the management and conduct of research.

Before a research project can start in the NHS in England it must receive approval from the HRA. This includes research taking place in NHS trusts, NHS foundation trusts, ICBs or primary care providers of NHS commissioned services in England, and all research under an NHS duty of care, including that undertaken by NHS staff working in social care or other non-NHS environments.

The HRA schemes indemnify NHS organisations accepting these assurances against any claim covered by the NHS Litigation Authority arising as a result of incorrect assurances. If an NHS organisation duplicates the HRA assessments, it will be liable for any consequences of the decisions it bases on its own checks.

ICBs and partner organisations should have processes for the set up and delivery of research that comply with national laws and systems, and does not duplicate them. Such national systems include confirmation of capacity, National Contract Value Review (NCVR), management of Excess Treatment Costs (ETCs) and contracting arrangements (see section 2.4).

The UK Policy Framework for Health and Social Care sets out the roles and responsibilities of individuals and organisations involved in research.

2.4 Contractual requirements around research

NHS England mandates commissioner use of the NHS Standard Contract for all contracts for healthcare services other than primary care. The contract is updated annually. References to research in the current NHS Standard Contract and service conditions fall into three main areas.

Recruitment of service users and staff into approved research studies

The NHS Standard Contract obliges every provider of NHS-funded services to assist the recruitment of suitable subjects (whether patients or staff) into approved research studies. This requirement aligns to those in the 2022 Act that require ICBs to facilitate or otherwise promote research (see section 2.1). Section 3 considers how this requirement can best be met. Research involving people or their data requires ethical and potentially other approvals (see section 2.3).

National Directive on Commercial Contract Research Studies

Adherence to the National Directive is mandated as part of the NHS Standard Contract. The directive states that providers must:

  • Use the unmodified model agreements for sponsor-to-site contracting; HRA and Health and Care Research Wales (HCRW) approval of studies will be dependent on use of these templates.
  • Use the standard costing methodology to set prices for commercial contract research undertaken by NHS providers; this is currently in the NIHR interactive costing tool (NIHR iCT).
  • Introduce the National Contract Value Review (NCVR) process in line with national rollout. NCVR is a standardised national approach to costing commercial contract research within the NHS. It currently covers acute, specialist and mental health trusts, but the intention is to roll it out to all NHS providers. The creation of ICSs is the ideal opportunity to explore how commercial study set up can be supported across these footprints, reducing the resource needed and time taken.

Comply with HRA/NIHR research reporting guidance

The provider must comply with HRA/NIHR research reporting guidance, as applicable.

2.5 Excess treatment costs

Patients in a research study may receive healthcare that differs from what is standard in the NHS, requires more clinician time or is delivered in a different location. The associated NHS treatment costs may exceed or be less than those of standard treatment. If greater, the difference is referred to as the NHS Excess Treatment Costs (ETCs).

In the case of commercial contract research, the commercial funder will pay the full cost of the study. In the case of non-commercial research, the commissioner of the service in which the study operates is responsible for funding the ETCs.

ICBs as commissioners of services are responsible for ETCs in services that they commission. Guidance for the management of ETCs is available.

DHSC and NIHR are piloting interim arrangements to support non-NHS ETCs for research in public health and social care (non-NHS intervention costs). Please refer to the further detail on the NIHR website .

2.6 Care Quality Commission

The CQC is currently developing its approach for ICS-level assessments, and its new assessment framework will be introduced towards the end of 2023 .

CQC inspection of NHS providers continue, with research assessed as part of the review of the trust-level Well-led framework. Providers are asked:

  • Are divisional staff aware of research undertaken in and through the trust, how it contributes to improvement and the service level needed across departments to support it?
  • How do senior leaders support internal investigators initiating and managing clinical studies?
  • Does the vision and strategy incorporate plans for supporting clinical research activity as a key contributor to best patient care?
  • Does the trust have clear internal reporting systems for its research range, volume, activity, safety and performance?
  • How are service users and carers given the opportunity to participate in or become actively involved in clinical research studies in the trust?

3. Developing a research strategy

3.1 why develop a research strategy.

Like the health and care system, the research environment is complex. Developing a research strategy will help bring together the legal and other duties around research in a coherent way, and help the ICS understand its local research capability, workforce, activity and needs, set ambitions around research and maximise the benefits associated with commercial research. It will help demonstrate the benefit of research locally, nationally and internationally, and guide the production of clear plans.

Example: Value of research partnerships and integration with ICSs

Bristol Health Partners (BHP) Academic Health Science Centre (AHSC) has a fully integrated relationship as the new Research and Innovation Steering Group for the Bristol, North Somerset and South Gloucestershire (BNSSG) ICS, and reports directly to ICB chief executives.

The group provides the strategic direction and oversight for all research undertaken and delivered across the system. Membership includes directors of research, clinical strategy, public health, social care, senior innovation and education leaders from its core funding partners. It also includes public contributors and senior representatives from primary care, NIHR Applied Research Collaboration West, NIHR CRN West of England, West of England Academic Health Science Network (WEAHSN), Healthier Together ICS, university research institutes and People in Health West of England.

The group has reviewed ICS programmes, identified current and potential research and innovation connections, and begun to establish new connections. It has also supported work with the ICS Ageing Well programme and secured funding for innovative pilots to improve dementia care and increase physical activity for older adults.

Since 2016 BHP has directly contributed an estimated additional £1.1 million to support ICS priorities through Health Integration Team projects and other activities, and has attracted more than £33 million of external research, service redesign and infrastructure into the region.

3.2 General considerations

In developing its research strategy, the ICS may find it helpful to consider these overarching questions alongside the suggested focused content covered in the sections below:

  • What do you hope to achieve within a given timeframe?
  • Are all the right organisations involved in developing the research strategy?
  • How will the health and care workforce be enabled to deliver the research strategy?
  • How can research be embedded in existing health and care delivery and pathways?
  • What mechanisms are in place to translate actionable research findings into practice and decision-making?
  • What inequalities exist in different areas, communities or groups? How will you ensure planning and delivery of research aligns to CORE20plus5 priorities?
  • Are you considering equality, diversity and inclusivity and the Public Sector Equality Duty in facilitating and promoting research opportunities for service users and for health and care staff?
  • Is the ICS considering the opportunities of developing their commercial research portfolio?
  • Is research informing or being informed by population health management?
  • How will you plan and deliver research in a sustainable manner, aligning it to the Greener NHS agenda and the ICB’s duties in relation to climate change ?

Buy-in from NHS staff, patients and the public will be vital if ICBs are to discharge their research duties and deliver on their research plans. An important consideration is how to develop sustainable, routine and accessible information flows to ensure the ICB, partners, staff, patients and public can access up-to-date and appropriate information around local research activity, regional, national and international research opportunities and findings, and contact information.

3.3 Leadership and governance across the ICS

Executive leadership.

The Explanatory Notes to the 2022 Act suggest that ICBs have board-level discussions on research activity, the use of the evidence from research, the research workforce and research culture within the ICS. ICSs should refer to the NHS Leadership Competency Framework for board-level leaders at organisation and ICS level for the competencies relating to the research duties of ICSs, once published.

All ICBs are encouraged to have an executive lead responsible for fulfilling the research duties conferred by the 2022 Act. They should help give the ICB a clear understanding of research across the area, regularly reporting on progress towards agreed aims. An executive lead can take responsibility for ensuring clear research ambitions and a research strategy are developed; oversight of organisational research portfolios, diversity in research, alignment to national priorities; promotion of research skills and the need for research skills training; and succession planning.

Senior leaders could engage, consult and be supported by representatives of each registered health and social care professional group when developing strategic plans, and for oversight of training, succession planning, and equality and inclusivity. They could use the capacity and capability of the research and development leads within provider organisations, although established lead roles across social care settings are rare so extra effort may be needed to garner social care research insight.

Research steering group, board or forum

Some CCGs had research steering groups and some of these have expanded with the widening remit of ICBs. ICSs that do not have a such a group should consider adopting a model similar to one in other ICSs where research is effectively embedded in ICS governance structures.

A dedicated steering research group, board and/or forum can:

  • provide dedicated time to plan, oversee and report on research
  • bring a range of representatives from research infrastructure organisations, patients and the public together with representation from across the ICS, to develop a common aim and objective
  • ensure board-level sight of research
  • take a cross-ICS approach to research, increasing participation and diversity in research, and reducing bureaucracy.

Example: A dedicated research and innovation subgroup

East and North Hertfordshire Health Care Partnership established a formal research and innovation subgroup to support its objectives to transform services, reduce health inequalities and improve patient health and wellbeing. This subgroup is dedicated to determining and supporting local research priorities and developing an innovation agenda. With effective patient and public involvement, it is working to ensure the local population has access to more research opportunities.

Bringing together the NIHR, academia, industry and local health and care services, the subgroup develops collaborative work plans that support the design, implementation and evaluation of local transformation needs, sharing resources, staff, expertise and facilities. Its work exemplifies a sustainable approach to partnership working and supports Hertfordshire and West Essex ICS’s developing strategy.

HWE ICS Partnership Board 14 September 2021

3.4 Understanding your research activity and working with local and national research infrastructure

Research in NHS and non-NHS settings across an ICS footprint will be supported by different organisations. In some areas networks or collaboratives already exist to bring these organisations together, but in others the links are not as well formed. ICBs would benefit from having a clear map of the research infrastructure and pre-existing local or national investment into research in their area.

It may be valuable to consider:

  • Who are the research leaders in your local health and care system, NIHR, higher education institutions, VCSE sector and businesses?
  • Are there any pre-existing local or regional research, researcher or research engagement networks?
  • What are the opportunities to inform, participate in, collaborate with or lead national and international research efforts in addition to local opportunities?

A list of organisations involved in research including NIHR-funded infrastructure and programmes is included in Annex 1 .

Much of the research undertaken in NHS and other health and care settings is funded though national calls and grants provided by funders such as NIHR, research charities , UK Research and Innovation (UKRI) , including the Medical Research Council (MRC ) and Economic and Social Research Council (ESRC) , and is aligned to national priorities. Other research may include national or international commercial or non-commercial clinical trials funders.

Partners within ICS systems can use NIHR research portfolio data to monitor and plan research activity; however, not all research is included within the NIHR’s portfolio, so this will not give a full picture of the research within the footprint. Mechanisms to map and monitor research more widely could be incorporated in ICB research strategies.

Some local needs may best be addressed through public health or social care research rather than research in primary, secondary or tertiary healthcare settings. Public health and social care research are described in Annex 2 .

Example: Mapping health and care research activity, expertise, interests and infrastructure

The Nottingham and Nottinghamshire Integrated Care System Research Partners Group meets bi-monthly and is chaired by the ICB Head of Research and Evidence. It brings together senior managers from the NHS providers, ICB, two local authorities, two universities and the NIHR CRN East Midlands, providing a forum for ICS-wide research discussions and the development of a system-wide collaborative approach to health and care research across the ICS. Among its aims, the group seeks to increase participation in research at both the organisational and population level, enable equity of access to research opportunities and generate impact on health and care pathways.

The group have mapped health and care research activity, expertise, interests and infrastructure in the constituent organisations. With this the ICS can see the research capabilities, strengths, expertise, and areas of synergy and opportunities for future collaboration that align to its needs and priorities, and also gaps for future development, recognising that organisations are at different stages of research development.

3.5 Understanding local needs

Universal NHS priorities will be reflected in local research needs, and each ICS footprint is likely to have its own specific local research needs. Joint strategic needs assessments (JSNAs) are undertaken jointly by local authorities and ICBs through health and wellbeing boards (HWBs) to identify current health and social care needs of local communities, where more information is needed to do so or to understand how best to address the need. People and communities should be directly involved in identifying local need, including by working with local charities, specific communities or groups who face inequalities in access to, experience of or outcomes from healthcare, eg to target health research at those areas and populations with greatest need.

ICPs are required to develop an integrated care strategy informed by JSNAs and the joint health and wellbeing strategy (JHWS). The integrated care strategy sets out how the assessed needs can be met through the exercise of the functions of the ICB, partner local authorities or NHS England, and is informed by research and practice-based evidence, as stated in the health and wellbeing guidance. In considering where such evidence is lacking, HWBs should identify in JSNAs those research needs that ICBs, local authorities and NHS England could meet through the exercise of their research functions.

Systems are encouraged to use their joint forward plan to develop a shared delivery plan for the Integrated Care Strategy and the JHWS that is supported by the whole system, including local authorities and VCSE partners. ICBs and trusts must also use their Joint Forward Plan to describe how the ICB will discharge its duty in respect of research.

The Explanatory Notes to the 2022 Act suggest how ICBs can discharge their duties around research. These include the articulating local research needs when assessing local needs and how they will be addressed when preparing strategies and plans, and encouraging partner organisations to play an active and collaborative role in pursuing these.

3.6 Supporting delivery of research

Once an ICS has a clear picture of its local research infrastructure it can consider how best to target and support research and the research workforce across its footprint and how research findings will be used. For this, the ICB should ensure that its approaches reflect national approaches to costing, contracting, approvals and information governance, and that they are also informed by learning from effective practices across equivalent ICBs.

As healthcare shifts into communities, ICSs should support the parallel shift in research by embedding research in health and care. Increasing access to research opportunities will give service users earlier access to new treatments, and faster research set up and delivery may provide the evidence needed to support improvements to local care sooner. Inclusive recruitment practices will be needed to ensure that all groups in society have the opportunity to help shape and take part in research, and benefit from research findings.

In developing its research strategy, an ICS has opportunities to reduce bureaucracy, and make research more efficient and effective across its own and with other ICS footprints, and across NHS and non-NHS boundaries, while meeting national regulatory guidance. ICBs will be expected to work with the HRA to co-develop, build on and implement strategies for further co-ordination and standardisation of study set-up and delivery processes. Any regional systems and processes that ICBs do establish must support consistent national practice in relation to the management and regulation of research, and should not duplicate them. The HRA will work with ICBs to address barriers to efficient and rapid study set-up, including model agreements, information governance and R&D office functions.

Other potential areas for streamlining and cross-organisational working include:

  • cross-ICS research proposals to identify research needs
  • research delivery – identifying how ICS-wide approaches could accelerate patient recruitment and deployment of research delivery staff
  • shared data architecture, including the NHS Secure Data Environment for Research Network and its subnational secure data environments (SDEs). Subnational SDEs cover multiple ICSs to achieve access to multimodal data at a scale of approximately 5 million citizens, and over time will achieve technical and governance interoperability
  • a greater focus on translation and implementation of research findings into health and care practice, supporting faster improvements
  • sharing access to and funding for knowledge and library services
  • shared processes and repositories for research assets.

The Explanatory Notes to the 2022 Act suggest that one way an ICB could discharge its research duty would be to have a dedicated research office or team supporting research.

3.7 Enabling cross-provider research

Health and care priorities can often only be addressed with complex, multiorganisational approaches and as such the research to inform these needs to span organisational boundaries. Organisational policies should promote cross-organisational research and dissemination of research findings, including through participation in collaborative research to address national priorities, joint staff posts, honorary contracts, and administratively easier movement of researchers between health and care organisations and other sector partners, including higher education, industry, charities and local authorities.

The HRA and ICS partners are developing national guidance to support cross-provider research.

The NIHR CRN can offer ICSs opportunities to participate in national and international research studies, including those the NIHR, industry and others commission.

3.8 Commercial research

Commercial contract research is research funded solely by industry, where NHS providers are contracted to carry out the research. Most of these research studies in the NHS are interventional clinical trials, such as the NHS-Galleri trial and Astra Zeneca’s COVID-19 vaccine development . Commercial research can give patients access to a wider range of research opportunities, earlier access to novel therapies and treatments, provide drugs free of charge to patients in trials, accelerate the development of new treatments and devices, generate income for providers, and fund NHS staff. It is vitally important for the benefit of patients, the NHS and the UK economy that we create an environment in the NHS that makes it easy and efficient for the NHS to undertake commercial research. This is particularly important when it comes to international commercial research, where companies can place their studies in a number of different countries and consideration of anticipated set up and recruitment times informs where they place trials.

Data gathered during some commercial research is specific to the study and is the property of the company, as is any Intellectual Property (IP) generated. In other cases, where the NHS contributes to the foreground IP – such as through the use of NHS data for research or where NHS expertise provides important contributions to a commercial product – it is important that the NHS shares in the value of IP generated as a consequence of its contributions.

The establishment of ICSs is an ideal opportunity for their creation of ambitions to enable, grow and benefit from commercial research. ICSs should explore how efficient commercial study set up and delivery could be streamlined across sites within their footprint, and should set ambitions around commercial research.

3.9 Involving patients, service users, carers and the public in research

In developing a research strategy ICSs should set out their approach to diverse public and patient involvement (PPI) in relation to research.

Areas where working with people and communities could add value in the context of research include:

  • identification of local research needs, including through JSNAs and JHWSs
  • designing research proposals in partnership with local or national experts
  • raising awareness of research opportunities and recruitment of participants
  • developing research outcome reports and identification of how and when participants will be able to access these
  • consideration of how members of the public can access the outputs from publicly-funded research
  • how volunteers should be involved and what they should be paid.

The UK Standards for Public Involvement sets out the core components of good public involvement. A guide outlining good practice in engaging underserved communities around research is available from NHS England. Resources about good practice around PPI in designing and delivering research, including around incentivisation , are also available from the HRA and NIHR .

It will be useful to link into established community involvement approaches. NIHR infrastructure organisations may have established networks of expert PPI representatives, and ICSs have extensive VCSE Alliances. A co-ordinated community engagement approach across health and care delivery and research will reduce the risk of overburdening communities with organisations wanting to work with them, and will support the identification of under-served communities.

3.10 Ensuring anyone can participate in research

Making research more visible within communities and increasing the public’s understanding of research can ensure greater diversity in research participation. Research findings will then be more generalisable to a broader range of groups or communities, or can be targeted and specific to relevant communities.

ICSs should seek mechanisms to ensure that opportunities to take part in research are available to all. They should consider encouraging patients and members of the public to register on NIHR Be Part of Research (a national registry where people can express their interest in being contacted about research that is relevant to them), widely disseminate research opportunities and make provision for inclusive access for communities to take part in research. Decentralised or virtual trials are remote access trials recruited to and delivered using electronic tools, making it easier for people to participate in some studies without needing to visit a recruiting hospital or attend appointments in person. ICBs should consider ways in which research delivery can increase access to research opportunities for people within their area. ICBs should also advise the public how they can access research outputs.

NIHR and UK Research and Innovation provide resources that help organisations address issues of equality, diversity and inclusion in research settings.

Example: RELIEVE-IBS decentralised trial

In 2020, Newcastle researchers launched RELIEVE-IBS, one of the first interventional decentralised clinical studies in the UK to trial Enterosgel, a new treatment for irritable bowel syndrome with diarrhoea (IBS-D). Decentralised trials are remote access trials that use electronic tools for trial recruitment and delivery, without the patient needing to visit a recruiting hospital site, which could be miles from their homes – a convenient option for patients with IBS-D. By running the trial remotely, researchers could reach beyond the small proportion of those with this condition who attend specialist clinics, as well as save resource for the sponsor.

Not only did this trial embrace technological developments to deliver research, but it empowered more patients to become involved regardless of where they lived. With in-depth patient input, the research team were able to shape the recruitment approach to be highly accessible to participants and were offered feedback on how to refine the trial design by the sponsors. The resulting patient-centric design ensured a good recruitment response when the trial opened.

NIHR (2020) Virtual trial recruits 67% faster led by NIHR Patient Recruitment Centre in Newcastle in collaboration with Enteromed

NIHR (2021) Pushing virtual boundaries to improve patient engagement and accessibility

NIHR (2022) RELIEVE IBS-D trial case study

3.11 Health data in research

Health data generated through care of service users in the NHS can fuel a revolution in the research and development of new diagnostics and treatments, maximising the potential to improve service user outcomes and experiences, support diversity in research, and minimise health inequalities through research. To do this, researchers need access to high quality and timely data to generate insights. The public expect data to be used legally and efficiently to conduct and support research.

National commitments around data for research can be found in Data saves lives: reshaping health and social care with data . This strategy shows how data will be used to bring benefits to all parts of health and social care. To achieve this vision, the NHS will be making a strategic move away from a system of data dissemination to one of data access when making NHS health and social care data available for research and analysis. This will be facilitated by the implementation of secure data environments (SDEs).

SDEs are data storage and access platforms with features that enable organisations to have greater control and oversight over their data. SDEs allow approved users to view and analyse data without it having to leave the environment. The SDE policy guidelines provide a clear signal to the sector that SDEs will become the default way of accessing NHS data for research.

This change is supported by major investments in digital infrastructure through the Data for Research & Development Programme, which is funding the development of national and subnational SDEs. The subnational SDEs will cover the entirety of England and individual platforms will cover several ICS.

ICBs should seek ways to promote and enable the use of these rich data sources for research and include them in their research strategy.

3.12 Using evidence for planning, commissioning and improving health and care

Evidence-based commissioning has advantages for the commissioner, workforce and service users, as it can:

  • lead to innovation in service design and delivery
  • enhance the quality of health and care provision
  • reduce clinical variation between locations and providers
  • improve equity of access to services
  • improve patient and population outcomes.

As part of the commissioning process, commissioners are expected to use evidence-based clinical policies, as per the Roadmap for integrating specialised services within integrated care systems . Knowledge and library services can help source and interpret evidence.

The Provider Selection Regime will reflect the research duties of the 2022 Act and should be referred to when commissioning provider services, once it has been published.

NHS knowledge and library services provide access to evidence and support for knowledge management; they train people in searching for, handling and publishing information. The Knowledge for Healthcare strategy encourages and equips NHS knowledge and library services to support NHS organisations with the translation of knowledge for the spread and adoption of research and innovation. To fulfil their obligations under the 2022 Act, ICBs could commit to active knowledge translation.

Evidence for commissioning information is available from a number of sources:

  • NHS Library and Knowledge Hub
  • Health Libraries and Information Services Directory
  • NICE guidance
  • NIHR evidence
  • NHS evidence works toolkit
  • Academy of Medical Royal Colleges: Evidence-based Intervention
  • A million decisions

The infographic for the role of research and evidence in commissioning also provides sources for evidence-based commissioning.

Example: Evidence mobilisation, knowledge sharing and improving outcomes

The STEMClub (Sustaining Transformation by Evidence Mobilisation) is a network in the North East and North Cumbria that brings together local policy and decision-makers with NHS knowledge and library specialists to facilitate evidence-based decision-making. The input of knowledge specialists ensures timely access to published research and provides knowledge management expertise to shape how soft intelligence is translated into knowledge assets.

As members within the STEMClub network, knowledge and library specialists are providing ongoing detailed evidence reviews and information management expertise to facilitate system-wide working , eg:

  • North East North Cumbria Frailty Framework
  • North East and North Cumbria Maternity Clinical Network
  • a review of optimal patient transfer times in the North East and North Cumbria
  • regular evidence summaries for the ICS Mental Health Evidence and Evaluation subgroup.

3.13 The health and care workforce and research

Staff involved in research have greater job satisfaction and research active trusts have lower staff turnover [3] . Clinical academic roles [7] , having research colleagues within services [8] and taking students on research placements [6] are felt to foster an increase in knowledge and skills across the wider staff workforce. The General Medical Council (GMC) and the Royal College of Physicians (RCP) and NIHR have issued position statements and recommendations around research, with additional signatories including UKRI, UKRD, the Academy of Medical Royal Colleges and the Royal College of Surgeons of England. Learning resources, including programmes for ongoing professional development of the research delivery workforce, are available through NIHR Learn.

In developing a research strategy ICSs could ensure that, as part of their people function and approach to workforce planning :

  • Staff roles in leading, delivering or facilitating research and in supervising those developing research skills are recognised, supported and enabled across all staff groups and health and care settings as part of a positive research culture.
  • The value of evidence is recognised, and education and training around research are facilitated. Opportunities to develop research careers or in overseeing the development of other researchers are enabled; this may include having protected time, inclusion in job plans and joint appointments across health and care providers and academic institutions.
  • Ensuring that there is capacity and systems that support research through services like imaging, pathology and pharmacy, as well as finance and human resources.
  • Individual organisations do not always have the necessary skills or services to support effective research and its impact, such as IP management, methodological expertise, regulatory compliance, statistical analysis, knowledge mobilisation expertise, genomics expertise, health informatics and data analytics. Mechanisms are needed to ensure that these can readily and rapidly be accessed across other health and care organisations, including from local authorities and other non-NHS care providers.

A UK Clinical Research Workforce Strategy is under development. ICSs should update their approaches to their research workforce once DHSC publishes this in 2023/24.

Example: Investing in the research workforce – developing capacity for chief investigators

Across the West Midlands NIHR CRN, an investment of approximately £750,000 to develop capacity for chief investigators returned additional research grant income of over £18 million in three years. This was achieved primarily by increasing the programme activity for consultants in areas where chief investigators were underrepresented.

The funding was provided through a competitive process and co-supported by the local NIHR CRN, with several local trusts jointly funded these scholars.

Kirk J, Willcocks J, Boyle P, Brocklehurst P, Morris K, Kearney R, et al (2022) Developing chief investigators within the NHS: the West Midlands clinical trials scholars programme. Clin Med 22(2): 149–52.

Kirk J, Reynolds F, Adey E, Boazman M, Brookes M, Brocklehurst P (2022) Developing paediatric chief investigators within the NHS: the Clinical Trials Scholars programme . Arch Dis Child Educ Pract Published online first: 22 February 2022. doi: 10.1136/archdischild-2021-322186

4. References

  • Varnai P, Rentel M, Dave A, De Scalzi M, Timmerman W, Rosemberg-Mantes C, Simmonds P, Technopolis Group (2017) The impact of collaboration: The value of UK medical research to EU science and health .
  • Boaz A, Hanney S, Jones T, Soper B (2015) Does the engagement of clinicians and organisations in research improve healthcare performance: a three-stage review. BMJ Open 5: e009415. doi:10.1136/ bmjopen-2015-009415 .
  • Rees MR, Bracewell M (2019) Academic factors in medical recruitment: evidence to support improvements in medical recruitment and retention by improving the academic content in medical posts. Postgrad Med J 95(1124): 323-327. doi:10.1136/postgradmedj-2019-136501 .
  • Ozdemir BA, Karthikesalingham A, Singha S, Poloniecki JD, Hinchliffe RJ, Thompson MM, et al (2015) Research activity and the association with mortality. PLoS ONE 10(2): .
  • Hunn A (2017) Survey of the general public: attitudes towards health research . Health Research Authority.
  • Angus RL, Hattingh HL, Weir KA (2022) Experiences of hospital allied health professionals in collaborative student research projects: a qualitative study. BMC Health Services Research 22(1). Available at: .
  • Newington L, Wells M, Adonis A, Bolten L, Bolton Saghdaoui L, Coffey M, et al (2021) A qualitative systematic review and thematic synthesis exploring the impacts of clinical academic activity by healthcare professionals outside medicine. BMC Health Serv Res 21(1). Available at: .
  • Wenke RJ, Hickman I, Hulcombe J, Phillips R, Mickan S (2017) Allied health research positions: A qualitative evaluation of their impact. Health Res Policy Syst 15(6). Available at:

Annex 1 – Organisations that may be involved in regional or local research

  • Clinical Research Networks (CRNs) , which will be retendered and renamed regional research delivery networks (RRDNs) from April 2024
  • Applied Research Collaborations (ARCs)
  • Biomedical Research Centres (BRCs)
  • Experimental Cancer Medicine Centres (ECMCs) , jointly funded with Cancer Research UK
  • Research Design Services (RDSs) and Clinical Trials Units (CTUs) which will be replaced by the NIHR Research Support Service from 1 October 2023
  • Patient Recruitment Centres (PRCs)
  • MedTech and In vitro diagnostic Co-operatives (MICs) , which will be replaced with HealthTech research centres from April 2024
  • School of Public Health Research, School of Primary Care Research and School of Social Care Research
  • Health Determinants Research Collaborations (HDRCs)
  • Clinical Research Facilities (CRFs)
  • Patient Safety Research Collaborations (PSRCs)
  • Translational Research Collaborations (TRCs)
  • Academic Health Science Centres (AHSCs)
  • university teaching hospitals and all trusts that deliver research activity
  • primary care organisations, including GP practices, that deliver research activity
  • higher education institutions (HEIs)
  • local authorities
  • social care partners
  • Local Government Association
  • local and national charities that fund, collaborate in or support participation in research
  • research and development offices in providers or CSUs, including primary care providers and ambulance, community and mental health trusts, and those in the VCSE sector
  • UKRD members
  • NHS subnational secure data environments for research
  • NHS R&D Forum
  • NHS Genomic Medicines Service Research Collaborative
  • NHS Knowledge and Library Services
  • Academic Health Science Networks (AHSNs) are often well linked with research organisations and infrastructure as part of their roles in development, adoption and spread of innovation.

Annex 2 – Public health and social care research

Public health research investigates issues that impact at a population rather than an individual level. This can be done within the NHS with system-level studies, such as secondary prevention of cardiovascular disease and examining the impact on health inequalities of changes to the NHS resource allocation formula, and outside the NHS for the wider determinants of health such as air quality, transport systems and housing. There is a substantial body of public health evidence for the clinical and cost effectiveness of prevention, health protection, health service redesign and addressing health inequalities.

Social care research aims to improve the lives of children and adults who need to draw on personal or practical care and support, and family members or other unpaid carers. It can include research around the introduction, use and impact of technologies, and changing social care interventions, policies and practice. Social care research also examines issues pertaining to the safeguarding of adults and children and workforce, commissioning of services, and questions about organisational and professional practice, including decision-making, training and the quality of care.

Publication reference: PR1662

  • Skip to main content
  • Keyboard shortcuts for audio player

Shots - Health News

Your Health

  • Treatments & Tests
  • Health Inc.
  • Public Health

Why writing by hand beats typing for thinking and learning

Jonathan Lambert

A close-up of a woman's hand writing in a notebook.

If you're like many digitally savvy Americans, it has likely been a while since you've spent much time writing by hand.

The laborious process of tracing out our thoughts, letter by letter, on the page is becoming a relic of the past in our screen-dominated world, where text messages and thumb-typed grocery lists have replaced handwritten letters and sticky notes. Electronic keyboards offer obvious efficiency benefits that have undoubtedly boosted our productivity — imagine having to write all your emails longhand.

To keep up, many schools are introducing computers as early as preschool, meaning some kids may learn the basics of typing before writing by hand.

But giving up this slower, more tactile way of expressing ourselves may come at a significant cost, according to a growing body of research that's uncovering the surprising cognitive benefits of taking pen to paper, or even stylus to iPad — for both children and adults.

Is this some kind of joke? A school facing shortages starts teaching standup comedy

In kids, studies show that tracing out ABCs, as opposed to typing them, leads to better and longer-lasting recognition and understanding of letters. Writing by hand also improves memory and recall of words, laying down the foundations of literacy and learning. In adults, taking notes by hand during a lecture, instead of typing, can lead to better conceptual understanding of material.

"There's actually some very important things going on during the embodied experience of writing by hand," says Ramesh Balasubramaniam , a neuroscientist at the University of California, Merced. "It has important cognitive benefits."

While those benefits have long been recognized by some (for instance, many authors, including Jennifer Egan and Neil Gaiman , draft their stories by hand to stoke creativity), scientists have only recently started investigating why writing by hand has these effects.

A slew of recent brain imaging research suggests handwriting's power stems from the relative complexity of the process and how it forces different brain systems to work together to reproduce the shapes of letters in our heads onto the page.

Your brain on handwriting

Both handwriting and typing involve moving our hands and fingers to create words on a page. But handwriting, it turns out, requires a lot more fine-tuned coordination between the motor and visual systems. This seems to more deeply engage the brain in ways that support learning.

Feeling Artsy? Here's How Making Art Helps Your Brain

Shots - Health News

Feeling artsy here's how making art helps your brain.

"Handwriting is probably among the most complex motor skills that the brain is capable of," says Marieke Longcamp , a cognitive neuroscientist at Aix-Marseille Université.

Gripping a pen nimbly enough to write is a complicated task, as it requires your brain to continuously monitor the pressure that each finger exerts on the pen. Then, your motor system has to delicately modify that pressure to re-create each letter of the words in your head on the page.

"Your fingers have to each do something different to produce a recognizable letter," says Sophia Vinci-Booher , an educational neuroscientist at Vanderbilt University. Adding to the complexity, your visual system must continuously process that letter as it's formed. With each stroke, your brain compares the unfolding script with mental models of the letters and words, making adjustments to fingers in real time to create the letters' shapes, says Vinci-Booher.

That's not true for typing.

To type "tap" your fingers don't have to trace out the form of the letters — they just make three relatively simple and uniform movements. In comparison, it takes a lot more brainpower, as well as cross-talk between brain areas, to write than type.

Recent brain imaging studies bolster this idea. A study published in January found that when students write by hand, brain areas involved in motor and visual information processing " sync up " with areas crucial to memory formation, firing at frequencies associated with learning.

"We don't see that [synchronized activity] in typewriting at all," says Audrey van der Meer , a psychologist and study co-author at the Norwegian University of Science and Technology. She suggests that writing by hand is a neurobiologically richer process and that this richness may confer some cognitive benefits.

Other experts agree. "There seems to be something fundamental about engaging your body to produce these shapes," says Robert Wiley , a cognitive psychologist at the University of North Carolina, Greensboro. "It lets you make associations between your body and what you're seeing and hearing," he says, which might give the mind more footholds for accessing a given concept or idea.

Those extra footholds are especially important for learning in kids, but they may give adults a leg up too. Wiley and others worry that ditching handwriting for typing could have serious consequences for how we all learn and think.

What might be lost as handwriting wanes

The clearest consequence of screens and keyboards replacing pen and paper might be on kids' ability to learn the building blocks of literacy — letters.

"Letter recognition in early childhood is actually one of the best predictors of later reading and math attainment," says Vinci-Booher. Her work suggests the process of learning to write letters by hand is crucial for learning to read them.

"When kids write letters, they're just messy," she says. As kids practice writing "A," each iteration is different, and that variability helps solidify their conceptual understanding of the letter.

Research suggests kids learn to recognize letters better when seeing variable handwritten examples, compared with uniform typed examples.

This helps develop areas of the brain used during reading in older children and adults, Vinci-Booher found.

"This could be one of the ways that early experiences actually translate to long-term life outcomes," she says. "These visually demanding, fine motor actions bake in neural communication patterns that are really important for learning later on."

Ditching handwriting instruction could mean that those skills don't get developed as well, which could impair kids' ability to learn down the road.

"If young children are not receiving any handwriting training, which is very good brain stimulation, then their brains simply won't reach their full potential," says van der Meer. "It's scary to think of the potential consequences."

Many states are trying to avoid these risks by mandating cursive instruction. This year, California started requiring elementary school students to learn cursive , and similar bills are moving through state legislatures in several states, including Indiana, Kentucky, South Carolina and Wisconsin. (So far, evidence suggests that it's the writing by hand that matters, not whether it's print or cursive.)

Slowing down and processing information

For adults, one of the main benefits of writing by hand is that it simply forces us to slow down.

During a meeting or lecture, it's possible to type what you're hearing verbatim. But often, "you're not actually processing that information — you're just typing in the blind," says van der Meer. "If you take notes by hand, you can't write everything down," she says.

The relative slowness of the medium forces you to process the information, writing key words or phrases and using drawing or arrows to work through ideas, she says. "You make the information your own," she says, which helps it stick in the brain.

Such connections and integration are still possible when typing, but they need to be made more intentionally. And sometimes, efficiency wins out. "When you're writing a long essay, it's obviously much more practical to use a keyboard," says van der Meer.

Still, given our long history of using our hands to mark meaning in the world, some scientists worry about the more diffuse consequences of offloading our thinking to computers.

"We're foisting a lot of our knowledge, extending our cognition, to other devices, so it's only natural that we've started using these other agents to do our writing for us," says Balasubramaniam.

It's possible that this might free up our minds to do other kinds of hard thinking, he says. Or we might be sacrificing a fundamental process that's crucial for the kinds of immersive cognitive experiences that enable us to learn and think at our full potential.

Balasubramaniam stresses, however, that we don't have to ditch digital tools to harness the power of handwriting. So far, research suggests that scribbling with a stylus on a screen activates the same brain pathways as etching ink on paper. It's the movement that counts, he says, not its final form.

Jonathan Lambert is a Washington, D.C.-based freelance journalist who covers science, health and policy.

  • handwriting

Stanford University

Along with Stanford news and stories, show me:

  • Student information
  • Faculty/Staff information

We want to provide announcements, events, leadership messages and resources that are relevant to you. Your selection is stored in a browser cookie which you can remove at any time using “Clear all personalization” below.

Over recent months, tech companies have been laying workers off by the thousands. It is estimated that in 2022 alone, over 120,000 people have been dismissed from their job at some of the biggest players in tech – Meta , Amazon , Netflix , and soon Google – and smaller firms and starts ups as well. Announcements of cuts keep coming.

Recent layoffs across the tech sector are an example of “social contagion” – companies are laying off workers because everyone is doing it, says Stanford business Professor Jeffrey Pfeffer. (Image credit: Courtesy Jeffrey Pfeffer)

What explains why so many companies are laying large numbers of their workforce off? The answer is simple: copycat behavior, according to Jeffrey Pfeffer, a professor at the Stanford Graduate School of Business .

Here, Stanford News talks to Pfeffer about how the workforce reductions that are happening across the tech industry are a result mostly of “social contagion”: Behavior spreads through a network as companies almost mindlessly copy what others are doing. When a few firms fire staff, others will probably follow suit. Most problematic, it’s a behavior that kills people : For example, research has shown that layoffs can increase the odds of suicide by two times or more .

Moreover, layoffs don’t work to improve company performance,  Pfeffer adds. Academic studies have shown that time and time again, workplace reductions don’t do much for paring costs. Severance packages cost money, layoffs increase unemployment insurance rates, and cuts reduce workplace morale and productivity as remaining employees are left wondering, “Could I be fired too?”

For over four decades, Pfeffer, the Thomas D. Dee II Professor of Organizational Behavior, has studied hiring and firing practices in companies across the world. He’s met with business leaders at some of the country’s top companies and their employees to learn what makes – and doesn’t make – effective, evidence-based management. His recent book Dying for a Paycheck: How Modern Management Harms Employee Health and Company Performance–And What We Can Do About It (Harper Business, 2018) looks at how management practices, including layoffs, are hurting, and in some cases, killing workers.  

This interview has been edited for length and clarity.

Why are so many tech companies laying people off right now?

The tech industry layoffs are basically an instance of social contagion, in which companies imitate what others are doing. If you look for reasons for why companies do layoffs, the reason is that everybody else is doing it. Layoffs are the result of imitative behavior and are not particularly evidence-based.

I’ve had people say to me that they know layoffs are harmful to company well-being, let alone the well-being of employees, and don’t accomplish much, but everybody is doing layoffs and their board is asking why they aren’t doing layoffs also.

Do you think layoffs in tech are some indication of a tech bubble bursting or the company preparing for a recession?

Could there be a tech recession? Yes. Was there a bubble in valuations? Absolutely. Did Meta overhire? Probably. But is that why they are laying people off? Of course not. Meta has plenty of money. These companies are all making money. They are doing it because other companies are doing it.

What are some myths or misunderstandings about layoffs?

Layoffs often do not cut costs, as there are many instances of laid-off employees being hired back as contractors, with companies paying the contracting firm. Layoffs often do not increase stock prices, in part because layoffs can signal that a company is having difficulty. Layoffs do not increase productivity. Layoffs do not solve what is often the underlying problem, which is often an ineffective strategy, a loss of market share, or too little revenue. Layoffs are basically a bad decision.

Companies sometimes lay off people that they have just recruited – oftentimes with paid recruitment bonuses. When the economy turns back in the next 12, 14, or 18 months, they will go back to the market and compete with the same companies to hire talent. They are basically buying labor at a high price and selling low. Not the best decision.

People don’t pay attention to the evidence against layoffs. The evidence is pretty extensive, some of it is reviewed in the book I wrote on human resource management, The Human Equation: Building Profits by Putting People First. If companies paid attention to the evidence, they could get some competitive leverage because they would actually be basing their decisions on science.

You’ve written about the negative health effects of layoffs. Can you talk about some of the research on this topic by you and others?

Layoffs kill people, literally . They kill people in a number of ways. Layoffs increase the odds of suicide by two and a half times. This is also true outside of the United States, even in countries with better social safety nets than the U.S., like New Zealand.

Layoffs increase mortality by 15-20% over the following 20 years.

There are also health and attitudinal consequences for managers who are laying people off as well as for the employees who remain . Not surprisingly, layoffs increase people’s stress . Stress, like many attitudes and emotions, is contagious. Depression is contagious , and layoffs increase stress and depression, which are bad for health.

Unhealthy stress leads to a variety of behaviors such as smoking and drinking more , drug taking , and overeating . Stress is also related to addiction , and layoffs of course increase stress.

What was your reaction to some of the recent headlines of mass layoffs, like Meta laying off 11,000 employees?

I am concerned. Most of my recent research is focused on the effect of the workplace on human health and how economic insecurity is bad for people. This is on the heels of the COVID pandemic and the social isolation resulting from that, which was also bad for people.

We ought to place a higher priority on human life.

If layoffs are contagious within an industry, could it then spread across industries, leading to other sectors cutting staff?

Of course, it already has. Layoffs are contagious across industries and within industries. The logic driving this, which doesn’t sound like very sensible logic because it’s not, is people say, “Everybody else is doing it, why aren’t we?”

Retailers are pre-emptively laying off staff, even as final demand remains uncertain. Apparently, many organizations will trade off a worse customer experience for reduced staffing costs, not taking into account the well-established finding that is typically much more expensive to attract new customers than it is to keep existing ones happy.

Are there past examples of contagious layoffs like the one we are seeing now, and what lessons were learned?

After the Sept. 11, 2001, terrorist attacks, every airline except Southwest did layoffs. By the end of that year, Southwest, which did not do any layoffs, gained market share. A.G. Lafley, who was the former CEO of Procter and Gamble, said the best time to gain ground on your competition is when they are in retreat – when they are cutting their services, when they are cutting their product innovation because they have laid people off. James Goodnight, the CEO of the software company SAS Institute, has also never done layoffs – he actually hired during the last two recessions because he said it’s the best time to pick up talent.

Any advice to workers who may have been laid off?

My advice to a worker who has been laid off is when they find a job in a company where they say people are their most important asset, they actually check to be sure that the company behaves consistently with that espoused value when times are tough.

If layoffs don’t work, what is a better solution for companies that want to mitigate the problems they believe layoffs will address?

One thing that Lincoln Electric, which is a famous manufacturer of arc welding equipment, did well is instead of laying off 10% of their workforce, they had everybody take a 10% wage cut except for senior management, which took a larger cut. So instead of giving 100% of the pain to 10% of the people, they give 100% of the people 10% of the pain.

Companies could use economic stringency as an opportunity, as Goodnight at the SAS Institute did in the 2008 recession and in the 2000 tech recession. He used the downturn to upgrade workforce skills as competitors eliminated jobs, thereby putting talent on the street. He actually hired during the 2000 recession and saw it as an opportunity to gain ground on the competition and gain market share when everybody was cutting jobs and stopped innovating. And it is [an opportunity]. Social media is not going away. Artificial intelligence, statistical software, and web services industries – none of these things are going to disappear.

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

  • Advanced Search
  • Journal List
  • Community Eye Health
  • v.35(117); 2022
  • PMC10061254

Why does research matter?

Victor h hu.

Assistant Clinical Professor: International Centre for Eye Health, London School of Hygiene & Tropical Medicine and Consultant Ophthalmologist: Mid Cheshire NHS Hospitals, UK.

A working knowledge of research – both how it is done, and how it can be used – is important for everyone involved in direct patient care and the planning & delivery of eye programmes.

An external file that holds a picture, illustration, etc.
Object name is jceh_35_117_001_f01.jpg

A research coordinator collecting data from a health extension worker. ethiopia

The mention of ‘research’ can be off-putting and may seem irrelevant in the busy environment of a clinic or hospital. However, research is central to all aspects of eye care delivery – both inside and outside the clinic.

Whether we are health workers, public health practitioners, managers, policy makers, or editors – all of us ‘stand on the shoulders of giants’: we rely on the research done by others before us. This can be as simple – and profound – as hand washing between patients; a habit that only became common practice in the 1870s, following the work of the Hungarian physician Ignaz Semmelweis and Scottish surgeon Joseph Lister. Or it can be as complex as making a diagnosis of glaucoma and knowing what treatment to give. All current eye care practice is based on research. Clinical, operational (eye care delivery) and public health practice will continue to be profoundly shaped by new research developments.

What is research?

In its simplest form, research is about investigating the world around us to increase our knowledge, so we can work out how to do things better.

In health care, we use a scientific approach to carry out research; there is a set way of doing things that ensures research is done in a logical way, and that results are published widely, so that other people can scrutinise what has been done. This gives us confidence that the results will be useful in everyday practice.

It is important to critically evaluate research and research findings, including checking that research has been carried out in the proper way, and whether the conclusions that have been made are reasonable and justified. One of the ways in which the scientific community ensures the quality of research is through the process of peer review. Before research papers are accepted for publication in a scientific journal, they are reviewed by other researchers (peer reviewed) to check the quality of the research and the validity of the results and conclusions. Even so, the quality of published research can vary.

This is why systematic reviews and meta-analyses are so valuable: they answer important questions by identifying, evaluating, and summarising good quality evidence from a range of published research papers. Often, systematic reviews conclude that there is not enough evidence to answer a question with absolute certainty, or to produce an answer that will be applicable in different countries or health care settings. This is useful, as it gives researchers guidance about where more research is needed (see article on page 13).

But this can be a challenge for clinicians – how can we make good decisions in the absence of definitive evidence? Clinical experience is very important, but where possible this should be informed by good research – see page 6 for practical tips.

Health care practitioners and managers can also use guidance from professional bodies such as the World Health Organization. The article on page 8 explains the process by which guidelines are developed and shows why we can rely on them.

In conclusion, research is fundamental to the everyday practice of health care professionals, including eye care workers. Research allows us to find out new things and to provide better care for patients. There are many different types of research that can be carried out and these can vary enormously. It is important to ask the right question, as this will determine the type of research that is done (see page 5).

All of us can participate in research: it starts with asking questions and then going to find out the answers. The article on page 10 offers practical suggestions for carrying out small-scale research that is relevant and useful to eye care.

Types of health research

Basic science research, such as in molecular genetics or cell biology, fills the gaps in our understanding of disease mechanisms (pathogenesis).

Clinical research addresses how diseases in individuals can present and be diagnosed, and how a condition progresses and can be managed.

Epidemiological research , which is at the population level (as opposed to the individual level), answers questions about the number of people in the population who have a condition, what factors (called exposures) are causing the condition, and how it can be treated or prevented at the population level.

Going beyond epidemiology, there is also operational and health systems research , which focuses on how best to deliver health interventions, clinical and rehabilitation services, or behaviour change initiatives.

Other types of research , which are also important for public health, include health economics, social science, and statistical modelling.

Finally, systematic literature reviews can be very useful, as they identify and summarise the available evidence on a specific topic.

By Clare Gilbert and GVS Murthy

Examples of research questions and how they have been answered

Can povidone iodine prevent endophthalmitis.

In many eye departments, cataract surgery is a frequently preformed operation. One of the most serious complications is infection within the eye (endophthalmitis) which can lead to loss of vision. Several well conducted randomised controlled clinical trials have shown that instilling 0.5% aqueous povidone iodine eye drops, an antiseptic agent, before surgery reduces the risk of this devastating infection, with the first trial undertaken in 1991. 1

What is the best treatment for primary open-angle glaucoma?

Chronic glaucoma can be a very difficult condition to manage, particularly when patients often only present to eye departments once they have already had significant vision loss. Eye drops which lower intraocular pressure are often prescribed; however, patients may not use the eyedrops because they are expensive, can be difficult to instil, and do not improve their vision. Surgery is an option, but patients can be reluctant to undergo surgery on their only good eye, and there can be postoperative complications. Laser treatment is another option. In a recent study in Tanzania, patients were randomly allocated to Timolol 0.5% eye drops or a form of laser called Selective Laser Trabeculoplasty (SLT). 2 After one year, SLT was found to be superior to drops for high-pressure glaucoma.

Why don't older adults in England have their eyes examined?

Focus group discussions among older adults in England revealed that, despite most participants being eligible for state-funded check-ups, wearing spectacles was associated with the appearance of being frail. They were also afraid of appearing to ‘fail’ tests, and had concerns about the cost of spectacles. 3

How cost effective is a diabetic retinopathy screening programme?

An economic evaluation in South Africa compared alternative interventions. Screening using non-mydriatic retinal photographs taken by a technician supervised by an ophthalmic nurse and read by a general medical officer was cost-effective and the savings made allowed the government to fund disability grants for people who went blind. 4


Stephen Gichuhi and Nyawira Mwangi contributed to preliminary work on this article.

Is college worth it? The answer for half of Americans is striking.

why is it important for research to be recent

A college degree has often been sold as the key to a higher-quality, affluent life. But a new survey from the Pew Research Center suggests Americans have mixed views about that narrative – and data shows people without degrees have seen their earnings increase in the last decade.

Just 1 in 4 U.S. adults said it was extremely or very important to have a four-year degree if you want a well-paying job in the current economy. Forty percent of respondents said it wasn’t too important or important at all. 

Mirroring those trends, just 22% of adults said the cost of getting a bachelor’s is worth it even if it means taking out student loans. Nearly half said the cost is only worth it when students don’t have to go into debt. 

Graphics explain: How are college costs adding up these days and how much has tuition risen?

Given trends in the labor and economy – combined with skyrocketing tuition and student debt levels – the lackluster confidence among Americans isn’t surprising. For several decades until about 2014, for example, the earnings for young men without a degree trended downward. But the past decade “has marked a turning point,” according to the Pew analysis.

Workforce participation for these young men has stabilized and their earnings have risen. The share of them living in poverty has also fallen significantly. In 2011, for example, 17% of young men with just a high school diploma were living in poverty; in 2023, that rate dropped to 12%. Young women’s outcomes also improved in recent years.

The changing circumstances help explain why people's mindsets about the value of college have shifted. Roughly half of Americans, according to the Pew report, say a four-year degree is less important today than it was in the past to secure a well-paying job. A smaller percentage – about a third – say it’s more important now. 

The skepticism is more pronounced among conservative Americans than people who identify as Democrats or somewhat Democrat. Most Republicans (57%) said it was less important to have a four-year degree. Still, Americans from both parties are more likely to say the importance of a college degree has declined than to say it's increased.

The findings come as the Biden administration works to forgive certain borrowers’ federal student loan debt, which now totals more than $1.6 trillion. On top of barriers to covering tuition, college life has been altered this year by an uptick in culture war tensions on campus, from bans on diversity, equity and inclusion programming to student protests prompted by the Israel-Hamas war. These challenges have fueled debates about whether college is worth it.

Still, the research shows that earnings for degree holders have also trended upward. The income gaps between college graduates and those with just high school degrees or incomplete credentials have persisted. 

And while employment prospects for young men without a degree improved in the past decade, their median annual earnings remain below their 1973 adjusted levels.

Financial aid crisis: How FAFSA 'fixes' have turned College Decision Day into chaos

  • Community and Engagement
  • Honors and Awards
  • Give Now 

Why Is It Important for Students to Understand How Scientific Decisions are Made? ‘If You Don’t Understand How Scientists Decide What Makes One Claim More Believable, Then It’s Actually Very Hard to Understand Science,’ Says STEM Education Department Head William Sandoval

why is it important for research to be recent

For William Sandoval, head of the Department of STEM Education in the NC State College of Education, when preparing K-12 students to engage with real-world science, developing the skills to become career scientists is not nearly as important as helping them to engage with the science that will occur all around them in their everyday lives. 

To help facilitate this, Sandoval’s research has focused on how kids understand scientific argumentation and how scientists make the case for why people should believe the theories they’ve developed based on causal claims and evidence. 

The concept is known as epistemic cognition, or thinking about how people know what they know. 

“If we want kids to understand how science works, they have to understand the standards that scientists use to evaluate competing claims. If you don’t understand how scientists decide what makes one claim more believable, then it’s actually very hard to understand science,” Sandoval said. “Every person who gets a science education through high school is eventually going to encounter some scientific issue in their adult lives that they didn’t learn about in school because science moves really fast. If you never thought about the way the scientific community makes choices, it’s very hard to make sense of it all as an individual citizen.” 

How Do Kids Typically Think About Science?

Broadly speaking, research shows that, oftentimes, kids’ initially find many scientific concepts to be implausible because so many of the causal agents behind these theories are invisible and insensible. 

“You can’t feel your genes; you can’t feel an atom, so there are these causal agents in science that are very far away from our sensory experience, which makes them hard to understand,” Sandoval said. 

Despite this, research shows that children, even from a young age, believe that it’s much better to have evidence for a claim than to not have evidence. Therefore, teachers can draw on this by helping students understand the evidence behind the science they are learning through engagement in research, data collection or experimentation in the classroom. 

How Can Teachers Help Students Understand How Scientific Decisions Are Made?

At its core, Sandoval said, science is all about evaluating competing claims about how the world works. 

One of the best ways to help students understand how to do this is to have them engage in scientific argumentation. Sandoval shared the following steps to help students in this endeavor: 

  • Identify topics within the curriculum that can elicit disagreement ; this can vary from asking elementary students how plants get water to how they inherit genetic traits from their parents: “Identify for everybody when there is disagreement and try to clarify what the nature of the disagreement is. Then, discuss with students how they are going to decide [which claim is valid] and what it will take for everyone to agree.” 
  • Give students an opportunity to come up with their own evidence-based claims: “Our current national standards want kids to be doing investigations of various kinds, so these are really good opportunities for kids to engage in these epistemic considerations about how we’re going to decide what we’re going to believe about this thing that we’re studying. Because, if they’re getting data themselves, they can argue about not just what the data show but how you got the data, whether you did it in a reasonable way or not and whether you’ve interpreted the data in a reasonable way.”
  • Have students with opposing claims engage in discussion: “We found that when this is public, that works better and part of the reason is that this holds kids accountable to each other. They also have to be held accountable to standards of evidence. So, that’s a really important role for the teacher to play is to kind of push kids to talk about what evidence they have for this claim over that claim.”
  • Come to a consensus: “Talk about the standards or the criteria that you’d use to resolve your disagreement. It’s not enough that I believe one thing and someone else believes another thing. You’ve got to push us to agree, so pushing to consensus seems to be the thing that really helps kids work through their understanding of the criteria.”

For teachers who want to engage their students in this type of scientific thinking, Sandoval recommends using free resources provided by the NGSX professional learning system along with the inquiryHub ,a research-practice partnership that develops materials, tools and processes to promote STEM learning. 

  • Research and Impact
  • Ask The Expert
  • homepage-news
  • resources for educators
  • science education
  • STEM Ed Research
  • William Sandoval

More From College of Education News

why is it important for research to be recent

Dean Paola Sztajn Completes Leadership for North Carolina Program 

Alumni Distinguished Graduate Professor Alyssa Rockenbach

Alumni Distinguished Graduate Professor of Higher Education Alyssa Rockenbach to Explore Interpartisian Friendships on College Campuses through National Center for Free Speech and Civic Engagement Fellowship 

Catherine Hartman

Catherine Hartman Named Assistant Professor of Community College Leadership 

Numbers, Facts and Trends Shaping Your World

Read our research on:

Full Topic List

Regions & Countries

  • Publications
  • Our Methods
  • Short Reads
  • Tools & Resources

Read Our Research On:

Is College Worth It?

As economic outcomes for young adults with and without degrees have improved, americans hold mixed views on the value of college, table of contents.

  • Labor force trends and economic outcomes for young adults
  • Economic outcomes for young men
  • Economic outcomes for young women
  • Wealth trends for households headed by a young adult
  • The importance of a four-year college degree
  • Getting a high-paying job without a college degree
  • Do Americans think their education prepared them for the workplace?
  • Is college worth the cost?
  • Acknowledgments
  • The American Trends Panel survey methodology
  • Current Population Survey methodology
  • Survey of Consumer Finances methodology

why is it important for research to be recent

Pew Research Center conducted this study to better understand public views on the importance of a four-year college degree. The study also explores key trends in the economic outcomes of young adults among those who have and have not completed a four-year college degree.

The analysis in this report is based on three data sources. The labor force, earnings, hours, household income and poverty characteristics come from the U.S. Census Bureau’s Annual Social and Economic Supplement of the Current Population Survey. The findings on net worth are based on the Federal Reserve’s Survey of Consumer Finances.

The data on public views on the value of a college degree was collected as part of a Center survey of 5,203 U.S. adults conducted Nov. 27 to Dec. 3, 2023. Everyone who took part in the survey is a member of Pew Research Center’s American Trends Panel (ATP), an online survey panel that is recruited through national, random sampling of residential addresses. Address-based sampling ensures that nearly all U.S. adults have a chance of selection. The survey is weighted to be representative of the U.S. adult population by gender, race, ethnicity, partisan affiliation, education and other categories. Read more about the ATP’s methodology .

Here are the questions used for this report , along with responses, and the survey’s methodology .

Young adults refers to Americans ages 25 to 34.

Noncollege adults include those who have some college education as well as those who graduated from high school but did not attend college. Adults who have not completed high school are not included in the analysis of noncollege adults. About 6% of young adults have not completed high school. Trends in some labor market outcomes for those who have not finished high school are impacted by changes in the foreign-born share of the U.S. population. The Census data used in this analysis did not collect information on nativity before 1994.

Some college includes those with an associate degree and those who attended college but did not obtain a degree.

The some college or less population refers to adults who have some college education, those with a high school diploma only and those who did not graduate high school.

A full-time, full-year worker works at least 50 weeks per year and usually 35 hours a week or more.

The labor force includes all who are employed and those who are unemployed but looking for work.

The labor force participation rate is the share of a population that is in the labor force.

Young adults living independently refers to those who are not living in the home of either of their parents.

Household income is the sum of incomes received by all members of the household ages 15 and older. Income is the sum of earnings from work, capital income such as interest and dividends, rental income, retirement income, and transfer income (such as government assistance) before payments for such things as personal income taxes, Social Security and Medicare taxes, union dues, etc. Non-cash transfers such as food stamps, health benefits, subsidized housing and energy assistance are not included. As household income is pretax, it does not include stimulus payments or tax credits for earned income and children/dependent care.

Net worth, or wealth, is the difference between the value of what a household owns (assets) and what it owes (debts).

All references to party affiliation include those who lean toward that party. Republicans include those who identify as Republicans and those who say they lean toward the Republican Party. Democrats include those who identify as Democrats and those who say they lean toward the Democratic Party.

At a time when many Americans are questioning the value of a four-year college degree, economic outcomes for young adults without a degree are improving.

Pie chart shows Only 22% of U.S. adults say the cost of college is worth it even if someone has to take out loans

After decades of falling wages, young U.S. workers (ages 25 to 34) without a bachelor’s degree have seen their earnings increase over the past 10 years. Their overall wealth has gone up too, and fewer are living in poverty today.

Things have also improved for young college graduates over this period. As a result, the gap in earnings between young adults with and without a college degree has not narrowed.

The public has mixed views on the importance of having a college degree, and many have doubts about whether the cost is worth it, according to a new Pew Research Center survey.

  • Only one-in-four U.S. adults say it’s extremely or very important to have a four-year college degree in order to get a well-paying job in today’s economy. About a third (35%) say a college degree is somewhat important, while 40% say it’s not too or not at all important.
  • Roughly half (49%) say it’s less important to have a four-year college degree today in order to get a well-paying job than it was 20 years ago; 32% say it’s more important, and 17% say it’s about as important as it was 20 years ago.
  • Only 22% say the cost of getting a four-year college degree today is worth it even if someone has to take out loans. Some 47% say the cost is worth it only if someone doesn’t have to take out loans. And 29% say the cost is not worth it.

These findings come amid rising tuition costs and mounting student debt . Views on the cost of college differ by Americans’ level of education. But even among four-year college graduates, only about a third (32%) say college is worth the cost even if someone has to take out loans – though they are more likely than those without a degree to say this.

Four-year college graduates (58%) are much more likely than those without a college degree (26%) to say their education was extremely or very useful in giving them the skills and knowledge they needed to get a well-paying job. (This finding excludes the 9% of respondents who said this question did not apply to them.)

Chart shows 4 in 10 Americans say a college degree is not too or not at all important in order to get a well-paying job

Views on the importance of college differ widely by partisanship. Republicans and Republican-leaning independents are more likely than Democrats and Democratic leaners to say:

  • It’s not too or not at all important to have a four-year college degree in order to get a well-paying job (50% of Republicans vs. 30% of Democrats)
  • A college degree is less important now than it was 20 years ago (57% vs. 43%)
  • It’s extremely or very likely someone without a four-year college degree can get a well-paying job (42% vs. 26%)

At the same time that the public is expressing doubts about the value of college, a new Center analysis of government data finds young adults without a college degree are doing better on some key measures than they have in recent years.

A narrow majority of workers ages 25 to 34 do not have a four-year college degree (54% in 2023). Earnings for these young workers mostly trended downward from the mid-1970s until roughly a decade ago.

Outcomes have been especially poor for young men without a college degree. Other research has shown that this group saw falling labor force participation and sagging earnings starting in the early 1970s , but the last decade has marked a turning point.

This analysis looks at young men and young women separately because of their different experiences in the labor force.

Trends for young men

  • Labor force participation: The share of young men without a college degree who were working or looking for work dropped steadily from 1970 until about 2014. Our new analysis suggests things have stabilized somewhat for this group over the past decade. Meanwhile, labor force participation among young men with a four-year degree has remained mostly flat.
  • Full-time, full-year employment: The share of employed young men without a college degree who are working full time and year-round has varied somewhat over the years – trending downward during recessions. It’s risen significantly since the Great Recession of 2007-09, with the exception of a sharp dip in 2021 due to the COVID-19 pandemic. For employed young men with a college degree, the share working full time, full year has remained more stable over the years.

Chart shows Earnings of young men without a college degree have increased over the past 10 years

  • Median annual earnings: Since 2014, earnings have risen for young men with some college education and for those whose highest attainment is a high school diploma. Even so, earnings for these groups remain below where they were in the early 1970s. Earnings for young men with a bachelor’s degree have also trended up, for the most part, over the past 10 years.
  • Poverty: Among young men without a college degree who are living independently from their parents, the share in poverty has fallen significantly over the last decade. For example, 12% of young men with a high school diploma were living in poverty in 2023, down from a peak of 17% in 2011. The share of young men with a four-year college degree who are in poverty has also fallen and remains below that of noncollege young men.

Trends for young women

  • Labor force participation: The shares of young women with and without a college degree in the labor force grew steadily from 1970 to about 1990. Among those without a college degree, the share fell after 2000, and the drop-off was especially sharp for young women with a high school diploma. Since 2014, labor force participation for both groups of young women has increased.
  • Full-time, full-year employment: The shares of employed young women working full time and year-round, regardless of their educational attainment, have steadily increased over the decades. There was a decline during and after the Great Recession and again (briefly) in 2021 due to the pandemic. Today, the shares of women working full time, full year are the highest they’ve ever been across education levels.

Chart shows Earnings of young women without a college degree have trended up in the past decade

  • Median annual earnings: Median earnings for young women without a college degree were relatively flat from 1970 until about a decade ago. These women did not experience the steady decline in earnings that noncollege young men did over this period. By contrast, earnings have grown over the decades for young women with a college degree. In the past 10 years, earnings for women both with and without a college degree have risen.
  • Poverty: As is the case for young men without a college degree, the share of noncollege young women living in poverty has fallen substantially over the past decade. In 2014, 31% of women with a high school diploma who lived independently from their parents were in poverty. By 2023, that share had fallen to 21%. Young women with a college degree remain much less likely to be in poverty than their counterparts with less education.

Sign up for our weekly newsletter

Fresh data delivery Saturday mornings

Sign up for The Briefing

Weekly updates on the world of news & information

  • Business & Workplace
  • Economic Conditions
  • Higher Education
  • Income & Wages
  • Recessions & Recoveries
  • Student Loans

Half of Latinas Say Hispanic Women’s Situation Has Improved in the Past Decade and Expect More Gains

From businesses and banks to colleges and churches: americans’ views of u.s. institutions, fewer young men are in college, especially at 4-year schools, key facts about u.s. latinos with graduate degrees, private, selective colleges are most likely to use race, ethnicity as a factor in admissions decisions, most popular, report materials.

1615 L St. NW, Suite 800 Washington, DC 20036 USA (+1) 202-419-4300 | Main (+1) 202-857-8562 | Fax (+1) 202-419-4372 |  Media Inquiries

Research Topics

  • Email Newsletters

ABOUT PEW RESEARCH CENTER  Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of  The Pew Charitable Trusts .

© 2024 Pew Research Center


  1. Why is Research Important for Undergraduate Students?

    why is it important for research to be recent

  2. 15 Reasons Why Research Is Important

    why is it important for research to be recent

  3. 7 Reasons Why Research Is Important

    why is it important for research to be recent

  4. Reasons Why Research Is Important

    why is it important for research to be recent

  5. What Is the Importance of Research? 5 Reasons Why Research is Critical

    why is it important for research to be recent

  6. Importance of Research

    why is it important for research to be recent


  1. Research Profile 1: Why is it so important?

  2. Why important is feeling

  3. HCA Middle East Symposium 2023 Day 2, Part 2

  4. Can you REALLY manifest your Specific Person using Law of Assumption? This needs to be said

  5. Why do research proposals get rejected?

  6. Why Important Expert Advice, Invest, Financial Advisor, Investment, mutual fund, SIP #trending #shER


  1. How recent is recent for good referencing?

    All being said, do we exactly know how old a research article can be before it gets the label of not being recent i.e., an old article not good for citing. There is consensus among scientists and researchers that articles less than five years old are recent publications. However, it may vary from discipline to discipline.

  2. When is it appropriate to describe research as "recent"?

    For example, if you cite sources from the current century as well as 1930s, then a paper from 2010 should be considered recent, but not one from 1950. If, on the other hand, your temporal range of references is rather narrow, say, recent 20 years, then you should refer to as "recent" for sources that are from approximately last 4-5 years.

  3. Importance of staying up-to-date in Research topics

    As Research entails testing, verifying, and rejecting hypotheses regularly, keeping up with recent publications will assist you in defining building blocks for your study. Guidance & Confirming that your Research is focused on a new topic. One of the key responsibilities for a doctorate adviser, department head, or field expertise is to advise ...

  4. The Literature Review: A Foundation for High-Quality Medical Education

    Purpose and Importance of the Literature Review. An understanding of the current literature is critical for all phases of a research study. Lingard 9 recently invoked the "journal-as-conversation" metaphor as a way of understanding how one's research fits into the larger medical education conversation. As she described it: "Imagine yourself joining a conversation at a social event.

  5. The Importance of Referencing

    The stamp of a good research worker is attention to detail at all levels of his/her research. Attention to detail cultivates good habits and the detail required in referencing and preparing a bibliography focuses attention on the whole research procedure. ... If the most recent reference is more than five years or so, this may indicate that a ...

  6. Peer Review in Scientific Publications: Benefits, Critiques, & A

    Peer review is a mutual responsibility among fellow scientists, and scientists are expected, as part of the academic community, to take part in peer review. If one is to expect others to review their work, they should commit to reviewing the work of others as well, and put effort into it. 2) Be pleasant. If the paper is of low quality, suggest ...

  7. Writing a literature review

    Writing a literature review requires a range of skills to gather, sort, evaluate and summarise peer-reviewed published data into a relevant and informative unbiased narrative. Digital access to research papers, academic texts, review articles, reference databases and public data sets are all sources of information that are available to enrich ...

  8. Evidence Based Research Series

    1. Introduction. This article is part of a series describing evidence-based research—the use of prior research in a systematic and transparent way to inform a new study so that it is answering questions that matter in a valid, efficient, and accessible manner [1].By prior research, we mean original studies (also called primary studies), but even when planning a new systematic review ...

  9. Keeping up to date: An academic researcher's information journey

    In the academic context, keeping up to date is critical for the development of research projects, writing articles, engaging in debates, and even hiring the right researcher (Jamali & Nicholas, 2008). To keep up to date, researchers use various information resources, many of which have become available online through search engines and web alerts.

  10. Evidence-Based Research Series-Paper 1: What Evidence-Based Research is

    In this first article of a three-article series, we introduce the evidence-based research approach. Evidence-based research is the use of prior research in a systematic and transparent way to inform a new study so that it is answering questions that matter in a valid, efficient, and accessible manner.

  11. 3 Reasons to stay on top of latest research in your field

    Here are some important reasons why you should be up-to-date with all the latest publications in your field: 1. To identify a novel research topic and future research opportunities. Understanding the current status of knowledge on a particular topic, identifying gaps, and zeroing in on a question that is relevant and answerable are the initial ...

  12. Research Impact: The What, Why, When and How

    The following section will seek to provide some answers to why research impact has become so important in the past decade. ... However, in recent years, research impact has begun to eclipse outputs with regard to importance, at least in the UK. Steven Hill, Director of Research Policy for Research England, has argued that the justification for ...

  13. 1 Chapter 1: The Importance of Research Methods and Becoming an

    What research is and why it is important to be an informed consumer of research. The sources of knowledge development and problems with each. How research methods can dispel myths about crime and the criminal justice system. The steps in the research process. How research has impacted criminal justice operations. Introduction

  14. Why Nursing Research Matters

    Abstract. Increasingly, nursing research is considered essential to the achievement of high-quality patient care and outcomes. In this month's Magnet® Perspectives column, we examine the origins of nursing research, its role in creating the Magnet Recognition Program®, and why a culture of clinical inquiry matters for nurses.

  15. 2.1 Why Is Research Important?

    Discuss how scientific research guides public policy. Appreciate how scientific research can be important in making personal decisions. Scientific research is a critical tool for successfully navigating our complex world. Without it, we would be forced to rely solely on intuition, other people's authority, and blind luck.

  16. What Is Research, and Why Do People Do It?

    Abstractspiepr Abs1. Every day people do research as they gather information to learn about something of interest. In the scientific world, however, research means something different than simply gathering information. Scientific research is characterized by its careful planning and observing, by its relentless efforts to understand and explain ...

  17. When is the evidence too old?

    A few weeks ago, when submitting an abstract to a nursing conference, I was suddenly faced with a dilemma about age. Not my own age, but the age of evidence I was using to support my work. One key element of the submission criteria was to provide five research citations to support the abstract, and all citations were to be less than ten years old.

  18. For stronger evidence, look for multiple studies

    Scientists often redo their own study multiple times to make sure their methods are sound. Scientists also redo each other's studies to see if they get the same results. Sometimes scientists repeat a study using the same data, and sometimes they use new data. When multiple studies come to similar conclusions, we can become more confident that ...

  19. Explaining research performance: investigating the importance of

    In this article, we study the motivation and performance of researchers. More specifically, we investigate what motivates researchers across different research fields and countries and how this motivation influences their research performance. The basis for our study is a large-N survey of economists, cardiologists, and physicists in Denmark, Norway, Sweden, the Netherlands, and the UK. The ...

  20. 5 Identifying the evidence: literature searching and evidence ...

    The Summarized Research in Information Retrieval for HTA (SuRe Info) ... Quality assuring the literature search is an important step in developing guideline recommendations. Studies have shown that errors do occur. For each search (including economic searches), the initial MEDLINE search strategy is quality assured by a second information ...

  21. Reliability and validity: Importance in Medical Research

    MeSH terms. Reliability and validity are among the most important and fundamental domains in the assessment of any measuring methodology for data-collection in a good research. Validity is about what an instrument measures and how well it does so, whereas reliability concerns the truthfulness in the data obtain ….

  22. The value of field research in academia

    From anthropology to zoology, immersion within communities, cultural settings, and study systems is integral to research and learning (1, 2). Fieldwork, the direct observation and collection of data in natural settings, enables researchers to collect relevant data, connect theory to complex social and ecological systems, and apply research findings to the real world (1). However, in addition ...

  23. Using Old Data: When Is It Appropriate?

    Some archival data sets offer important insights but are not updated to the recent past. For example, the U.S. census is updated every 10 years. A scholar asking a research question requiring census data for an answer might be forced to use data up to 10 years old. A second example is the Pew religious data, which are updated every 7 years.

  24. NHS England » Maximising the benefits of research: Guidance for

    1.2 Why research is important. The UK is a world leader for research and invention in healthcare, with around 25% of the world's top 100 prescription medicines being discovered and developed in the UK (The impact of collaboration: The value of UK medical research to EU science and health). Research in the health and care system is important ...

  25. As schools reconsider cursive, research homes in on handwriting's ...

    A slew of recent brain imaging research suggests handwriting's power stems from the relative complexity of the process and how it forces different brain systems to work together to reproduce the ...

  26. What explains recent tech layoffs, and why should we be worried?

    Over recent months, tech companies have been laying workers off by the thousands. It is estimated that in 2022 alone, over 120,000 people have been dismissed from their job at some of the biggest ...

  27. Why does research matter?

    A working knowledge of research - both how it is done, and how it can be used - is important for everyone involved in direct patient care and the planning & delivery of eye programmes. The mention of 'research' can be off-putting and may seem irrelevant in the busy environment of a clinic or hospital. However, research is central to all ...

  28. Is college worth it? Americans are split about cost and debt.

    A college degree has often been sold as the key to a higher-quality, affluent life. But a new survey from the Pew Research Center suggests Americans have mixed views about that narrative - and ...

  29. Why Is It Important for Students to Understand How Scientific Decisions

    To help facilitate this, Sandoval's research has focused on how kids understand scientific argumentation and how scientists make the case for why people should believe the theories they've developed based on causal claims and evidence. The concept is known as epistemic cognition, or thinking about how people know what they know.

  30. Is a College Degree Worth It in 2024?

    The public has mixed views on the importance of having a college degree, and many have doubts about whether the cost is worth it, according to a new Pew Research Center survey. Only one-in-four U.S. adults say it's extremely or very important to have a four-year college degree in order to get a well-paying job in today's economy.