explore the scientific problem solving process pdf

How it works

For Business

Join Mind Tools

Article • 5 min read

Using the Scientific Method to Solve Problems

How the scientific method and reasoning can help simplify processes and solve problems.

By the Mind Tools Content Team

The processes of problem-solving and decision-making can be complicated and drawn out. In this article we look at how the scientific method, along with deductive and inductive reasoning can help simplify these processes.

‘It is a capital mistake to theorize before one has information. Insensibly one begins to twist facts to suit our theories, instead of theories to suit facts.’ Sherlock Holmes

The Scientific Method

The scientific method is a process used to explore observations and answer questions. Originally used by scientists looking to prove new theories, its use has spread into many other areas, including that of problem-solving and decision-making.

The scientific method is designed to eliminate the influences of bias, prejudice and personal beliefs when testing a hypothesis or theory. It has developed alongside science itself, with origins going back to the 13th century. The scientific method is generally described as a series of steps.

• observations/theory
• explanation/conclusion

The first step is to develop a theory about the particular area of interest. A theory, in the context of logic or problem-solving, is a conjecture or speculation about something that is not necessarily fact, often based on a series of observations.

Once a theory has been devised, it can be questioned and refined into more specific hypotheses that can be tested. The hypotheses are potential explanations for the theory.

The testing, and subsequent analysis, of these hypotheses will eventually lead to a conclus ion which can prove or disprove the original theory.

Applying the Scientific Method to Problem-Solving

How can the scientific method be used to solve a problem, such as the color printer is not working?

1. Use observations to develop a theory.

In order to solve the problem, it must first be clear what the problem is. Observations made about the problem should be used to develop a theory. In this particular problem the theory might be that the color printer has run out of ink. This theory is developed as the result of observing the increasingly faded output from the printer.

2. Form a hypothesis.

Note down all the possible reasons for the problem. In this situation they might include:

• The printer is set up as the default printer for all 40 people in the department and so is used more frequently than necessary.
• There has been increased usage of the printer due to non-work related printing.
• In an attempt to reduce costs, poor quality ink cartridges with limited amounts of ink in them have been purchased.
• The printer is faulty.

All these possible reasons are hypotheses.

3. Test the hypothesis.

Once as many hypotheses (or reasons) as possible have been thought of, then each one can be tested to discern if it is the cause of the problem. An appropriate test needs to be devised for each hypothesis. For example, it is fairly quick to ask everyone to check the default settings of the printer on each PC, or to check if the cartridge supplier has changed.

4. Analyze the test results.

Once all the hypotheses have been tested, the results can be analyzed. The type and depth of analysis will be dependant on each individual problem, and the tests appropriate to it. In many cases the analysis will be a very quick thought process. In others, where considerable information has been collated, a more structured approach, such as the use of graphs, tables or spreadsheets, may be required.

5. Draw a conclusion.

Based on the results of the tests, a conclusion can then be drawn about exactly what is causing the problem. The appropriate remedial action can then be taken, such as asking everyone to amend their default print settings, or changing the cartridge supplier.

Inductive and Deductive Reasoning

The scientific method involves the use of two basic types of reasoning, inductive and deductive.

Inductive reasoning makes a conclusion based on a set of empirical results. Empirical results are the product of the collection of evidence from observations. For example:

‘Every time it rains the pavement gets wet, therefore rain must be water’.

There has been no scientific determination in the hypothesis that rain is water, it is purely based on observation. The formation of a hypothesis in this manner is sometimes referred to as an educated guess. An educated guess, whilst not based on hard facts, must still be plausible, and consistent with what we already know, in order to present a reasonable argument.

Deductive reasoning can be thought of most simply in terms of ‘If A and B, then C’. For example:

• if the window is above the desk, and
• the desk is above the floor, then
• the window must be above the floor

It works by building on a series of conclusions, which results in one final answer.

Social Sciences and the Scientific Method

The scientific method can be used to address any situation or problem where a theory can be developed. Although more often associated with natural sciences, it can also be used to develop theories in social sciences (such as psychology, sociology and linguistics), using both quantitative and qualitative methods.

Quantitative information is information that can be measured, and tends to focus on numbers and frequencies. Typically quantitative information might be gathered by experiments, questionnaires or psychometric tests. Qualitative information, on the other hand, is based on information describing meaning, such as human behavior, and the reasons behind it. Qualitative information is gathered by way of interviews and case studies, which are possibly not as statistically accurate as quantitative methods, but provide a more in-depth and rich description.

The resultant information can then be used to prove, or disprove, a hypothesis. Using a mix of quantitative and qualitative information is more likely to produce a rounded result based on the factual, quantitative information enriched and backed up by actual experience and qualitative information.

In terms of problem-solving or decision-making, for example, the qualitative information is that gained by looking at the ‘how’ and ‘why’ , whereas quantitative information would come from the ‘where’, ‘what’ and ‘when’.

It may seem easy to come up with a brilliant idea, or to suspect what the cause of a problem may be. However things can get more complicated when the idea needs to be evaluated, or when there may be more than one potential cause of a problem. In these situations, the use of the scientific method, and its associated reasoning, can help the user come to a decision, or reach a solution, secure in the knowledge that all options have been considered.

Join Mind Tools and get access to exclusive content.

This resource is only available to Mind Tools members.

Already a member? Please Login here

Get 20% off your first year of Mind Tools

Our on-demand e-learning resources let you learn at your own pace, fitting seamlessly into your busy workday. Join today and save with our limited time offer!

Sign-up to our newsletter

Subscribing to the Mind Tools newsletter will keep you up-to-date with our latest updates and newest resources.

Subscribe now

Business Skills

Personal Development

Leadership and Management

Most Popular

Newest Releases

Team Management Skills

5 Phrases That Kill Collaboration

Mind Tools Store

About Mind Tools Content

Discover something new today

How do i manage a hybrid team.

Adjusting your management style to a hybrid world

The Life Career Rainbow

Finding a Work-Life Balance That Suits You

How Emotionally Intelligent Are You?

Boosting Your People Skills

Self-Assessment

What's Your Leadership Style?

Learn About the Strengths and Weaknesses of the Way You Like to Lead

Recommended for you

New year resolutions.

Planning for a Year of Achievement

Business Operations and Process Management

Strategy Tools

Customer Service

Business Ethics and Values

Handling Information and Data

Project Management

Knowledge Management

Self-Development and Goal Setting

Time Management

Presentation Skills

Learning Skills

Career Skills

Communication Skills

Negotiation, Persuasion and Influence

Working With Others

Difficult Conversations

Creativity Tools

Self-Management

Work-Life Balance

Stress Management and Wellbeing

Coaching and Mentoring

Change Management

Team Management

Managing Conflict

Delegation and Empowerment

Performance Management

Leadership Skills

Developing Your Team

Talent Management

Problem Solving

Decision Making

Pain Points

• school Campus Bookshelves
• menu_book Bookshelves
• perm_media Learning Objects
• login Login
• how_to_reg Request Instructor Account
• hub Instructor Commons
• Download Page (PDF)
• Download Full Book (PDF)
• Periodic Table
• Physics Constants
• Scientific Calculator
• Reference & Cite
• Tools expand_more
• Readability

selected template will load here

This action is not available.

1.12: Scientific Problem Solving

• Last updated
• Save as PDF
• Page ID 52325

How can we use problem solving in our everyday routines?

One day you wake up and realize your clock radio did not turn on to get you out of bed. You are puzzled, so you decide to find out what happened. You list three possible explanations:

• There was a power failure and your radio cannot turn on.
• Your little sister turned it off as a joke.
• You did not set the alarm last night.

Upon investigation, you find that the clock is on, so there is no power failure. Your little sister was spending the night with a friend and could not have turned the alarm off. You notice that the alarm is not set—your forgetfulness made you late. You have used the scientific method to answer a question.

Scientific Problem Solving

Humans have always wondered about the world around them. One of the questions of interest was (and still is): what is this world made of? Chemistry has been defined in various ways as the study of matter. What matter consists of has been a source of debate over the centuries. One of the key areas for this debate in the Western world was Greek philosophy.

The basic approach of the Greek philosophers was to discuss and debate the questions they had about the world. There was no gathering of information to speak of, just talking. As a result, several ideas about matter were put forth, but never resolved. The first philosopher to carry out the gathering of data was Aristotle (384-322 B.C.). He recorded many observations on the weather, on plant and animal life and behavior, on physical motions, and a number of other topics. Aristotle could probably be considered the first "real" scientist, because he made systematic observations of nature and tried to understand what he was seeing.

Inductive and Deductive Reasoning

Two approaches to logical thinking developed over the centuries. These two methods are inductive reasoning and deductive reasoning . Inductive reasoning involves getting a collection of specific examples and drawing a general conclusion from them. Deductive reasoning takes a general principle and then draws a specific conclusion from the general concept. Both are used in the development of scientific ideas.

Inductive reasoning first involves the collection of data: "If I add sodium metal to water, I observe a very violent reaction. Every time I repeat the process, I see the same thing happen." A general conclusion is drawn from these observations: the addition of sodium to water results in a violent reaction.

In deductive reasoning, a specific prediction is made based on a general principle. One general principle is that acids turn blue litmus paper red. Using the deductive reasoning process, one might predict: "If I have a bottle of liquid labeled 'acid', I expect the litmus paper to turn red when I immerse it in the liquid."

The Idea of the Experiment

Inductive reasoning is at the heart of what is now called the " scientific method ." In European culture, this approach was developed mainly by Francis Bacon (1561-1626), a British scholar. He advocated the use of inductive reasoning in every area of life, not just science. The scientific method, as developed by Bacon and others, involves several steps:

• Ask a question - identify the problem to be considered.
• Make observations - gather data that pertains to the question.
• Propose an explanation (a hypothesis) for the observations.
• Make new observations to test the hypothesis further.

Note that this should not be considered a "cookbook" for scientific research. Scientists do not sit down with their daily "to do" list and write down these steps. The steps may not necessarily be followed in order. But this does provide a general idea of how scientific research is usually done.

When a hypothesis is confirmed repeatedly, it eventually becomes a theory—a general principle that is offered to explain natural phenomena. Note a key word— explain , or  explanation . A theory offers a description of why something happens. A law, on the other hand, is a statement that is always true, but offers no explanation as to why. The law of gravity says a rock will fall when dropped, but does not explain why (gravitational theory is very complex and incomplete at present). The kinetic molecular theory of gases, on the other hand, states what happens when a gas is heated in a closed container (the pressure increases), but also explains why (the motions of the gas molecules are increased due to the change in temperature). Theories do not get "promoted" to laws, because laws do not answer the "why" question.

• The early Greek philosophers spent their time talking about nature, but did little or no actual exploration or investigation.
• Inductive reasoning - to develop a general conclusion from a collection of observations.
• Deductive reasoning - to make a specific statement based on a general principle.
• Scientific method - a process of observation, developing a hypothesis, and testing that hypothesis.
• What was the basic shortcoming of the Greek philosophers approach to studying the material world?
• How did Aristotle improve the approach?
• Define “inductive reasoning” and give an example.
• Define “deductive reasoning” and give an example.
• What is the difference between a hypothesis and a theory?
• What is the difference between a theory and a law?

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Identifying problems and solutions in scientific text

Kevin heffernan.

Department of Computer Science and Technology, University of Cambridge, 15 JJ Thomson Avenue, Cambridge, CB3 0FD UK

Simone Teufel

Research is often described as a problem-solving activity, and as a result, descriptions of problems and solutions are an essential part of the scientific discourse used to describe research activity. We present an automatic classifier that, given a phrase that may or may not be a description of a scientific problem or a solution, makes a binary decision about problemhood and solutionhood of that phrase. We recast the problem as a supervised machine learning problem, define a set of 15 features correlated with the target categories and use several machine learning algorithms on this task. We also create our own corpus of 2000 positive and negative examples of problems and solutions. We find that we can distinguish problems from non-problems with an accuracy of 82.3%, and solutions from non-solutions with an accuracy of 79.7%. Our three most helpful features for the task are syntactic information (POS tags), document and word embeddings.

Introduction

Problem solving is generally regarded as the most important cognitive activity in everyday and professional contexts (Jonassen 2000 ). Many studies on formalising the cognitive process behind problem-solving exist, for instance (Chandrasekaran 1983 ). Jordan ( 1980 ) argues that we all share knowledge of the thought/action problem-solution process involved in real life, and so our writings will often reflect this order. There is general agreement amongst theorists that state that the nature of the research process can be viewed as a problem-solving activity (Strübing 2007 ; Van Dijk 1980 ; Hutchins 1977 ; Grimes 1975 ).

One of the best-documented problem-solving patterns was established by Winter ( 1968 ). Winter analysed thousands of examples of technical texts, and noted that these texts can largely be described in terms of a four-part pattern consisting of Situation, Problem, Solution and Evaluation. This is very similar to the pattern described by Van Dijk ( 1980 ), which consists of Introduction-Theory, Problem-Experiment-Comment and Conclusion. The difference is that in Winter’s view, a solution only becomes a solution after it has been evaluated positively. Hoey changes Winter’s pattern by introducing the concept of Response in place of Solution (Hoey 2001 ). This seems to describe the situation in science better, where evaluation is mandatory for research solutions to be accepted by the community. In Hoey’s pattern, the Situation (which is generally treated as optional) provides background information; the Problem describes an issue which requires attention; the Response provides a way to deal with the issue, and the Evaluation assesses how effective the response is.

An example of this pattern in the context of the Goldilocks story can be seen in Fig.  1 . In this text, there is a preamble providing the setting of the story (i.e. Goldilocks is lost in the woods), which is called the Situation in Hoey’s system. A Problem in encountered when Goldilocks becomes hungry. Her first Response is to try the porridge in big bear’s bowl, but she gives this a negative Evaluation (“too hot!”) and so the pattern returns to the Problem. This continues in a cyclic fashion until the Problem is finally resolved by Goldilocks giving a particular Response a positive Evaluation of baby bear’s porridge (“it’s just right”).

Example of problem-solving pattern when applied to the Goldilocks story.

Reproduced with permission from Hoey ( 2001 )

It would be attractive to detect problem and solution statements automatically in text. This holds true both from a theoretical and a practical viewpoint. Theoretically, we know that sentiment detection is related to problem-solving activity, because of the perception that “bad” situations are transformed into “better” ones via problem-solving. The exact mechanism of how this can be detected would advance the state of the art in text understanding. In terms of linguistic realisation, problem and solution statements come in many variants and reformulations, often in the form of positive or negated statements about the conditions, results and causes of problem–solution pairs. Detecting and interpreting those would give us a reasonably objective manner to test a system’s understanding capacity. Practically, being able to detect any mention of a problem is a first step towards detecting a paper’s specific research goal. Being able to do this has been a goal for scientific information retrieval for some time, and if successful, it would improve the effectiveness of scientific search immensely. Detecting problem and solution statements of papers would also enable us to compare similar papers and eventually even lead to automatic generation of review articles in a field.

There has been some computational effort on the task of identifying problem-solving patterns in text. However, most of the prior work has not gone beyond the usage of keyword analysis and some simple contextual examination of the pattern. Flowerdew ( 2008 ) presents a corpus-based analysis of lexio-grammatical patterns for problem and solution clauses using articles from professional and student reports. Problem and solution keywords were used to search their corpora, and each occurrence was analysed to determine grammatical usage of the keyword. More interestingly, the causal category associated with each keyword in their context was also analysed. For example, Reason–Result or Means-Purpose were common causal categories found to be associated with problem keywords.

The goal of the work by Scott ( 2001 ) was to determine words which are semantically similar to problem and solution, and to determine how these words are used to signal problem-solution patterns. However, their corpus-based analysis used articles from the Guardian newspaper. Since the domain of newspaper text is very different from that of scientific text, we decided not to consider those keywords associated with problem-solving patterns for use in our work.

Instead of a keyword-based approach, Charles ( 2011 ) used discourse markers to examine how the problem-solution pattern was signalled in text. In particular, they examined how adverbials associated with a result such as “thus, therefore, then, hence” are used to signal a problem-solving pattern.

Problem solving also has been studied in the framework of discourse theories such as Rhetorical Structure Theory (Mann and Thompson 1988 ) and Argumentative Zoning (Teufel et al. 2000 ). Problem- and solutionhood constitute two of the original 23 relations in RST (Mann and Thompson 1988 ). While we concentrate solely on this aspect, RST is a general theory of discourse structure which covers many intentional and informational relations. The relationship to Argumentative Zoning is more complicated. The status of certain statements as problem or solutions is one important dimension in the definitions of AZ categories. AZ additionally models dimensions other than problem-solution hood (such as who a scientific idea belongs to, or which intention the authors might have had in stating a particular negative or positive statement). When forming categories, AZ combines aspects of these dimensions, and “flattens” them out into only 7 categories. In AZ it is crucial who it is that experiences the problems or contributes a solution. For instance, the definition of category “CONTRAST” includes statements that some research runs into problems, but only if that research is previous work (i.e., not if it is the work contributed in the paper itself). Similarly, “BASIS” includes statements of successful problem-solving activities, but only if they are achieved by previous work that the current paper bases itself on. Our definition is simpler in that we are interested only in problem solution structure, not in the other dimensions covered in AZ. Our definition is also more far-reaching than AZ, in that we are interested in all problems mentioned in the text, no matter whose problems they are. Problem-solution recognition can therefore be seen as one aspect of AZ which can be independently modelled as a “service task”. This means that good problem solution structure recognition should theoretically improve AZ recognition.

In this work, we approach the task of identifying problem-solving patterns in scientific text. We choose to use the model of problem-solving described by Hoey ( 2001 ). This pattern comprises four parts: Situation, Problem, Response and Evaluation. The Situation element is considered optional to the pattern, and so our focus centres on the core pattern elements.

Goal statement and task

Many surface features in the text offer themselves up as potential signals for detecting problem-solving patterns in text. However, since Situation is an optional element, we decided to focus on either Problem or Response and Evaluation as signals of the pattern. Moreover, we decide to look for each type in isolation. Our reasons for this are as follows: It is quite rare for an author to introduce a problem without resolving it using some sort of response, and so this is a good starting point in identifying the pattern. There are exceptions to this, as authors will sometimes introduce a problem and then leave it to future work, but overall there should be enough signal in the Problem element to make our method of looking for it in isolation worthwhile. The second signal we look for is the use of Response and Evaluation within the same sentence. Similar to Problem elements, we hypothesise that this formulation is well enough signalled externally to help us in detecting the pattern. For example, consider the following Response and Evaluation: “One solution is to use smoothing”. In this statement, the author is explicitly stating that smoothing is a solution to a problem which must have been mentioned in a prior statement. In scientific text, we often observe that solutions implicitly contain both Response and Evaluation (positive) elements. Therefore, due to these reasons there should be sufficient external signals for the two pattern elements we concentrate on here.

When attempting to find Problem elements in text, we run into the issue that the word “problem” actually has at least two word senses that need to be distinguished. There is a word sense of “problem” that means something which must be undertaken (i.e. task), while another sense is the core sense of the word, something that is problematic and negative. Only the latter sense is aligned with our sense of problemhood. This is because the simple description of a task does not predispose problemhood, just a wish to perform some act. Consider the following examples, where the non-desired word sense is being used:

• “Das and Petrov (2011) also consider the problem of unsupervised bilingual POS induction”. (Chen et al. 2011 ).
• “In this paper, we describe advances on the problem of NER in Arabic Wikipedia”. (Mohit et al. 2012 ).

Here, the author explicitly states that the phrases in orange are problems, they align with our definition of research tasks and not with what we call here ‘problematic problems’. We will now give some examples from our corpus for the desired, core word sense:

• “The major limitation of supervised approaches is that they require annotations for example sentences.” (Poon and Domingos 2009 ).
• “To solve the problem of high dimensionality we use clustering to group the words present in the corpus into much smaller number of clusters”. (Saha et al. 2008 ).

When creating our corpus of positive and negative examples, we took care to select only problem strings that satisfy our definition of problemhood; “ Corpus creation ” section will explain how we did that.

Corpus creation

Our new corpus is a subset of the latest version of the ACL anthology released in March, 2016 1 which contains 22,878 articles in the form of PDFs and OCRed text. 2

The 2016 version was also parsed using ParsCit (Councill et al. 2008 ). ParsCit recognises not only document structure, but also bibliography lists as well as references within running text. A random subset of 2500 papers was collected covering the entire ACL timeline. In order to disregard non-article publications such as introductions to conference proceedings or letters to the editor, only documents containing abstracts were considered. The corpus was preprocessed using tokenisation, lemmatisation and dependency parsing with the Rasp Parser (Briscoe et al. 2006 ).

Definition of ground truth

Our goal was to define a ground truth for problem and solution strings, while covering as wide a range as possible of syntactic variations in which such strings naturally occur. We also want this ground truth to cover phenomena of problem and solution status which are applicable whether or not the problem or solution status is explicitly mentioned in the text.

To simplify the task, we only consider here problem and solution descriptions that are at most one sentence long. In reality, of course, many problem descriptions and solution descriptions go beyond single sentence, and require for instance an entire paragraph. However, we also know that short summaries of problems and solutions are very prevalent in science, and also that these tend to occur in the most prominent places in a paper. This is because scientists are trained to express their contribution and the obstacles possibly hindering their success, in an informative, succinct manner. That is the reason why we can afford to only look for shorter problem and solution descriptions, ignoring those that cross sentence boundaries.

To define our ground truth, we examined the parsed dependencies and looked for a target word (“problem/solution”) in subject position, and then chose its syntactic argument as our candidate problem or solution phrase. To increase the variation, i.e., to find as many different-worded problem and solution descriptions as possible, we additionally used semantically similar words (near-synonyms) of the target words “problem” or “solution” for the search. Semantic similarity was defined as cosine in a deep learning distributional vector space, trained using Word2Vec (Mikolov et al. 2013 ) on 18,753,472 sentences from a biomedical corpus based on all full-text Pubmed articles (McKeown et al. 2016 ). From the 200 words which were semantically closest to “problem”, we manually selected 28 clear synonyms. These are listed in Table  1 . From the 200 semantically closest words to “solution” we similarly chose 19 (Table  2 ). Of the sentences matching our dependency search, a subset of problem and solution candidate sentences were randomly selected.

Selected words for use in problem candidate phrase extraction

Selected words for use in solution candidate phrase extraction

An example of this is shown in Fig.  2 . Here, the target word “drawback” is in subject position (highlighted in red), and its clausal argument (ccomp) is “(that) it achieves low performance” (highlighted in purple). Examples of other arguments we searched for included copula constructions and direct/indirect objects.

Example of our extraction method for problems using dependencies. (Color figure online)

If more than one candidate was found in a sentence, one was chosen at random. Non-grammatical sentences were excluded; these might appear in the corpus as a result of its source being OCRed text.

800 candidates phrases expressing problems and solutions were automatically extracted (1600 total) and then independently checked for correctness by two annotators (the two authors of this paper). Both authors found the task simple and straightforward. Correctness was defined by two criteria:

• An unexplained phenomenon or a problematic state in science; or
• A research question; or
• An artifact that does not fulfil its stated specification.
• The phrase must not lexically give away its status as problem or solution phrase.

The second criterion saves us from machine learning cues that are too obvious. If for instance, the phrase itself contained the words “lack of” or “problematic” or “drawback”, our manual check rejected it, because it would be too easy for the machine learner to learn such cues, at the expense of many other, more generally occurring cues.

Sampling of negative examples

We next needed to find negative examples for both cases. We wanted them not to stand out on the surface as negative examples, so we chose them so as to mimic the obvious characteristics of the positive examples as closely as possible. We call the negative examples ‘non-problems’ and ‘non-solutions’ respectively. We wanted the only differences between problems and non-problems to be of a semantic nature, nothing that could be read off on the surface. We therefore sampled a population of phrases that obey the same statistical distribution as our problem and solution strings while making sure they really are negative examples. We started from sentences not containing any problem/solution words (i.e. those used as target words). From each such sentence, we at random selected one syntactic subtree contained in it. From these, we randomly selected a subset of negative examples of problems and solutions that satisfy the following conditions:

• The distribution of the head POS tags of the negative strings should perfectly match the head POS tags 3 of the positive strings. This has the purpose of achieving the same proportion of surface syntactic constructions as observed in the positive cases.
• The average lengths of the negative strings must be within a tolerance of the average length of their respective positive candidates e.g., non-solutions must have an average length very similar (i.e. + / -  small tolerance) to solutions. We chose a tolerance value of 3 characters.

Again, a human quality check was performed on non-problems and non-solutions. For each candidate non-problem statement, the candidate was accepted if it did not contain a phenomenon, a problematic state, a research question or a non-functioning artefact. If the string expressed a research task, without explicit statement that there was anything problematic about it (i.e., the ‘wrong’ sense of “problem”, as described above), it was allowed as a non-problem. A clause was confirmed as a non-solution if the string did not represent both a response and positive evaluation.

If the annotator found that the sentence had been slightly mis-parsed, but did contain a candidate, they were allowed to move the boundaries for the candidate clause. This resulted in cleaner text, e.g., in the frequent case of coordination, when non-relevant constituents could be removed.

From the set of sentences which passed the quality-test for both independent assessors, 500 instances of positive and negative problems/solutions were randomly chosen (i.e. 2000 instances in total). When checking for correctness we found that most of the automatically extracted phrases which did not pass the quality test for problem-/solution-hood were either due to obvious learning cues or instances where the sense of problem-hood used is relating to tasks (cf. “ Goal statement and task ” section).

Experimental design

In our experiments, we used three classifiers, namely Naïve Bayes, Logistic Regression and a Support Vector Machine. For all classifiers an implementation from the WEKA machine learning library (Hall et al. 2009 ) was chosen. Given that our dataset is small, tenfold cross-validation was used instead of a held out test set. All significance tests were conducted using the (two-tailed) Sign Test (Siegel 1956 ).

Linguistic correlates of problem- and solution-hood

We first define a set of features without taking the phrase’s context into account. This will tell us about the disambiguation ability of the problem/solution description’s semantics alone. In particular, we cut out the rest of the sentence other than the phrase and never use it for classification. This is done for similar reasons to excluding certain ‘give-away’ phrases inside the phrases themselves (as explained above). As the phrases were found using templates, we know that the machine learner would simply pick up on the semantics of the template, which always contains a synonym of “problem” or “solution”, thus drowning out the more hidden features hopefully inherent in the semantics of the phrases themselves. If we allowed the machine learner to use these stronger features, it would suffer in its ability to generalise to the real task.

ngrams Bags of words are traditionally successfully used for classification tasks in NLP, so we included bags of words (lemmas) within the candidate phrases as one of our features (and treat it as a baseline later on). We also include bigrams and trigrams as multi-word combinations can be indicative of problems and solutions e.g., “combinatorial explosion”.

Polarity Our second feature concerns the polarity of each word in the candidate strings. Consider the following example of a problem taken from our dataset: “very conservative approaches to exact and partial string matches overgenerate badly”. In this sentence, words such as “badly” will be associated with negative polarity, therefore being useful in determining problem-hood. Similarly, solutions will often be associated with a positive sentiment e.g. “smoothing is a good way to overcome data sparsity” . To do this, we perform word sense disambiguation of each word using the Lesk algorithm (Lesk 1986 ). The polarity of the resulting synset in SentiWordNet (Baccianella et al. 2010 ) was then looked up and used as a feature.

Syntax Next, a set of syntactic features were defined by using the presence of POS tags in each candidate. This feature could be helpful in finding syntactic patterns in problems and solutions. We were careful not to base the model directly on the head POS tag and the length of each candidate phrase, as these are defining characteristics used for determining the non-problem and non-solution candidate set.

Negation Negation is an important property that can often greatly affect the polarity of a phrase. For example, a phrase containing a keyword pertinent to solution-hood may be a good indicator but with the presence of negation may flip the polarity to problem-hood e.g., “this can’t work as a solution”. Therefore, presence of negation is determined.

Exemplification and contrast Problems and solutions are often found to be coupled with examples as they allow the author to elucidate their point. For instance, consider the following solution: “Once the translations are generated, an obvious solution is to pick the most fluent alternative, e.g., using an n-gram language model”. (Madnani et al. 2012 ). To acknowledge this, we check for presence of exemplification. In addition to examples, problems in particular are often found when contrast is signalled by the author (e.g. “however, “but”), therefore we also check for presence of contrast in the problem and non-problem candidates only.

Discourse Problems and solutions have also been found to have a correlation with discourse properties. For example, problem-solving patterns often occur in the background sections of a paper. The rationale behind this is that the author is conventionally asked to objectively criticise other work in the background (e.g. describing research gaps which motivate the current paper). To take this in account, we examine the context of each string and capture the section header under which it is contained (e.g. Introduction, Future work). In addition, problems and solutions are often found following the Situation element in the problem-solving pattern (cf. “ Introduction ” section). This preamble setting up the problem or solution means that these elements are likely not to be found occurring at the beginning of a section (i.e. it will usually take some sort of introduction to detail how something is problematic and why a solution is needed). Therefore we record the distance from the candidate string to the nearest section header.

Subcategorisation and adverbials Solutions often involve an activity (e.g. a task). We also model the subcategorisation properties of the verbs involved. Our intuition was that since problematic situations are often described as non-actions, then these are more likely to be intransitive. Conversely solutions are often actions and are likely to have at least one argument. This feature was calculated by running the C&C parser (Curran et al. 2007 ) on each sentence. C&C is a supertagger and parser that has access to subcategorisation information. Solutions are also associated with resultative adverbial modification (e.g. “thus, therefore, consequently”) as it expresses the solutionhood relation between the problem and the solution. It has been seen to occur frequently in problem-solving patterns, as studied by Charles ( 2011 ). Therefore, we check for presence of resultative adverbial modification in the solution and non-solution candidate only.

Embeddings We also wanted to add more information using word embeddings. This was done in two different ways. Firstly, we created a Doc2Vec model (Le and Mikolov 2014 ), which was trained on  ∼  19  million sentences from scientific text (no overlap with our data set). An embedding was created for each candidate sentence. Secondly, word embeddings were calculated using the Word2Vec model (cf. “ Corpus creation ” section). For each candidate head, the full word embedding was included as a feature. Lastly, when creating our polarity feature we query SentiWordNet using synsets assigned by the Lesk algorithm. However, not all words are assigned a sense by Lesk, so we need to take care when that happens. In those cases, the distributional semantic similarity of the word is compared to two words with a known polarity, namely “poor” and “excellent”. These particular words have traditionally been consistently good indicators of polarity status in many studies (Turney 2002 ; Mullen and Collier 2004 ). Semantic similarity was defined as cosine similarity on the embeddings of the Word2Vec model (cf. “ Corpus creation ” section).

Modality Responses to problems in scientific writing often express possibility and necessity, and so have a close connection with modality. Modality can be broken into three main categories, as described by Kratzer ( 1991 ), namely epistemic (possibility), deontic (permission / request / wish) and dynamic (expressing ability).

Problems have a strong relationship to modality within scientific writing. Often, this is due to a tactic called “hedging” (Medlock and Briscoe 2007 ) where the author uses speculative language, often using Epistemic modality, in an attempt to make either noncommital or vague statements. This has the effect of allowing the author to distance themselves from the statement, and is often employed when discussing negative or problematic topics. Consider the following example of Epistemic modality from Nakov and Hearst ( 2008 ): “A potential drawback is that it might not work well for low-frequency words”.

To take this linguistic correlate into account as a feature, we replicated a modality classifier as described by (Ruppenhofer and Rehbein 2012 ). More sophisticated modality classifiers have been recently introduced, for instance using a wide range of features and convolutional neural networks, e.g, (Zhou et al. 2015 ; Marasović and Frank 2016 ). However, we wanted to check the effect of a simpler method of modality classification on the final outcome first before investing heavily into their implementation. We trained three classifiers using the subset of features which Ruppenhofer et al. reported as performing best, and evaluated them on the gold standard dataset provided by the authors 4 . The results of the are shown in Table  3 . The dataset contains annotations of English modal verbs on the 535 documents of the first MPQA corpus release (Wiebe et al. 2005 ).

Modality classifier results (precision/recall/f-measure) using Naïve Bayes (NB), logistic regression, and a support vector machine (SVM)

Italicized results reflect highest f-measure reported per modal category

Logistic Regression performed best overall and so this model was chosen for our upcoming experiments. With regards to the optative and concessive modal categories, they can be seen to perform extremely poorly, with the optative category receiving a null score across all three classifiers. This is due to a limitation in the dataset, which is unbalanced and contains very few instances of these two categories. This unbalanced data also is the reason behind our decision of reporting results in terms of recall, precision and f-measure in Table  3 .

The modality classifier was then retrained on the entirety of the dataset used by Ruppenhofer and Rehbein ( 2012 ) using the best performing model from training (Logistic Regression). This new model was then used in the upcoming experiment to predict modality labels for each instance in our dataset.

As can be seen from Table  4 , we are able to achieve good results for distinguishing a problematic statement from non-problematic one. The bag-of-words baseline achieves a very good performance of 71.0% for the Logistic Regression classifier, showing that there is enough signal in the candidate phrases alone to distinguish them much better than random chance.

Results distinguishing problems from non-problems using Naïve Bayes (NB), logistic regression (LR) and a support vector machine (SVM)

Each feature set’s performance is shown in isolation followed by combinations with other features. Tenfold stratified cross-validation was used across all experiments. Statistical significance with respect to the baseline at the p  < 0.05 , 0.01, 0.001 levels is denoted by *, ** and *** respectively

Taking a look at Table  5 , which shows the information gain for the top lemmas,

Information gain (IG) in bits of top lemmas from the bag-of-words baseline in Table  4

we can see that the top lemmas are indeed indicative of problemhood (e.g. “limit”,“explosion”). Bigrams achieved good performance on their own (as did negation and discourse) but unfortunately performance deteriorated when using trigrams, particularly with the SVM and LR. The subcategorisation feature was the worst performing feature in isolation. Upon taking a closer look at our data, we saw that our hypothesis that intransitive verbs are commonly used in problematic statements was true, with over 30% of our problems (153) using them. However, due to our sampling method for the negative cases we also picked up many intransitive verbs (163). This explains the almost random chance performance (i.e.  50%) given that the distribution of intransitive verbs amongst the positive and negative candidates was almost even.

The modality feature was the most expensive to produce, but also didn’t perform very well is isolation. This surprising result may be partly due to a data sparsity issue

where only a small portion (169) of our instances contained modal verbs. The breakdown of how many types of modal senses which occurred is displayed in Table  6 . The most dominant modal sense was epistemic. This is a good indicator of problemhood (e.g. hedging, cf. “ Linguistic correlates of problem- and solution-hood ” section) but if the accumulation of additional data was possible, we think that this feature may have the potential to be much more valuable in determining problemhood. Another reason for the performance may be domain dependence of the classifier since it was trained on text from different domains (e.g. news). Additionally, modality has also shown to be helpful in determining contextual polarity (Wilson et al. 2005 ) and argumentation (Becker et al. 2016 ), so using the output from this modality classifier may also prove useful for further feature engineering taking this into account in future work.

Number of instances of modal senses

Polarity managed to perform well but not as good as we hoped. However, this feature also suffers from a sparsity issue resulting from cases where the Lesk algorithm (Lesk 1986 ) is not able to resolve the synset of the syntactic head.

Knowledge of syntax provides a big improvement with a significant increase over the baseline results from two of the classifiers.

Examining this in greater detail, POS tags with high information gain mostly included tags from open classes (i.e. VB-, JJ-, NN- and RB-). These tags are often more associated with determining polarity status than tags such as prepositions and conjunctions (i.e. adverbs and adjectives are more likely to be describing something with a non-neutral viewpoint).

The embeddings from Doc2Vec allowed us to obtain another significant increase in performance (72.9% with Naïve Bayes) over the baseline and polarity using Word2Vec provided the best individual feature result (77.2% with SVM).

Combining all features together, each classifier managed to achieve a significant result over the baseline with the best result coming from the SVM (81.8%). Problems were also better classified than non-problems as shown in the confusion matrix in Table  7 . The addition of the Word2Vec vectors may be seen as a form of smoothing in cases where previous linguistic features had a sparsity issue i.e., instead of a NULL entry, the embeddings provide some sort of value for each candidate. Particularly wrt. the polarity feature, cases where Lesk was unable to resolve a synset meant that a ZERO entry was added to the vector supplied to the machine learner. Amongst the possible combinations, the best subset of features was found by combining all features with the exception of bigrams, trigrams, subcategorisation and modality. This subset of features managed to improve results in both the Naïve Bayes and SVM classifiers with the highest overall result coming from the SVM (82.3%).

Confusion matrix for problems

The results for disambiguation of solutions from non-solutions can be seen in Table  8 . The bag-of-words baseline performs much better than random, with the performance being quite high with regard to the SVM (this result was also higher than any of the baseline performances from the problem classifiers). As shown in Table  9 , the top ranked lemmas from the best performing model (using information gain) included “use” and “method”. These lemmas are very indicative of solutionhood and so give some insight into the high baseline returned from the machine learners. Subcategorisation and the result adverbials were the two worst performing features. However, the low performance for subcategorisation is due to the sampling of the non-solutions (the same reason for the low performance of the problem transitivity feature). When fitting the POS-tag distribution for the negative samples, we noticed that over 80% of the head POS-tags were verbs (much higher than the problem heads). The most frequent verb type being the infinite form.

Results distinguishing solutions from non-solutions using Naïve Bayes (NB), logistic regression (LR) and a support vector machine (SVM)

Each feature set’s performance is shown in isolation followed by combinations with other features. Tenfold stratified cross-validation was used across all experiments

Information gain (IG) in bits of top lemmas from the bag-of-words baseline in Table  8

This is not surprising given that a very common formulation to describe a solution is to use the infinitive “TO” since it often describes a task e.g., “One solution is to find the singletons and remove them”. Therefore, since the head POS tags of the non-solutions had to match this high distribution of infinitive verbs present in the solution, the subcategorisation feature is not particularly discriminatory. Polarity, negation, exemplification and syntactic features were slightly more discriminate and provided comparable results. However, similar to the problem experiment, the embeddings from Word2Vec and Doc2Vec proved to be the best features, with polarity using Word2Vec providing the best individual result (73.4% with SVM).

Combining all features together managed to improve over each feature in isolation and beat the baseline using all three classifiers. Furthermore, when looking at the confusion matrix in Table  10 the solutions were classified more accurately than the non-solutions. The best subset of features was found by combining all features without adverbial of result, bigrams, exemplification, negation, polarity and subcategorisation. The best result using this subset of features was achieved by the SVM with 79.7%. It managed to greatly improve upon the baseline but was just shy of achieving statistical significance ( p = 0.057 ).

Confusion matrix for solutions

In this work, we have presented new supervised classifiers for the task of identifying problem and solution statements in scientific text. We have also introduced a new corpus for this task and used it for evaluating our classifiers. Great care was taken in constructing the corpus by ensuring that the negative and positive samples were closely matched in terms of syntactic shape. If we had simply selected random subtrees for negative samples without regard for any syntactic similarity with our positive samples, the machine learner may have found easy signals such as sentence length. Additionally, since we did not allow the machine learner to see the surroundings of the candidate string within the sentence, this made our task even harder. Our performance on the corpus shows promise for this task, and proves that there are strong signals for determining both the problem and solution parts of the problem-solving pattern independently.

With regard to classifying problems from non-problems, features such as the POS tag, document and word embeddings provide the best features, with polarity using the Word2Vec embeddings achieving the highest feature performance. The best overall result was achieved using an SVM with a subset of features (82.3%). Classifying solutions from non-solutions also performs well using the embedding features, with the best feature also being polarity using the Word2Vec embeddings, and the highest result also coming from the SVM with a feature subset (79.7%).

In future work, we plan to link problem and solution statements which were found independently during our corpus creation. Given that our classifiers were trained on data solely from the ACL anthology, we also hope to investigate the domain specificity of our classifiers and see how well they can generalise to domains other than ACL (e.g. bioinformatics). Since we took great care at removing the knowledge our classifiers have of the explicit statements of problem and solution (i.e. the classifiers were trained only on the syntactic argument of the explicit statement of problem-/solution-hood), our classifiers should in principle be in a good position to generalise, i.e., find implicit statements too. In future work, we will measure to which degree this is the case.

To facilitate further research on this topic, all code and data used in our experiments can be found here: www.cl.cam.ac.uk/~kh562/identifying-problems-and-solutions.html

Acknowledgements

The first author has been supported by an EPSRC studentship (Award Ref: 1641528). We thank the reviewers for their helpful comments.

1 http://acl-arc.comp.nus.edu.sg/ .

2 The corpus comprises 3,391,198 sentences, 71,149,169 words and 451,996,332 characters.

3 The head POS tags were found using a modification of the Collins’ Head Finder. This modified algorithm addresses some of the limitations of the head finding heuristics described by Collins ( 2003 ) and can be found here: http://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/trees/ModCollinsHeadFinder.html .

4 https://www.uni-hildesheim.de/ruppenhofer/data/modalia_release1.0.tgz.

Contributor Information

Kevin Heffernan, Email: [email protected] .

Simone Teufel, Email: [email protected] .

• Baccianella S, Esuli A, Sebastiani F. Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. LREC. 2010; 10 :2200–2204. [ Google Scholar ]
• Becker, M., Palmer, A., & Frank, A. (2016). Clause types and modality in argumentative microtexts. In Workshop on foundations of the language of argumentation (in conjunction with COMMA) .
• Briscoe, T., Carroll, J., & Watson, R. (2006). The second release of the rasp system. In Proceedings of the COLING/ACL on interactive presentation sessions, association for computational linguistics pp. 77–80.
• Chandrasekaran B. Towards a taxonomy of problem solving types. AI Magazine. 1983; 4 (1):9. [ Google Scholar ]
• Charles M. Adverbials of result: Phraseology and functions in the problem-solution pattern. Journal of English for Academic Purposes. 2011; 10 (1):47–60. doi: 10.1016/j.jeap.2011.01.002. [ CrossRef ] [ Google Scholar ]
• Chen, D., Dyer, C., Cohen, S. B., & Smith, N. A. (2011). Unsupervised bilingual pos tagging with markov random fields. In Proceedings of the first workshop on unsupervised learning in NLP, association for computational linguistics pp. 64–71.
• Collins M. Head-driven statistical models for natural language parsing. Computational Linguistics. 2003; 29 (4):589–637. doi: 10.1162/089120103322753356. [ CrossRef ] [ Google Scholar ]
• Councill, I. G., Giles, C. L., & Kan, M. Y. (2008). Parscit: An open-source CRF reference string parsing package. In LREC .
• Curran, J. R., Clark, S., & Bos, J. (2007). Linguistically motivated large-scale NLP with C&C and boxer. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions, association for computational linguistics pp. 33–36.
• Flowerdew L. Corpus-based analyses of the problem-solution pattern: A phraseological approach. Amsterdam: John Benjamins Publishing; 2008. [ Google Scholar ]
• Grimes JE. The thread of discourse. Berlin: Walter de Gruyter; 1975. [ Google Scholar ]
• Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH. The weka data mining software: An update. ACM SIGKDD Explorations Newsletter. 2009; 11 (1):10–18. doi: 10.1145/1656274.1656278. [ CrossRef ] [ Google Scholar ]
• Hoey M. Textual interaction: An introduction to written discourse analysis. Portland: Psychology Press; 2001. [ Google Scholar ]
• Hutchins J. On the structure of scientific texts. UEA Papers in Linguistics. 1977; 5 (3):18–39. [ Google Scholar ]
• Jonassen DH. Toward a design theory of problem solving. Educational Technology Research and Development. 2000; 48 (4):63–85. doi: 10.1007/BF02300500. [ CrossRef ] [ Google Scholar ]
• Jordan MP. Short texts to explain problem-solution structures-and vice versa. Instructional Science. 1980; 9 (3):221–252. doi: 10.1007/BF00177328. [ CrossRef ] [ Google Scholar ]
• Kratzer, A. (1991). Modality. In von Stechow & Wunderlich (Eds.), Semantics: An international handbook of contemporary research .
• Le QV, Mikolov T. Distributed representations of sentences and documents. ICML. 2014; 14 :1188–1196. [ Google Scholar ]
• Lesk, M. (1986). Automatic sense disambiguation using machine readable dictionaries: How to tell a pine cone from an ice cream cone. In Proceedings of the 5th annual international conference on Systems documentation, ACM (pp. 24–26).
• Madnani, N., Tetreault, J., & Chodorow, M. (2012). Exploring grammatical error correction with not-so-crummy machine translation. In Proceedings of the seventh workshop on building educational applications using NLP, association for computational linguistics pp. 44–53.
• Mann WC, Thompson SA. Rhetorical structure theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse. 1988; 8 (3):243–281. doi: 10.1515/text.1.1988.8.3.243. [ CrossRef ] [ Google Scholar ]
• Marasović, A., & Frank, A. (2016). Multilingual modal sense classification using a convolutional neural network. In Proceedings of the 1st Workshop on Representation Learning for NLP .
• McKeown K, Daume H, Chaturvedi S, Paparrizos J, Thadani K, Barrio P, Biran O, Bothe S, Collins M, Fleischmann KR, et al. Predicting the impact of scientific concepts using full-text features. Journal of the Association for Information Science and Technology. 2016; 67 :2684–2696. doi: 10.1002/asi.23612. [ CrossRef ] [ Google Scholar ]
• Medlock B, Briscoe T. Weakly supervised learning for hedge classification in scientific literature. ACL, Citeseer. 2007; 2007 :992–999. [ Google Scholar ]
• Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (pp. 3111–3119).
• Mohit, B., Schneider, N., Bhowmick, R., Oflazer, K., & Smith, N. A. (2012). Recall-oriented learning of named entities in arabic wikipedia. In Proceedings of the 13th conference of the European chapter of the association for computational linguistics, association for computational linguistics (pp. 162–173).
• Mullen T, Collier N. Sentiment analysis using support vector machines with diverse information sources. EMNLP. 2004; 4 :412–418. [ Google Scholar ]
• Nakov, P., Hearst, M. A. (2008). Solving relational similarity problems using the web as a corpus. In: ACL (pp. 452–460).
• Poon, H., & Domingos, P. (2009). Unsupervised semantic parsing. In Proceedings of the 2009 conference on empirical methods in natural language processing: Volume 1-association for computational linguistics (pp. 1–10).
• Ruppenhofer, J., & Rehbein, I. (2012). Yes we can!? Annotating the senses of English modal verbs. In Proceedings of the 8th international conference on language resources and evaluation (LREC), Citeseer (pp. 24–26).
• Saha, S. K., Mitra, P., & Sarkar, S. (2008). Word clustering and word selection based feature reduction for maxent based hindi ner. In ACL (pp. 488–495).
• Scott, M. (2001). Mapping key words to problem and solution. In Patterns of text: In honour of Michael Hoey Benjamins, Amsterdam (pp. 109–127).
• Siegel S. Nonparametric statistics for the behavioral sciences. New York: McGraw-hill; 1956. [ Google Scholar ]
• Strübing, J. (2007). Research as pragmatic problem-solving: The pragmatist roots of empirically-grounded theorizing. In The Sage handbook of grounded theory (pp. 580–602).
• Teufel, S., et al. (2000). Argumentative zoning: Information extraction from scientific text. PhD Thesis, Citeseer .
• Turney, P. D. (2002). Thumbs up or thumbs down?: Semantic orientation applied to unsupervised classification of reviews. In Proceedings of the 40th annual meeting on association for computational linguistics, association for computational linguistics (pp. 417–424).
• Van Dijk TA. Text and context explorations in the semantics and pragmatics of discourse. London: Longman; 1980. [ Google Scholar ]
• Wiebe J, Wilson T, Cardie C. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation. 2005; 39 (2):165–210. doi: 10.1007/s10579-005-7880-9. [ CrossRef ] [ Google Scholar ]
• Wilson, T., Wiebe, J., & Hoffmann, P. (2005). Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the conference on human language technology and empirical methods in natural language processing, association for computational linguistics (pp. 347–354).
• Winter, E. O. (1968). Some aspects of cohesion. In Sentence and clause in scientific English . University College London.
• Zhou, M., Frank, A., Friedrich, A., & Palmer, A. (2015). Semantically enriched models for modal sense classification. In Workshop on linking models of lexical, sentential and discourse-level semantics (LSDSem) (p. 44).

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Perspective
• Open access
• Published: 18 June 2021

Nobel Turing Challenge: creating the engine for scientific discovery

• Hiroaki Kitano   ORCID: orcid.org/0000-0002-3589-1953 1  

npj Systems Biology and Applications volume  7 , Article number:  29 ( 2021 ) Cite this article

17k Accesses

25 Citations

163 Altmetric

Metrics details

• Computational biology and bioinformatics

Scientific discovery has long been one of the central driving forces in our civilization. It uncovered the principles of the world we live in, and enabled us to invent new technologies reshaping our society, cure diseases, explore unknown new frontiers, and hopefully lead us to build a sustainable society. Accelerating the speed of scientific discovery is therefore one of the most important endeavors. This requires an in-depth understanding of not only the subject areas but also the nature of scientific discoveries themselves. In other words, the “science of science” needs to be established, and has to be implemented using artificial intelligence (AI) systems to be practically executable. At the same time, what may be implemented by “AI Scientists” may not resemble the scientific process conducted by human scientist. It may be an alternative form of science that will break the limitation of current scientific practice largely hampered by human cognitive limitation and sociological constraints. It could give rise to a human-AI hybrid form of science that shall bring systems biology and other sciences into the next stage. The Nobel Turing Challenge aims to develop a highly autonomous AI system that can perform top-level science, indistinguishable from the quality of that performed by the best human scientists, where some of the discoveries may be worthy of Nobel Prize level recognition and beyond.

Similar content being viewed by others

Artificial intelligence and illusions of understanding in scientific research

Lisa Messeri & M. J. Crockett

Accelerating science with human-aware artificial intelligence

Jamshid Sourati & James A. Evans

Scientific discovery in the age of artificial intelligence

Hanchen Wang, Tianfan Fu, … Marinka Zitnik

Nobel Turing Challenge as an ultimate grand challenge

Understanding, reformulating, and accelerating the process of scientific discovery is critical in solving problems we are facing and exploring the future. Building the machine to make it happen could be one of the most important contribution to society, and it will transform many areas of science and technology including systems biology. Since scientific research has been one of the most important activities that drove our civilization forward, the implications of such development will be profound.

Attempts to understand the process of scientific discoveries have a long tradition in the philosophy of science as well as artificial intelligence. Karl Popper introduced a concept of falsifiability as a criterion and process of solid scientific process but the process of hypothesis and concept generation do not have particular logic behind it 1 . Thomas Kuhn proposed a concept of a Paradigm shift where two competing paradigms are incommensurable and a set of knowledge, rather than a single knowledge, has to be switched with the transition of paradigm 2 . Imre Lakatos reconciles them by proposing actual science makes progress based on a “research program” composed of a hardcore that is immune to revision and flexible peripheral theories 3 . Contrary to these positions, Paul Feyerabend argued that there are no methodological rules in the scientific process 4 . Although these arguments are important thoughts in the philosophy of science, ideas are philosophical and not concrete to be implemented computationally. In addition, these studies are focused on how science is carried out as a part of human social activities. A rare example of implementing such concepts can be seen in the model inference system implemented by Ehud Shapiro that reflects Popper’s falsifiability 5 .

Not surprisingly, scientific discovery has been a major topic in artificial intelligence research that dates back to DENDRAL 6 and META-DENDRAL, followed by MYCIN, BEACON 7 , AM, and EURISKO 6 , 8 . It continues to be one of the main topics of AI 9 , 10 . Recently, an automated experimental system that closed-the-loop of hypothesis generation, experimental planning, and execution has developed for budding yeast genetics that clearly marks the next step towards an AI Scientist 11 , 12 , 13 . While these pioneering works have focused on a single data set or a specific task using limited resources, it signifies the state-of-the-art of technology today that can be the basis of more ambitious challenges.

The obvious next step is to develop a system that makes scientific discoveries that shall truly impact the way we do science and aim for major discoveries. Therefore, I propose the launch of a grand challenge to develop AI systems that can make significant scientific discoveries that can outperform the best human scientist, with the ultimate purpose of creating the alternative form of scientific discovery 14 . Such a system, or systems, may be called “AI Scientist” that is most likely a constellation of software and hardware modules dynamically interacting to accomplish tasks. Since the critical feature that distinguishes it from conventional laboratory automation is its capability to generate hypotheses, learn from data and interactions with humans and other parts of the system, reasoning, and a high level of autonomous decision-making, the term “AI Scientist” best represents the characteristics of the system to be developed. The best way to accelerate the grand challenge of this nature is to define a clear mission statement with an audacious yet provocative goal such as winning the Nobel Prize 14 . Therefore, I propose the Nobel Turing Challenge as a grand challenge for artificial intelligence that aims at “developing AI Scientists capable of autonomously carrying out research to make major scientific discoveries and win a Nobel Prize by 2050”. While the previous article 14 focused on rationales for such a challenge with emphasis on human cognitive limitations and needs for exhaustive search of hypothesis space, this article formulates the vision as the Nobel Turing Challenge and implications of massive and unbiased search of hypothesis space and verification, architectural issues, and interaction with human scientists are discussed as a transformative paradigm in science. The distinct characteristic of this challenge is to field the system into an open-ended domain to explore significant discoveries rather than rediscovering what we already know or trying to mimic speculated human thought processes. The vision is to reformulate scientific discovery itself and to create an alternative form of scientific discovery.

The accomplishment of this challenge requires two goals to be achieved that are: (1) to develop an AI Scientist that performs scientific research highly autonomously enabling scientific discoveries at scale, and (2) to develop an AI Scientist capable of making strategic choices on the topic of research, that can communicate in the form of publications and other means to explain the value, methods, reasoning behind the discovery, and their applications and social implications. When both goals are met, the machine will be (almost) comparable to the top-level human scientist as well as being scientific collaborators. The question and challenge is whether the machine will be indistinguishable from the top-level human scientists and most likely pass the Feigenbaum Test which is a variation of the Turing Test 15 , or whether it will exhibit patterns of scientific discovery that are different from human scientists.

It should be noted “winning the Nobel Prize” is used as a symbolic target illustrating the level of discoveries the challenge is aiming at. The value lies in the development of machines that can make discoveries continuously and autonomously, rather than winning any award including the Nobel Prize. It is used as a symbolism that triggers inspiration and controversy.

At the same time, the implication of the statement that explicitly aims to win the Nobel Prize poses a series of interesting questions whether such AI systems making a decisive discovery may also evolve to be indistinguishable from the top human scientist (passing the Feigenbaum Test 16 ). As witness in case of Satoshi Nakamoto’s blockchain and bitcoin, there is a case where a decisive contribution was simply published as a blogpost 17 and taken seriously, yet no one ever met him and his identity (at the time of writing) is a complete mystery. Given the possibility of creating a highly sophisticated virtual agent to interact with a human, with natural language capability to generate professional article, it will be non-trivial to distinguish whether such a scientist is human or AI. If a developer of an AI Scientist determined to create a virtual persona of a scientist with an ORCID iD, for demonstration of technological achievement, product promotion, or for another motivation, it would be almost impossible to distinguish between the AI and human scientist. The challenge shall be considered practically achieved when the Nobel Prize committee is alerted for any confusion on potential recipients. We might expect an AI Scientist detection system to be developed to identify who is an AI Scientist or not, that may resemble Deckard’s interrogation of Rachael in Blade Runner.

It shall be made clear that this goal does not state or imply that “all major discoveries will be made by AI Scientists”, nor that completeness of hypotheses or discoveries made by AI Scientist will be achieved. The challenge is fundamentally different from any attempts to prove the completeness of the system, including the Hilbert Program intended to prove axiomatic completeness of mathematic where feasibility is debunked by Incompleteness theorems by Kurt Gödel. It simply implies: “among discoveries made by AI Scientists, there should be discoveries that are considered very significant at the level worthy of the Nobel Prize or beyond”. The challenge is initiated based on the belief any significant acceleration of scientific discovery would benefits our civilization. This will be achieved by creating an alternative form of scientific discovery, and will change the form of science as we know as well as uncovering the essence of scientific discovery. The utility of such technology shall benefit broader areas of science, industry, and society.

The core of the research program shall be about “Science of Science” rather than “Science of the process of science by human scientists”. As in the case of past AI grand challenges, the best and perhaps the only way to demonstrate that scientific discovery can be reformulated computationally is to develop an AI system that outperforms the best human scientists. Furthermore, it is not sufficient to have AI Scientist make one discovery, it shall generate a continuous flow of discoveries at scale. The fundamental purpose behind this challenge is to uncover and reformulate the process of scientific discovery and develop a scalable system to perform it that may result in an alternative form of scientific discovery that we have not seen before.

Case studies: scientific discovery as a problem-solving

Herbert Simon argued that science is problem-solving in his article “The Scientist as Problem Solver” 18 . Scientists set themselves tasks of solving significant scientific problems. If this postulation holds, defining the problem and strategy and tactics to solve these problems is the essence of scientific discoveries.

An example of the discovery of cellular reprogramming leading to iPS cell and regenerative medicine by Shinya Yamanaka is consistent with this framework 19 , 20 . It has a well-defined goal with obvious scientific and medical implications, and search and optimization has been performed beautifully to discover cellular reprogramming capability using four transcription factors, now known to be Yamanaka factors 21 .

Another example is the discovery of conducting polymer by Hideki Shirakawa, Alan MacDiarmid, and Alan Heeger. It started with an accident that an intern researcher at Shirakawa’s Lab mistakenly used an abnormally high concentration of chemicals that formed thin film. Shikawaka noticed this accidental discovery and optimized the condition of thin-film formation. Then, with MacDiarmiad and Heeger, they identified a condition for conducting polymer formation. The initial experiments with an accidentally high dose of the chemical can be view as a stochastic search process where search space was extended beyond normal scope followed by extensive search and optimization for stable thin-film formation 22 .

The very simplified processes of these discoveries are shown in Fig. 1 . These examples, among many other cases, exemplify the process of scientific discovery as problems solving and typical tactics are search and optimization.

Search and optimization plays a critical role in the process of discovery. Yamanaka’s case is interesting because a search was conducted in bioinformatics followed by experiment-driven optimization that may be well suited for AI Scientist in the future.

Deep and unbiased exploration of hypothesis space as an alternative form of science

Discovery with an exhaustive search of hypothesis space is what characterizes AI Scientist. In the traditional approach, scientists wish to maximize the probability that the discovery they make will be significant under certain criteria. In other words, scientists are focusing on making significant discoveries and are not interested in the number of discoveries made. This is the value-driven approach. In the alternative approach, the system will learn to maximize the probability that discovery at any level of significance can be made without imposing any value-driven criteria. This is an exploration-driven approach that is an unbiased exploration of hypotheses and makes sharp contrast against the current practice of science. This approach subsumes the problem-solving approach because specific problems will be included in hypotheses generated by an unbiased search of hypothesis space and their verification. This means AI Scientist will generate-and-verify as many hypotheses as possible, expecting some of them may lead to major discoveries by themselves or be a basis of major discoveries. A capability to generate hypotheses exhaustively and efficiently verify them is the core of the system and it can be the engine that gives horsepower to the system.

This transition of a value-based approach to exploration-based approach driven by the unbiased search of hypotheses space may resemble a transition from the intuition-driven design of experiments to unbiased exhaustive measurements represented by omics-approach made possible with microarray and high-throughput genome sequencers, combined with bioinformatics supported by powerful computing resources. Unbiased hypotheses generation and verification will be built on the unbiased measurement of the well-established omics-approach, hypothesis generation system, a series of machine learning and reasoning systems, and robotics-based experimental systems. This is a logical evolution of the modality of science where a vast hypothesis space is searched in an unbiased manner rather than depending on human intuition.

A potential argument against this approach is that such a brute force approach is too inefficient and may not lead to any significant discoveries. Furthermore, one may argue that asking the right question is most important in science rather than brute force exploration. It is interesting to note that in the early days of AI research, it was widely accepted that a brute force approach would not work for complex problems such as chess, and that heuristic programming was essential for very large and complex problems 23 . The actual history of AI clearly demonstrated massive computing and machine learning is the key to success as seen in DeepBlue 24 and AlphaGo 25 .

Does that lessen apply to scientific discovery? There are three notable differences that are: (1) hypothesis space in scientific discovery is vast, open-ended, and possibly infinite as opposed to huge but finite space as in most games, (2) description in science, either knowledge or data, is not well-defined and often inaccurate whereas the well-defined description of game states and records exist in most games, and (3) evaluating hypothesis is likely to be more costly and time-consuming in science due to involvement of experiments. However, these issues can be made manageable and series of technologies to make them manageable will transform science and bring it to the next stage.

The exploration of hypothesis space in scientific discovery

The hypothesis space for scientific discovery is huge and complex as opposed to very big but finite, well-defined, and monolithic state-space in games. State-space for games such as Chess, Shogi, and Go are finite, quantized, completely observable, and monolithic. For example, the game of Go is known to have a state-space complexity in order of 10 170 and a game tree complexity in order of 10 360 . Every state of the game can be fully and unambiguously describable with a set of coordinates. There is no hierarchical structure in the state space. This is not the case in scientific discovery. The size of an entire hypothesis space is infinite or undefinable. States of objects involve substantial continuous values of higher-order dimensions. Nested hierarchical structures are prevalent. While it appears to be fundamentally different, much can be learned and adapted from experiences in building AI systems for gaming.

The most recent and significant success of building AI system for a board game is AlphaGo series that beat the best human players. AlphaGo combined deep learning, reinforcement learning, and Monte-Carlo Tree Search (MCTS) to explore possible state space and game tree to learn best possible play within an explored state space 25 . State spaces are explored based on predictions of possible next moves generated by networks trained through supervised learning of past records of Game of Go (SL policy network) and MCTS expanded a search space. Reinforcement learning using self-play improves policy network and a value network is trained to properly evaluate game status. This approach enabled AlphaGo to learn how humans played, and how humans may play in the possible game state that has proximity to the past game (Fig. 2a , orange circle). AlphaGo Zero starts from a random move and learns to play purely using reinforcement learning without human knowledge 26 . Interestingly, AlphaGo Zero not only outperforms the best human players, it outperforms AlphaGo as well. This demonstrates the strength of unbiased exploration of state space as AlphaGo Zero explores an entire state space of Go where AlphaGo incrementally searches the vicinity of human play styles (Fig. 2a , green space). AlphaZero 27 and MuZero 28 further extend such approaches to be able to learn and exhibit superhuman capability in multiple different games by learning game dynamics with model-free and mode-based reinforcement learning, respectively.

Search space structures for a perfect information games as represented by the Game of GO and b scientific discovery are illustrated with commonalities and differences. While the search space for the Game of GO is well-defined, the search space for scientific discovery is open-ended. A practical initial strategy is to augment search space based on current scientific knowledge with human-centric AI-Human Hybrid system. An extreme option is to set search space broadly into distant hypothesis spaces where AI Scientist may discover knowledge that was unlikely to be discovered by the human scientist.

A part of such an approach can be applied to scientific discovery. With AlphaGo levels of approach, a set of hypotheses can be generated using a body of knowledge accumulated to date, and it can be tested against a body of knowledge for their consistency and verified experimentally (Fig. 2b , orange circle). Enhancing the level of complexity of hypothesis and automation of experimental verification, exploration can be extended to hypothesis space where it was not practical with an incremental extension of current scientific practice (Fig. 2b , blue circle). The challenge would be to implement AlphaGo Zero strategy to randomly generate hypotheses for an entire hypothesis space because the hypothesis space can be infinite and undefinable (Fig. 2b , green zone). However, practical approaches may exist to solve this issue by leveraging the intrinsic structure of problem domains.

In biomedical sciences, any biological phenomena are the result of molecular interactions. It can be a simple interaction or involve a very complex network composed of very large numbers of molecules. Interactions among cells or even individuals can be attributed to molecular interactions. Information of any kind will be received by receptors to be meaningful for a biological system, therefore converted into molecular interactions. The exploration-driven approach in biomedical science leverages such intrinsic characteristics of application domains and may start from generating and testing hypotheses for basic biological mechanisms such as molecular interactions, genetic functions, metabolic reactions, material properties, and so forth and explore them at an unprecedented scale. Since most discoveries in biomedical science are on mechanisms behind diseases or specific biological phenomena exhaustive and unbiased exploration of molecular mechanisms shall be a building block for uncovering complex mechanisms for more complex biological phenomena (Fig. 3 ).

Most discoveries in biology and medicine are concerned with the identification of mechanisms behind important biological processes. It can be fundamental processes such as cell cycle and cellular reprogramming or clinically relevant processes such as mechanisms of disease outbreak and progression. In many cases, this basic structure will be nested into multiple levels. It should be noted that “Molecular mechanisms” are biological processes by themselves, thus multi-layer construction of this basic structure of discovery are inevitable.

Therefore, it is reasonable to assume that the first stage of the project focuses on the hypothesis of a particular form, rather than unlimited and complex forms, specifically to identify molecular mechanisms behind biological processes. By focusing on the specific type of canonical form of knowledge, the problem is now relatively well-defined which is important as an opening game of the challenge. While the omics-approach uncovered massive data on genomes, transcriptomes, metabolomes, and interactomes, detailed and exhaustive characterization and precision measurements using low-throughput methods are required to verify specific molecular characteristics and nature interactions. Such processes are generally time-consuming and often not automated thus experiments are performed only for high priority targets. Automating such processes to match omics-scale enables exhaustive search and verification of a broader range of hypotheses, which shall lead to discoveries with high-impact biomedical and biotechnology applications.

There are pioneering works to turn this idea into reality. Adam, the first closed-loop system for scientific discovery, is designed to execute the discovery of orphan enzymes in budding yeast 11 , 13 . Eve was designed to perform an automated drug repositioning screen for neglected diseases and identified TP-470, originally developed as an angiogenesis inhibiting anti-cancer drug for its irreversible binding to methionine aminopeptidase-2, to be as effective as an anti-Malaria drug as a DHFR inhibitor 29 . These systems automated low-throughput assay processes and enabled exhaustive verification based on the hypothesis generated. These systems are highly automated, but not autonomous, as the problem to be solved and the process are fully designed by humans to the detail. These are special-purpose machines optimized for specific types of problems.

This process can be applied iteratively (Fig. 4a ). A biological process in questions (h 1 ) may be explained by hypothesis h 2 or a combination of h 3 and h 4 , where h 2 , h 3 , and h 4 may have possible underlying molecular mechanisms of h 5 and h 6 , h 6 and h 7 , and h 8 , respectively. In such a case, a process of hypothesis generation and verification will be performed iteratively to verify or reject h 1 with a verified supporting mechanism either h 2 , h 3 , and h 4 . Generating experimental protocols and executing them is rather straightforward.

a Each hypothesis is dependent upon other hypotheses that are related to molecular mechanisms. b A hypothesis in question can be verified only at the system-level analysis of molecular interaction network behaviors, c a set of hypotheses and their dependency tree where each element is also a set (e.g., \(H_1 = \left\{ {h_1^0,h_1^1,h_1^2, \cdots ,h_1^n} \right\}\) ). In massive and exhaustive search of hypothesis space, a set of hypotheses, rather than a single hypothesis, is generated to cover specific hypothesis space and verified.

There are cases where a biological process can be only understood from a system dynamics perspective such as bifurcation and phase transition. A simple application of the iterative procedure to identify molecular mechanism is not sufficient. It requires reconstruction of molecular interaction network and analysis of their dynamical behaviors possibly underlying the process in question (Fig. 4b ). This is more challenging as it requires the generation of hypothesis that link biological process with mathematical concepts and verifying them through experimental verification of network behaviors and molecular mechanisms composing the network.

Exhaustive exploration of hypothesis means a set of hypotheses is generated and verified rather than a single hypothesis. Thus, nodes of a hypothesis dependency tree in Fig. 4 shall be sets, rather than an element (Fig. 4c ). Experiments shall be executed to verify an entire set of hypotheses, and protocols enabling such a process shall be generated. This may also include the generation of sub-hypothesis to be verified (Fig. 5 ).

A hierarchical generation of hypothesis sets and data to verify them will be automatically generated and executed. Verification of Hypothesis set C requires both Hypothesis sets A and B to be verified. Verification data for Hypothesis sets A and B shall be obtained from experiments in general. In general, multiple data sets are required to fill various parameters of elements in Hypothesis set before finally tested in the verification process. This requires Data Set 1 for Hypothesis set A, and Data Sets 2 and 3 for Hypothesis set B need to be collected. Data sets 1, 2, and 3 can be obtained from databases, or through automated experiments. Verified Hypothesis sets A and B mean a set of elements of Hypothesis sets A and B that are verified to be true or entire sets with a score for each element. Given the hypothesis set to be verified, this process automatically generates hypothesis sets that need to be verified first and specifies the data sets required.

The value of exhaustive generation of hypothesis and verification is in its potential capability to overcome the horizon problem. Assume that hypotheses are generated to maximize the expected significance of the discovery and tuned to focus highly expected value, such a strategy would work when the landscape is monotonically increasing (Fig. 6a ). However, it may avert exploring paths to significant discovery, when a series of discoveries precondition to the significant one was low in expected significance (Fig. 6b ). Searches may be terminated before reaching a significant discovery that may be located over-the-horizon. The landscape of discovery significance may be complex and non-monotonic. By enabling exhaustive exploration of hypothesis space, AI Scientist can go beyond the area that is over-the-horizon without it. At the same time, AI Scientist is not free from resource limitation. One of the most important areas of research would be to find out how to sample hypothesis space to effectively identify its landscape. Machine learning-guided experimental design was shown to be effective in chemistry 30 and some of the principles can be applied here.

a The landscape is monotonic, and b the landscape is non-monotonic. A simplified illustration on why there are cases that research outcomes not immediately recognized to be significant lead to a major discovery. “Estimated significance of discoveries” is used only as a conceptual index. There is no rigid method to estimate the significance of the discovery. The numbers of citations and their temporal changes can be an interesting index, but it may be biased toward short-term popularity unless the time horizon is set appropriately.

Knowledge of the world that science deals with is composed of multiple layers of abstraction, generally corresponding to the layers of systems in the domain. Discussions so far are centered around exploring and verifying hypothesis at molecular mechanisms, although it can be complex and nested. The next step shall be to uncover more complex phenomena and their dynamics that are interlinked with multiple layers of interaction, cells, organs, and individuals. This level further requires the identification of design principles and concepts behind complex systems (Fig. 7a ). As discussed already, system dynamics play a central role in discoveries of this level, hence mathematical concept shall be linked to biological processes (Fig. 4b ). At the same time, actual biological systems are constrained by fundamental principles such as biochemistry and physics, systems principles such as feedback theory and information theory, selected through evolution and manifested in the context of the environment it lives (Fig. 7b ). AI Scientist need to learn what are possible and impossible and what possible biology exist at present, and this could be potentially similar to learning models of game dynamics 28 , but in open-ended and a highly complex environment.

a In scientific discovery, knowledge is layered from tangible knowledge to conceptual knowledge. Properties of molecules and their interactions are tangible and knowledge of systems dynamics and design principles are conceptual as they are not directly associated with tangible objects such as molecules and cells. Conceptual knowledge is often backed by mathematical and system-oriented theories. b Biology that we observe (“existing biology”) is constrained by multiple factors such as fundamental principles, systems principles, environmental constraints, and evolutionary selection. “Possible biology” meets all constraints but has not been observed or realized yet.

The challenge is to find a way for the system to generate and test conceptual level hypotheses and test them. This shall be done in an unbiased manner. Methods such as model-based reinforcement learning and generative adversarial learning can be applied initially to investigate how to develop system that learn laws of nature at scale. Some recent studies demonstrate deep learning networks trained over millions of articles generate extensive molecular interactions 31 and the potential relationship between molecules and disease only using articles a year (or years) before such a relationship was discovered 32 . Deep learning was also used to uncover hierarchical structure and functions of cells 33 , deep generative models for discovering hidden structures 34 , precision phenotyping to predict genetic anomalies 35 , and many more. The outcome of such approaches is a set of hypotheses generated by deep learning and other AI methods from unbiased data, and hypotheses are generated in an unbiased manner. Such predictions can be a basis for the search for in-depth molecular relationships and functions. Furthermore, recent success in the Ramanujan Machine 36 in mathematics and the project Debater 37 in adversarial reasoning augmented possible approach that can be incorporated in the hypothesis generation process. Qualitative physics offers the opportunity to generate, match, and explain physical and mathematical concepts such as bifurcation and phase transition 38 , 39 . Combined with the capability of deep learning neural network to learn, classify, and generate non-linear dynamics 40 , 41 , qualitative physics approach can be a powerful method for hypothesis generation and verification at the level of dynamical system concept. There are studies to use qualitative physics for biological processes 42 , 43 . This illustrates the potential of AI to be able to generate conceptual model exhaustively, assemble basic knowledge to be consistent with the conceptual model, and experimentally verify them. This approach essentially forces us to create a set of possible substructures of systems and search for structural matching with reality. With unbiased exploration at this level, AI Scientist shall be capable of exploring the complex dynamical system and may be able to discover new knowledge that is less likely to be discovered by human scientists.

The multiverse of knowledge in scientific discovery

Generating hypotheses and maintaining a set of consistent body of knowledge in science is a formidable task due to the vast number of hypothesis generated and maintained, complexity and non-monotonicity, and unreliability of knowledge and data published. While publications and data already available today will be the initial foundation of hypothesis generation, the problem is that this initial foundation is not necessarily a solid ground; they contain substantial errors, missing information, and even fabrications. Manually checking statements with misinterpretation and biased interpretation of data individually and exhaustively is not practical given the volume of publications that shall be processed by AI Scientist. Currently, certain types of experimental results are known to be difficult to reproduce 44 , where some aspects of reproducibility issues shall be reduced by automated, transparent, and traceable experimental systems 45 , 46 , 47 . Intrinsic variability of biological systems due to noise and individual variations are treated as intrinsic features 48 , 49 and shall be treated separately from ambiguities and inaccuracy caused by the process of research itself. Aside from the immediate reproducibility problem, some observations may hold true in some contexts but may not be applicable in the other context as an intrinsic nature of the complex system. Scientific knowledge is probabilistic and non-monotonic and a representation system shall be able to reflect this reality 50 . Knowledge shall be contextualized, and a new context can be added incrementally. While this is a nature of scientific research, this poses a serious issue in the computational process as hypothesis will be generated using a body of knowledge that is sure to be revised constantly. It is like making reasoning in the twilight zone where what is correct or not is always ambiguous. In this regard, hypothesis and knowledge cannot be clearly distinguished. It is a matter of degree of confidence. Verification in the context of inductive reasoning means that “a certain hypothesis is still surviving against all falsifiability challenges, thus considered most likely so far”. This implies for all hypotheses, survived or not, the trace of tests and their outcome need to be recorded. Unless errors are obvious, every statement in publications shall be converted into knowledge graph and the knowledge graph shall be constantly updated (Fig. 8 ). Obviously, inconsistencies will emerge which will trigger forks of knowledge graph, each of them consistent internally. Whenever some of the assumptions are altered, the relevant hypothesis shall be automatically re-evaluated. This can be accomplished by maintaining a very large number of multiple consistent set of knowledge and data with explicit breaking point which set to be considered more probable. Truth maintenance system, brief revision system, and non-monotonic reasoning can be applied to maintain consistency with multiple contexts 51 , 52 , 53 . Given the nature of scientific knowledge that is essentially probabilistic, multiple sets of knowledge graph shall be maintained persistently, unless sets are proven to be inconsistent, and likelihood of each knowledge set to be most probable change dynamically.

The original knowledge graph (KG 1 0 ) is split into two incompatible KGs (KG 1 1 and KG 1 2 ) with the addition of a new data “d1” and associated arguments. Further addition of data “d2” resulted in the additional split of KGs. Data d3 and associated argumentation contextualized conflicting interpretations separating two KGs (KG 1 4 and KG 1 5 ) that resulted in the merger of them. Such a merge happens when KG 1 4 and KG 1 5 are not compatible due to conflicting interpretation of data d2, but data d3 and associated argumentation clarified conflict can be resolved that two interpretation of data d2 is context-dependent thus both interpretations hold in a different context. For two competing KGs (KG 1 6 and KG 1 7 ), d4 and associated argumentation eliminated KG 1 7 , and KG 1 6 survived and augmented to be KG 1 10 .

To evaluate the probability and possibly eliminate inconsistent knowledge sets, mechanisms to resolve ambiguities and falsify hypothesis need to be implemented. When a hypothesis is generated and verified, it always associated with data and justification why data support or reject the hypothesis. Theory of argumentation 54 and non-monotonic reasoning shall be the basis of argumentation structure generation and processing 55 , 56 . Hypothesis, or claim, can be rejected or needs further delineation with multiple cases such as (a) data is fabricated, inaccurate, or incomplete, or (b) interpretation of data/assumption is not sufficient to justify the hypothesis, (c) scope of the hypothesis shall be limited, and (d) effective rebuttle exists that denies reasoning connecting data/assumptions and the claim. It is possible that reasoning presented in publications are insufficient to justify hypothesis, and detailed justification may need to be re-generated or argument against the claim to be created to make knowledge set complete. Recent progress in computational debate may be a first step to implement mechanisms to generate such argumentations 37 , 56 , 57 . The argumentation module shall generate argumentation to support or falsify existing hypothesis thereby justification can be strengthened through additional experiments and reasoning. At the same time, argumentation generated need to be understood by the human scientist. Qualitative simulation 38 , 58 , 59 , 60 should be able to generate qualitative explanations consistent with human reasoning 39 . A closed-loop system involving such a process shall be developed that can incrementally improve confidence and consistency of knowledge and data thereby incrementally building up rigidly fortified data, argumentations, and hypotheses.

Challenges in technology platform: automation, precision, and efficiency

Development of high precision, fast, and low operation cost experimental system and data analysis system is mandatory for this challenge. Unbiased search of hypothesis space means an unprecedented number of hypotheses will be generated and tested. The test requires both computational and experimental tests. The volume of experiments required to execute unbiased exploration would be a magnitude larger than current scientific practices. The revolutionary precise, cost-effective, and fast experimental systems need to be developed and deployed. Since the cycle of hypothesis generation and verification is the rate-limiting factor of the entire process, how fast and accurately perform experiments will determine the chance of success of the challenge. Some experiments will involve hypothesis exploring unusual conditions such as 1000 times off from the conventional parameters such as the concentration of chemicals. The first step would be to make laboratories fully connected and automated. Then, equipment will be replaced over time for high precision and efficient devices including microfluidics, followed by the use AI modules for each process before reaching the high level of autonomy expected in the AI Scientist.

Therefore, experimental systems shall be less resource-demanding and accurate yet reliable, reproducible, and integrated. While automation of various experimental processes has been commercialized already these are fragments of an entire process. The challenge requires an entire process of various types of experiments to be automated and part of such system may be installed as robotics cloud laboratory 46 . Recently, a robotics experimental system successfully identified proper condition for cell culture of medical-grade iPS-derived retinal pigment epithelial (RPE) cells after searching 200 million possible parameter combinations through Bayesian optimization with local penalization 47 . Optimization of lycopene biosynthesis pathway and biofuels for synthetic biology-based bio-manufacturing are other examples automated search of design space was shown to be effective 61 , 62 . Such success demonstrates the introduction of robotics-AI systems for each process shall improve the quality and efficiency of experiments. Automated closed-loop system impacts synthetic biology as well due to its quality and reproducibility 63 . Currently, only a system closed-the-loop of hypothesis generation and experimental verification is on budding yeast genetics 13 . Variety of experiments and their complexity shall be significantly augmented to cope with an extensive set of hypotheses to be verified. A literature analysis of over 1628 papers indicates 86–89% of experimental protocols in these papers can be automated by readily available commercial robotics systems 64 . This implies the progress can be quick initially, and what matters will be how to integrate different processes, data management, and how to automate processes that are not automated at this moment as well as novel protocols in future.

To achieve this, precise process management shall be imposed for the flows of control, materials, data, and physical agents. Due to vast numbers of experiments required for verification, experimental systems shall be compact and requirements for experimental samples and reagents shall be minimized. Organs-on-Chips is a recent addition to technologies that can reproduce experimental context closer to in vivo condition while maintaining controllability, traceability, and requires smaller amounts of experimental materials 65 , 66 . A novel origami-inspired surgical robot has interesting characteristics of being compact and high precision that can be applied for a range of experiments 67 . In future, the combination of microfluidics and robotics system will be used extensively in biological experiments to meet the demands of large numbers of experiments and requirements for controllability, accuracy, and traceability 68 .

Experimental devices shall be controlled by a platform that combines software tools, data access, and experimental systems embedded in the closed loop. Machine learning-guided experimental design was shown to be effective in chemistry 30 and some of the principles can be applied to broader domains. Some of the technological platforms are readily available today, as seen in Garuda Connectivity and Automation Platform 69 , Wings workflow management tool 70 , 71 , and DISK Data Analysis and Hypothesis Evolution framework 72 , but many have to be developed as a part of the technology challenge. Extensive efforts are made to develop bioinformatics and systems biology analysis and modeling software and data standard that are fundamental to obtain data, analyze them properly, make accurate curation, and enabling dynamical simulations. Annual workshops such as COMBINE and HARMONY drive the development and adaptation of standards ( http://co.mbine.org/home ). Interoperability of software and data is mandatory to ensure connectivity of laboratory that is essential to automation of not only experimental processes but also analysis and modeling processes. More effort shall be made on the representation of hypothesis and knowledge reflecting the reality of scientific knowledge.

Evolving relationship between AI Scientists, human scientists, and society

How does AI Scientist evolve and transform scientific activities? It is clear superhuman-AI Scientist would not emerge out of blue. It will co-evolve with the scientific community over time. A possible, and logical, evolutional path of AI Scientist is to increase the level of automation first, followed by the increase of autonomy level (Fig. 9 ). Most current use of AI for research is a tool for specific tasks such as image classification, text-mining, and other tasks that are isolated and fully instructed by the human scientist. This is an AI tool stage. An early stage of AI scientist will take a form of a group of useful and highly competent software, including hypothesis generation module, and robotics to execute complex but pre-define tasks as instructed. Robot Scientist Adam and Eve are pioneering examples of this stage. Increasing repertoire of experiments and complexity of hypothesis are the next step. Substantial investment and user feedback are essential to make such systems useful and widely adapted.

AI Scientist requires a highly automated and connected laboratory to be able to design and execute experiments, as well as extensive access to databases and publication archives to process, extract, and evaluate current knowledge. Sophisticated laboratory automation is mandatory. Robot Scientist, Adam & Eve, is highly specialized automation with a certain level of intelligence for hypothesis generation and experimental protocol generation. The next step is to fully automate and connect laboratory equipment with layers of control for data flow, material flow, and physical control flow. Numbers of AI assistants shall be installed for each task initially, but need to be integrated as an integrated and highly autonomous system. The transition of automated system to autonomous system will be one of the most challenging part of the initiative.

Evolutionary pressure imposed on AI Scientist is whether it will be used by human scientists and widely adopted. Investment to develop AI Scientist, either by public funding or private sources, will be driven by the utility of such systems for human scientists. Therefore, AI Scientist will be inevitably interlocked with the research ecosystem of human scientists, and highly competent and user-friendly systems will survive for further development. This path inevitably make AI Scientist designed to be highly interactive with human scientists. Researchers will quickly understand the value and the power of AI Scientist, and will soon start asking questions that require the exhaustive generation of hypotheses and verification that exploits the full potential of AI Scientist at each stage of evolution. This will trigger the transformative change in biology as we witness in genomics when the unbiased measurement of genome sequence and transcriptome uncovered new realities in biology such as non-coding RNA 73 , 74 . Even without large-scale experiments, hypothesis generation capability of AI Scientist shall help researchers to explore hypotheses that may not be considered without such AI Scientist as well as being an extremely effective dialog-based creativity and discovery support system. Institutions without AI Scientist will no longer be competitive in science and technology.

With an increasing level of autonomy, AI Scientists are expected to make an autonomous decision on what to investigate next. While mechanisms to make it possible is yet to be seen, multiple strategies can be considered such as (1) goal-oriented approach of defining very high-level goals and find multiples paths to best achieve such goals or (2) bottom-up approach of exploring hypothesis search space based on discoveries already made by a specific AI Scientist. In either case, questions to be asked can be automatically extracted from publication, defined by human researchers, or randomly generating questions to be answered.

With an increased level of connectivity and flexibility to generate hypotheses and their verification process, instruction from human scientist will be more abstract, and AI Scientist will have an increased autonomous process to make decision of priorities of hypotheses to be tested and experimental protocols to be performed. This is a semi-autonomous stage because instruction on what to investigate is provided from outside although how to investigate them may be generated internally by AI Scientist. The level of abstraction of instruction by Human scientists need to be carefully chosen so that AI Scientist can execute the task with success. Instruction such as “find a set of protocols (transcription factors, chemicals, procedures) that can transform types of somatic cell (X) into defined cell types (Y)” is a difficult but tangible one. For such an instruction, multiple experimental protocol shall be generated, and prioritize choice of source and target cell types, interventions to use tested, and analysis procedures. However, much higher goals, such as “cure cancer”, “increase human life span to 150 years”, or “minimize climate change” would be problematic as some of these goals are too abstract, at least at the initial phase of AI Scientist. With the evolution of AI Scientist over years, some these questions may be addressed in future, still it requires user understanding on capability of AI Scientist to utilize its power.

With potentially expensive running cost of AI Scientists, especially when large-scale experiments are required, a certain level of monitoring will be enforced in most institutions. AI Scientist may include a function to generate questions more relevant to its owner or to the society. At least, it is highly plausible larger investment will be made to deliver high return on investment outcome. In this case, the choice of problem and evaluation of the significance of discoveries will reflect human-centric value system, most specifically the value of the stakeholders.

However, AI Scientists under this circumstance are less likely to make unexpected discoveries since the problems to be solved are pre-defined. Researchers with a priori expectations may sometime miss the big picture when one without such expectation may notice 75 . There are many cases discoveries initially received minor attention led to major discoveries later. It is extremely difficult, if not impossible, to evaluate the significance of the discovery when a few more discoveries may be needed to translate the discovery into high-impact outcome due to the over-the-horizon problem. The real value of AI Scientist is its capability to explore hypothesis space magnitude more efficiently into seemingly low-value domains with expectation that may eventually leads to major outcomes. Such systematic explorations into seemingly low-value hypothesis space are infeasible to be performed by human scientists. Both aspects of discovery are important that implies two roles for AI Scientist can be assumed that are “AI Scientist as a Problem Solver” aligned with the value of the stakeholders and “AI Scientist as an Explorer” that boldly explore hypothesis space nobody have gone before. However, in either cases, exhaustive hypothesis generation and verification will be the core of the AI Scientist that distinguishes it from the traditional approach.

AI Scientist will be a multiplexed multi-agent system generating multiple instances of AI Scientist (Fig. 10 ). It is comprised of many software and hardware agents (highly functional modules with a certain level of autonomy) with a high level of interactions, interoperability, and scalability in problem size and complexity. There may be two characteristics for an architecture of AI Scientist.

They evolve, merge, and interact with humans. Human experts can be a part of the system as human-in-the-loop system. Scientists who wish to work with AI Scientist are most likely to work with instances of AI Scientist.

First, it may be a multiplexing multi-agent system. It is possible multiple instances of AI Scientist are created each specialized in a certain area extensively exploring hypothesis space organically. They are almost identical in components but differ in hypothesis space exploring. Communication among AI Scientists may enable them to merge discoveries for further exploration. This may take a form of communication between AI Scientists through a series of inquiries or the creation of a new synthetic new AI Scientist. Therefore, AI Scientist as a whole entails multiple instances of AI Scientist with focused areas. In this case, discoveries may be made systematically centered around initial core domains and eventually as a combination of multiple domains forming specific path-dependencies in discoveries. In the community of AI Scientist, a series of discoveries and publications made by AI Scientist may resemble that of the successful scientist. Interaction between AI Scientists is equivalent to the search and exchange of new knowledge and style of discovery specialized by each AI Scientist. When the critical mass of knowledge and data is required to generate significant hypothesis combining multiple domains, forming the community of AI Scientist would make sense. The discovery of CRISPR–Cas9 may be one of the examples of revolutionary discoveries coming from the combination of basic research seemingly distant areas of research 76 . It is well recognized that many discoveries considered groundbreaking was triggered by connecting two or more seemingly unrelated ideas. If AI Scientist shall be able to make discoveries of this nature, it must be able to access and connect very broad and less related domains where there is already a sufficient accumulation of knowledge and data by each AI Scientist.

Second, it may be a human-in-the-loop system. From AI Scientist perspective, agents composing them do not have to be exclusively software or hardware, it can be human expert as far as it can interface with the rest of the system. Human experts can be in the role of domain experts or in the commanding and monitoring role. The commanding and monitoring role is important to avoid misuse of the system. Potentially, AI Scientist can make discoveries that are harmful to human and our planet. What to discover fully depends on how the owner uses such capability. A strict ethical guideline and enforcement may be required with increased level of autonomy of AI Scientist. Ultimately, it will impact national security of the highest level.

There will be multiple AI Scientists either by institutions, academic community, country, or other societal boundaries. Some of them may communicate each other, some may be configured in isolation, and some would form local networks. Such configuration may be decided based on ownership of data and intellectual properties generated. Some modules and databases will be publicly maintained, and some would be proprietary. Although the level of autonomy can be very high, intellectual property is still retained with human researchers who run this system because human researchers make the decision to run AI Scientist and monitor their progress, and are responsible for the outcome. Some institutions may run AI Scientist at free-run modes with very high levels of autonomy, to let it explore hypothesis space that human researchers may not think of. Even in such a case, completely autonomous may not be achievable, as the intention of the owner will influence the running of the system.

AI Scientist to transform systems biology and broader area of sciences

Automating the process of hypothesis generation and verification shall transform broad areas of sciences. Systems biology is one of the representative areas most affected by such technology, not only because it enables researchers to cope with massive data and publications otherwise under-utilized, but also it enables researchers to develop large-scale high-precision models as well as performing investigation significantly broader in scope and more extensively in parameter space than current approaches.

One of the initial expectations of systems biology was to develop a high-precision large-scale model of biological systems such as virtual humans that can be used as digital-twin of patients with in-depth molecular mechanisms behind 77 , 78 . While it is a holy grail of systems biology, it was proven to be extremely challenging as anticipated. There are fundamental difficulties for such a task partly due to the limitation of our cognitive capabilities and sociological constraints 14 . The research landscape of systems biology is clustered around two modalities that are; (1) a high-precision mechanistic model for the smaller and tractable system and (2) a large-scale network model based on omics data but less on detailed mechanisms. There is an inherent trade-off between these two modalities and attempts to overcome such trade-off have fallen short of expectations. First, there are human cognitive constraints. A vast amount of data and complexity of the system often goes beyond human comprehension and non-linear nature of biological process make things more difficult. Second, there are practical constraints as well. Building a large-scale precision model requires details of almost every interaction and molecular behavior to be investigated both computationally and experimentally which is beyond the capability of most research groups. Investigating each of such interactions and molecular behavior would require major efforts while many of them may not result in immediate major discoveries by itself. While interesting discoveries shall spring out from some of such efforts, tasks are designed to fill in every detail of a large model, rather than speculating the potential importance of interactions and molecules. It is not practical to assume dedicated efforts by members of the research group to be sustainable for many years unless most of such process is automated.

Perhaps, systems biology, particularly studies for large-scale precision models, is not a research field for human alone to investigate as possible causes of difficulties lies in human cognitive and sociological limitations. Once we accept the reality such a trade-off is inherent in human cognitive limitations and sociological constraints, the path to overcoming the trade-off is obvious. It is a field suitable for AI or AI-human hybrid system. Building high-precision large-scale models and efficiently exploit such models and aggregated knowledge to back it up requires powerful AI systems to support our scientific activities.

The AI system is not only useful for building large-scale in-depth models but will exhibit its power to discover new mechanisms and principles we have not imagined as well as discovering novel drug targets efficiently with a significantly extended search of target candidates. Extensive use of AI for drug discovery has been discussed with the implication of dramatically improving its efficiency and the transformation of the process 79 . Early successful cases including rapid identification of kinase inhibitor for DDR1 are encouraging 80 . A recent success of AlphaFold represents how AI technologies impact biomedical studies 81 . Studies on the relationship between drug target proteins and numbers of interactions of proteins demonstrate there is a low but reasonable probability that proteins with small numbers of identified interactions to be drug targets 82 . Although chances each protein can be a drug target may be small because the total numbers of such proteins are huge, exhaustive search of this class of proteins may result in abundant novel drug targets. With the same issues that arose in high-precision large-scale models, automation of the research process is essential to explore such opportunities.

Extending such an approach to synthetic biology to automate design and verification processes 63 , 83 , 84 .

Ultimately, a series of new discoveries will be integrated into an integrated model that is large-scale, high-precision, and in-depth. The implication is massive. It does not only mean researchers use AI Scientist as one of the tools, but it implies the practice of scientific discovery will be transformed dramatically with AI Scientist because discoveries will be made at scale and autonomously. At the same time, this will be a golden opportunity for systems biology since it will transform system biology into the next stage.

AI Scientist can be transformative not only in life science but also for broader science and technology domains. This is especially the case that requires hypothesis generation and verification to broader range parameter search of chemical synthesis and material discovery. Already, there are emerging interests in chemistry and material science for automation of experiments coupled with machine learning guide experimental design at various levels 30 , 46 , 85 , 86 , 87 , 88 , 89 , 90 . The idea of massive search of hypothesis space and verification applies to these domains as well. However, if such efforts can be applied to the discovery of novel concepts are yet to be seen. Recently, The Ramanujan Machine was announced for automated generation of conjectures in mathematics 36 . The Ramanujan Machine added a new perspective as it is not a parameter search and extensive generation of conjectures. With the rapid advances in robotics, sensors, AI with the increasing availability of computing powers, AI Scientists for broader domains of science will be inevitable. Research institutions without such capability will no longer be competitive in the coming decade.

The Nobel Turing Challenge is the ultimate challenge for AI and systems biology. Any progress toward achieving the goal will generate high utility technologies that shall accelerate science. Due to its breadth of expertise required and possible length of duration to achieve the goal, it may best be organized as a virtual big science 91 . Once the initiative taking off, it will uncover the essence of scientific discovery, and results in the creation of an alternative form of science. AI Scientist and human scientists will work together to solve formidable problems and to explore new intellectual territories where no one have gone before.

Popper, K. The Logic of Scientific Discovery (Taylor & Francis, 1959).

Kuhn, T. S. The Structure of Scientific Revolution (University of Chicago Press, 1962).

Lakatos, I. The Methodology of Scientific Research Programmes (Cambridge University Press, 1978).

Feyerabend, P. Against Method: Outline of an Anarchistic Theory of Knowledge (Humanities Press, 1975).

Shapiro, E. Inductive Inference of Theories From Facts (Yale University, 1981).

Lindsay, R., Buchanan, B., Feigenbaum, E. & Lederberg, J. DENDRAL: a case study of the first expert system for scientific hypothesis formation. Artif. Intell. 61 , 209–261 (1993).

Article   Google Scholar  

Langley, P. & Simon, H. Scientific Discovery: Computational Exploration of the Creative Processes (The MIT Press, 1987).

Lenat, D. & Brown, J. Why AM and EURISKO appear to work. Artif. Intell. 23 , 269–294 (1984).

Gil, Y. & Hirsh, H. Discovery Informatics: AI Opportunities in Scientific Discovery (AAAI, 2012).

Gil, Y., Greaves, M., Hendler, J. & Hirsh, H. Artificial Intelligence. Amplify scientific discovery with artificial intelligence. Science 346 , 171–172 (2014).

Article   CAS   PubMed   Google Scholar  

King, R. D. et al. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature 427 , 247–252 (2004).

King, R. D. et al. Make way for robot scientists. Science 325 , 945 (2009).

King, R. D. et al. The automation of science. Science 324 , 85–89 (2009).

Kitano, H. Artificial intelligence to win the nobel prize and beyond: creating the engine for scientific discovery. AI Mag. 37 , 39–49 (2016).

Google Scholar  

Turing, A. M. Computing machinery and intelligence. Mind 59 , 433-460 (1950).

Feigenbaum, E. Some challenges and grand challenges for computational intelligence. J. ACM 50 , 32–40 (2003).

Nakamoto, S. Bitcoin: A Peer-to-Peer Electronic Cash System , http://www.bitcoin.org/bitcoin.pdf (2008).

Simon, H. A. in Complex Information Processing: The Impact of Herbert A. Simon (eds Klahr, D. & Kotovsky. K.) 375–398 (Lawrence Erlbaum Associates, Publishers, 1989).

Takahashi, K. & Yamanaka, S. Induction of pluripotent stem cells from mouse embryonic and adult fibroblast cultures by defined factors. Cell 126 , 663–676 (2006).

Takahashi, K. et al. Induction of pluripotent stem cells from adult human fibroblasts by defined factors. Cell 131 , 861–872 (2007).

Yamanaka, S. The Nobel Prize in Physiology or Medicine 2012 — Shinya Yamanaka - Biographical , https://www.nobelprize.org/prizes/medicine/2012/yamanaka/biographical/ (2012).

Shirakawa, H. The Nobel Prize in Chemistry 2000 — Hideki Shirakawa - Biographical , https://www.nobelprize.org/prizes/chemistry/2000/shirakawa/biographical/ (2000).

Feigenbaum, E. & Feldman, J. Computers and Thought (McGraw-Hill Book Company, 1963).

Campbell, M., Hoane, J. Jr. & Hsu, F.-H. Deep blue. Artif. Intell. 134 , 57–83 (2002).

Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529 , 484–489 (2016).

Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550 , 354–359 (2017).

Silver, D. et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362 , 1140–1144 (2018).

Schrittwieser, J. et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature 588 , 604–609 (2020).

Williams, K. et al. Cheaper faster drug development validated by the repositioning of drugs against neglected tropical diseases. J. R. Soc. Interface 12 , 20141289 (2015).

Article   PubMed   PubMed Central   Google Scholar  

Raccuglia, P. et al. Machine-learning-assisted materials discovery using failed experiments. Nature 533 , 73–76 (2016).

Spranger, M., Palaniappan, S. & Ghosh, S. in BioNLP 2016 Vol. BioNLP 2016, 119–127 (Association of Computaitonal Linguistics, Germany, 2016).

Akujuobi, U., Spranger, M., Palaniappan, S., Zhang, X. T-PAIR: Temporal node-pair embedding for automatic biomedical hypothesis generation. In IEEE Transactions on Knowledge and Data Engineering https://doi.org/10.1109/TKDE.2020.3017687 (2020).

Ma, J. et al. Using deep learning to model the hierarchical structure and function of a cell. Nat. Methods 15 , 290–298 (2018).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Lopez, R., Gayoso, A. & Yosef, N. Enhancing scientific discoveries in molecular biology with deep generative models. Mol. Syst. Biol. 16 , e9198 (2020).

Ruderfer, D. M. & Dudley, J. T. Deep phenotyping predicts Huntington’s genotype. Nat. Biotechnol. 34 , 823–824 (2016).

Raayoni, G. et al. Generating conjectures on fundamental constants with the Ramanujan Machine. Nature 590 , 67–73 (2021).

Slonim, N. et al. An autonomous debating system. Nature 591 , 379–384 (2021).

Forbus, K. D. Qualitative modeling. Wiley Interdiscip. Rev. 2 , 374-391, https://doi.org/10.1002/wcs.115 (2011).

Forbus, K. D. Qualiatative Representations: How People Reason and Learn About the Continuous World (The MIT Press, 2019).

Raissi, M. Deep hidden physics models: deep learning of nonlinear partial differential equations. J. Mach. Learn. Res. 19 , 1–24 (2018).

Teng, Q. & Zhang, L. Data driven nonlinear dynamical systems identification using multi-step CLDNN. AIP Advances 9 , 085311 (2019).

Chaouiya, C. et al. SBML qualitative models: a model representation format and infrastructure to foster interactions between qualitative modelling formalisms and tools. BMC Syst. Biol. 7 , 135 (2013).

Langley, P., Shiran, O., Shrager, J., Todorovski, L. & Pohorille, A. Constructing explanatory process models from biological data and knowledge. Artif. Intell. Med. 37 , 191–201 (2006).

Article   PubMed   Google Scholar  

Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10 , 712 (2011).

Miles, B. & Lee, P. L. Achieving reproducibility and closed-loop automation in biological experimentation with an IoT-enabled lab of the future. SLAS Technol. 23 , 432–439 (2018).

PubMed   Google Scholar  

Yachie, N., Robotic Biology, C. & Natsume, T. Robotic crowd biology with Maholo LabDroids. Nat. Biotechnol. 35 , 310–312 (2017).

Kanda, G. N. et al. Robotic search for optimal cell culture in regenerative medicine. Preprint at bioRxiv https://doi.org/10.1101/2020.11.25.392936 (2020).

Mitchell, S. & Hoffmann, A. Identifying noise sources governing cell-to-cell variability. Curr. Opin. Syst. Biol. 8 , 39–45 (2018).

Sherman, M. S., Lorenz, K., Lanier, M. H. & Cohen, B. A. Cell-to-cell variability in the propensity to transcribe explains correlated fluctuations in gene expression. Cell Syst. 1 , 315–325 (2015).

Soldatova, L. N., Rzhetsky, A., De Grave, K. & King, R. D. Representation of probabilistic scientific knowledge. J. Biomed. Semant. 4 , S7 (2013).

Doyle, J. A truth maintenance system. Artif. Intell. 12 , 251–272 (1979).

de Kleer, J. An assumption-based TMS. Artif. Intell. 28 , 127–162 (1986).

Martinez, M. V. & Varzinczak, I., NMR-2020: Workshop Notes of the 18th International Workshop on Non-Monotonic Reasoning, (Buenos Aires and Lens, 2020).

Toulmin, S. The Uses of Argument (Cambridge University Press, 1958).

Hunter, A., Polberg, S. & Thimm, M. Epistemic graphs for representing and reasoning with positive and negative influences of arguments. Artif. Intell. 281 , 103236 (2020).

Atkinson, K. et al. Toward artificial argumentation. AI Magazine 25–36 (Fall, 2017).

Bench-Capon, T. J. M. & Dunne, P. Argumentation in artificial intelligence. Artif. Intell. 171 , 619–641 (2007).

Kuipers, B. Qualitative simulation. Artif. Intell. 29 , 289–338 (1986).

Kuipers, B. Qualitative simulation: then and now. Artif. Intell. 59 , 133–140 (1993).

Davis, E. & Marcus, G. The scope and limits of simulation in automated reasoning. Artif. Intell. 233 , 60–72 (2016).

Zhang, J. et al. Accelerating strain engineering in biofuel research via build and test automation of synthetic biology. Curr. Opin. Biotechnol. 67 , 88–98 (2021).

HamediRad, M. et al. Towards a fully automated algorithm driven platform for biosystems design. Nat. Commun. 10 , 5150 (2019).

Article   PubMed   PubMed Central   CAS   Google Scholar  

Jessop-Fabre, M. M. & Sonnenschein, N. Improving reproducibility in synthetic biology. Front. Bioeng. Biotechnol. 7 , 18 (2019).

Groth, P. & Cox, J. Indicators for the use of robotic labs in basic biomedical research: a literature analysis. PeerJ 5 , e3997 (2017).

Low, L. A., Mummery, C., Berridge, B. R., Austin, C. P. & Tagle, D. A. Organs-on-chips: into the next decade. Nat. Rev. Drug Discov. 20 , 345–361 (2020).

Wikswo, J. P. et al. Scaling and systems biology for integrating multiple organs-on-a-chip. Lab Chip 13 , 3496–3511 (2013).

Suzuki, H. & Wood, R. J. Origami-inspired miniature manipulator for teleoperated microsurgery. Nat. Mach. Intell. 2 , 437–446 (2020).

Zhong, J. et al. When robotics met fluidics. Lab Chip 20 , 709–716 (2020).

Ghosh, S., Matsuoka, Y., Asai, Y., Hsin, K. Y. & Kitano, H. Software for systems biology: from tools to integrated platforms. Nat. Rev. Genet. 12 , 821–832 (2011).

Gil, Y., Ratnakar, V., Deelman, E., Mehta, G. & Kim, J. In Innovative Applications of Artificial Intelligence (IAAI-07) (Vancouver, British Columbia, Canada, 2007).

Gil, Y. et al. Wings: intelligent workflow-based design of computational experiments. IEEE Intell. Syst. 26 , 62–72 (2011).

Gil, Y. et al. in The Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17) (The Association for the Advancement of Artificial Intelligence, 2017).

de Hoon, M., Shin, J. W. & Carninci, P. Paradigm shifts in genomics through the FANTOM projects. Mamm. Genome 26 , 391–402 (2015).

Abugessaisa, I. et al. FANTOM enters 20th year: expansion of transcriptomic atlases and functional annotation of non-coding RNAs. Nucleic Acids Res. 49 , D892–D898 (2021).

Yanai, I. & Lercher, M. A hypothesis is a liability. Genome Biol. 21 , 231 (2020).

Doudna, J. & Sternberg, S. A Crack in Creation: Gene Editing and the Unthinkable Power to Control Evolution (Houghton Mifflin Harcourt, 2017).

Jones, D. All systems go. Nat. Rev. Drug Discov. 7 , 128–129 (2008).

PricewaterhouseCoopers. Pharma 2020: Virtual R&D --- Which Path Will You Take? (2007).

Schneider, P. et al. Rethinking drug design in the artificial intelligence era. Nat. Rev. Drug Discov. 19 , 353–364 (2020).

Zhavoronkov, A. et al. Deep learning enables rapid identification of potent DDR1 kinase inhibitors. Nat. Biotechnol. 37 , 1038–1040 (2019).

Senior, A. W. et al. Improved protein structure prediction using potentials from deep learning. Nature 577 , 706–710 (2020).

Hase, T., Tanaka, H., Suzuki, Y., Nakagawa, S. & Kitano, H. Structure of protein interaction networks and their implications on drug design. PLoS Comput. Biol. 5 , e1000550 (2009).

Appleton, E., Madsen, C., Roehner, N. & Densmore, D. Design automation in synthetic biology. Cold Spring Harb. Perspect. Biol. 9 , https://doi.org/10.1101/cshperspect.a023978 (2017).

Appleton, E., Densmore, D., Madsen, C. & Roehner, N. Needs and opportunities in bio-design automation: four areas for focus. Curr. Opin. Chem. Biol. 40 , 111–118 (2017).

MacLeod, B. P. et al. Self-driving laboratory for accelerated discovery of thin-film materials. Science Adv. 6 , eaaz8867 (2020).

Häse, F., Roch, L. M. & Aspuru-Guzik, A. Next-generation experimentation with self-driving laboratories. Trends Chem. 1 , 282–291 (2019).

Article   CAS   Google Scholar  

Stein, H. S. & Gregoire, J. M. Progress and prospects for accelerating materials science with automated and autonomous workflows. Chem. Sci. 10 , 9640–9649 (2019).

Tabor, D. P. et al. Accelerating the discovery of materials for clean energy in the era of smart automation. Nat. Rev. Mater. 3 , 5–20 (2018).

Nikolaev, P. et al. Autonomy in materials research: a case study in carbon nanotube growth. npj Comput. Mater. 2 , 16031 (2016).

DeCost, B. L. et al. Scientific AI in materials science: a path to a sustainable and scalable paradigm. Mach. Learn Sci. Technol . 1 , https://doi.org/10.1088/2632-2153/ab9a20 (2020).

Kitano, H., Ghosh, S. & Matsuoka, Y. Social engineering for virtual ‘big science’ in systems biology. Nat. Chem. Biol. 7 , 323–326 (2011).

Download references

Acknowledgements

I would like to thank members of the Systems Biology Institute, Okinawa Institute of Science and Technology Graduate School, Sony CSL, and Sony AI for fruitful discussions. Ross King and Yolanda Gil for working together in starting the challenge in reality. Special thanks to Ed Feigenbaum for insightful discussions and encouragements. This work was supported, in part, through an Office of Naval Research Global (ONRG) grant awarded to the Alan Turing Institute.

Author information

Authors and affiliations.

The Systems Biology Institute, Tokyo, Japan; Okinawa Institute of Science and Technology Graduate School, Okinawa, Japan; Sony Computer Science Laboratories, Inc., Tokyo, Japan; Sony AI, Inc., Tokyo, Japan; and The Alan Turing Institute, London, UK

Hiroaki Kitano

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Hiroaki Kitano .

Ethics declarations

Competing interests.

The author is Editor-in-Chief of npj Systems Biology and Applications. The author was not involved in the review and decision on publication of this article.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Kitano, H. Nobel Turing Challenge: creating the engine for scientific discovery. npj Syst Biol Appl 7 , 29 (2021). https://doi.org/10.1038/s41540-021-00189-3

Download citation

Received : 28 January 2021

Accepted : 03 June 2021

Published : 18 June 2021

DOI : https://doi.org/10.1038/s41540-021-00189-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

A dynamic knowledge graph approach to distributed self-driving laboratories.

• Sebastian Mosbach
• Markus Kraft

Nature Communications (2024)

Intrinsic and post-hoc XAI approaches for fingerprint identification and response prediction in smart manufacturing processes

• Abhilash Puthanveettil Madathil

Journal of Intelligent Manufacturing (2024)

Leveraging pre-trained language models for mining microbiome-disease relationships

• Nikitha Karkera
• Sathwik Acharya
• Sucheendra K. Palaniappan

BMC Bioinformatics (2023)

Hypotheses devised by AI could find ‘blind spots’ in research

• Matthew Hutson

Nature (2023)

Asking the right questions for mutagenicity prediction from BioMedical text

• Nicolas K. Shinada

npj Systems Biology and Applications (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Change Password

Your password must have 8 characters or more and contain 3 of the following:.

• a lower case character, 
• an upper case character, 
• a special character 

Password Changed Successfully

Your password has been changed

• Sign in / Register

Request Username

Can't sign in? Forgot your username?

Enter your email address below and we will send you your username

If the address matches an existing account you will receive an email with instructions to retrieve your username

Teaching Creativity and Inventive Problem Solving in Science

• Robert L. DeHaan

Division of Educational Studies, Emory University, Atlanta, GA 30322

Search for more papers by this author

Engaging learners in the excitement of science, helping them discover the value of evidence-based reasoning and higher-order cognitive skills, and teaching them to become creative problem solvers have long been goals of science education reformers. But the means to achieve these goals, especially methods to promote creative thinking in scientific problem solving, have not become widely known or used. In this essay, I review the evidence that creativity is not a single hard-to-measure property. The creative process can be explained by reference to increasingly well-understood cognitive skills such as cognitive flexibility and inhibitory control that are widely distributed in the population. I explore the relationship between creativity and the higher-order cognitive skills, review assessment methods, and describe several instructional strategies for enhancing creative problem solving in the college classroom. Evidence suggests that instruction to support the development of creativity requires inquiry-based teaching that includes explicit strategies to promote cognitive flexibility. Students need to be repeatedly reminded and shown how to be creative, to integrate material across subject areas, to question their own assumptions, and to imagine other viewpoints and possibilities. Further research is required to determine whether college students' learning will be enhanced by these measures.

INTRODUCTION

Dr. Dunne paces in front of his section of first-year college students, today not as their Bio 110 teacher but in the role of facilitator in their monthly “invention session.” For this meeting, the topic is stem cell therapy in heart disease. Members of each team of four students have primed themselves on the topic by reading selected articles from accessible sources such as Science, Nature, and Scientific American, and searching the World Wide Web, triangulating for up-to-date, accurate, background information. Each team knows that their first goal is to define a set of problems or limitations to overcome within the topic and to begin to think of possible solutions. Dr. Dunne starts the conversation by reminding the group of the few ground rules: one speaker at a time, listen carefully and have respect for others' ideas, question your own and others' assumptions, focus on alternative paths or solutions, maintain an atmosphere of collaboration and mutual support. He then sparks the discussion by asking one of the teams to describe a problem in need of solution.

Science in the United States is widely credited as a major source of discovery and economic development. According to the 2005 TAP Report produced by a prominent group of corporate leaders, “To maintain our country's competitiveness in the twenty-first century, we must cultivate the skilled scientists and engineers needed to create tomorrow's innovations.” ( www.tap2015.org/about/TAP_report2.pdf ). A panel of scientists, engineers, educators, and policy makers convened by the National Research Council (NRC) concurred with this view, reporting that the vitality of the nation “is derived in large part from the productivity of well-trained people and the steady stream of scientific and technical innovations they produce” ( NRC, 2007 ).

For many decades, science education reformers have promoted the idea that learners should be engaged in the excitement of science; they should be helped to discover the value of evidence-based reasoning and higher-order cognitive skills, and be taught to become innovative problem solvers (for reviews, see DeHaan, 2005 ; Hake, 2005 ; Nelson, 2008 ; Perkins and Wieman, 2008 ). But the means to achieve these goals, especially methods to promote creative thinking in scientific problem solving, are not widely known or used. An invention session such as that led by the fictional Dr. Dunne, described above, may seem fanciful as a means of teaching students to think about science as something more than a body of facts and terms to memorize. In recent years, however, models for promoting creative problem solving were developed for classroom use, as detailed by Treffinger and Isaksen (2005) , and such techniques are often used in the real world of high technology. To promote imaginative thinking, the advertising executive Alex F. Osborn invented brainstorming ( Osborn, 1948 , 1979 ), a technique that has since been successful in stimulating inventiveness among engineers and scientists. Could such strategies be transferred to a class for college students? Could they serve as a supplement to a high-quality, scientific teaching curriculum that helps students learn the facts and conceptual frameworks of science and make progress along the novice–expert continuum? Could brainstorming or other instructional strategies that are specifically designed to promote creativity teach students to be more adaptive in their growing expertise, more innovative in their problem-solving abilities? To begin to answer those questions, we first need to understand what is meant by “creativity.”

What Is Creativity? Big-C versus Mini-C Creativity

How to define creativity is an age-old question. Justice Potter Stewart's famous dictum regarding obscenity “I know it when I see it” has also long been an accepted test of creativity. But this is not an adequate criterion for developing an instructional approach. A scientist colleague of mine recently noted that “Many of us [in the scientific community] rarely give the creative process a second thought, imagining one either ‘has it’ or doesn't.” We often think of inventiveness or creativity in scientific fields as the kind of gift associated with a Michelangelo or Einstein. This is what Kaufman and Beghetto (2008) call big-C creativity, borrowing the term that earlier workers applied to the talents of experts in various fields who were identified as particularly creative by their expert colleagues ( MacKinnon, 1978 ). In this sense, creativity is seen as the ability of individuals to generate new ideas that contribute substantially to an intellectual domain. Howard Gardner defined such a creative person as one who “regularly solves problems, fashions products, or defines new questions in a domain in a way that is initially considered novel but that ultimately comes to be accepted in a particular cultural setting” ( Gardner, 1993 , p. 35).

But there is another level of inventiveness termed by various authors as “little-c” ( Craft, 2000 ) or “mini-c” ( Kaufman and Beghetto, 2008 ) creativity that is widespread among all populations. This would be consistent with the workplace definition of creativity offered by Amabile and her coworkers: “coming up with fresh ideas for changing products, services and processes so as to better achieve the organization's goals” ( Amabile et al. , 2005 ). Mini-c creativity is based on what Craft calls “possibility thinking” ( Craft, 2000 , pp. 3–4), as experienced when a worker suddenly has the insight to visualize a new, improved way to accomplish a task; it is represented by the “aha” moment when a student first sees two previously disparate concepts or facts in a new relationship, an example of what Arthur Koestler identified as bisociation: “perceiving a situation or event in two habitually incompatible associative contexts” ( Koestler, 1964 , p. 95).

In this essay, I maintain that mini-c creativity is not a mysterious, innate endowment of rare individuals. Instead, I argue that creative thinking is a multicomponent process, mediated through social interactions, that can be explained by reference to increasingly well-understood mental abilities such as cognitive flexibility and cognitive control that are widely distributed in the population. Moreover, I explore some of the recent research evidence (though with no effort at a comprehensive literature review) showing that these mental abilities are teachable; like other higher-order cognitive skills (HOCS), they can be enhanced by explicit instruction.

Creativity Is a Multicomponent Process

Efforts to define creativity in psychological terms go back to J. P. Guilford ( Guilford, 1950 ) and E. P. Torrance ( Torrance, 1974 ), both of whom recognized that underlying the construct were other cognitive variables such as ideational fluency, originality of ideas, and sensitivity to missing elements. Many authors since then have extended the argument that a creative act is not a singular event but a process, an interplay among several interactive cognitive and affective elements. In this view, the creative act has two phases, a generative and an exploratory or evaluative phase ( Finke et al. , 1996 ). During the generative process, the creative mind pictures a set of novel mental models as potential solutions to a problem. In the exploratory phase, we evaluate the multiple options and select the best one. Early scholars of creativity, such as J. P. Guilford, characterized the two phases as divergent thinking and convergent thinking ( Guilford, 1950 ). Guilford defined divergent thinking as the ability to produce a broad range of associations to a given stimulus or to arrive at many solutions to a problem (for overviews of the field from different perspectives, see Amabile, 1996 ; Banaji et al. , 2006 ; Sawyer, 2006 ). In neurocognitive terms, divergent thinking is referred to as associative richness ( Gabora, 2002 ; Simonton, 2004 ), which is often measured experimentally by comparing the number of words that an individual generates from memory in response to stimulus words on a word association test. In contrast, convergent thinking refers to the capacity to quickly focus on the one best solution to a problem.

The idea that there are two stages to the creative process is consistent with results from cognition research indicating that there are two distinct modes of thought, associative and analytical ( Neisser, 1963 ; Sloman, 1996 ). In the associative mode, thinking is defocused, suggestive, and intuitive, revealing remote or subtle connections between items that may be correlated, or may not, and are usually not causally related ( Burton, 2008 ). In the analytical mode, thought is focused and evaluative, more conducive to analyzing relationships of cause and effect (for a review of other cognitive aspects of creativity, see Runco, 2004 ). Science educators associate the analytical mode with the upper levels (analysis, synthesis, and evaluation) of Bloom's taxonomy (e.g., Crowe et al. , 2008 ), or with “critical thinking,” the process that underlies the “purposeful, self-regulatory judgment that drives problem-solving and decision-making” ( Quitadamo et al. , 2008 , p. 328). These modes of thinking are under cognitive control through the executive functions of the brain. The core executive functions, which are thought to underlie all planning, problem solving, and reasoning, are defined ( Blair and Razza, 2007 ) as working memory control (mentally holding and retrieving information), cognitive flexibility (considering multiple ideas and seeing different perspectives), and inhibitory control (resisting several thoughts or actions to focus on one). Readers wishing to delve further into the neuroscience of the creative process can refer to the cerebrocerebellar theory of creativity ( Vandervert et al. , 2007 ) in which these mental activities are described neurophysiologically as arising through interactions among different parts of the brain.

The main point from all of these works is that creativity is not some single hard-to-measure property or act. There is ample evidence that the creative process requires both divergent and convergent thinking and that it can be explained by reference to increasingly well-understood underlying mental abilities ( Haring-Smith, 2006 ; Kim, 2006 ; Sawyer, 2006 ; Kaufman and Sternberg, 2007 ) and cognitive processes ( Simonton, 2004 ; Diamond et al. , 2007 ; Vandervert et al. , 2007 ).

Creativity Is Widely Distributed and Occurs in a Social Context

Although it is understandable to speak of an aha moment as a creative act by the person who experiences it, authorities in the field have long recognized (e.g., Simonton, 1975 ) that creative thinking is not so much an individual trait but rather a social phenomenon involving interactions among people within their specific group or cultural settings. “Creativity isn't just a property of individuals, it is also a property of social groups” ( Sawyer, 2006 , p. 305). Indeed, Osborn introduced his brainstorming method because he was convinced that group creativity is always superior to individual creativity. He drew evidence for this conclusion from activities that demand collaborative output, for example, the improvisations of a jazz ensemble. Although each musician is individually creative during a performance, the novelty and inventiveness of each performer's playing is clearly influenced, and often enhanced, by “social and interactional processes” among the musicians ( Sawyer, 2006 , p. 120). Recently, Brophy (2006) offered evidence that for problem solving, the situation may be more nuanced. He confirmed that groups of interacting individuals were better at solving complex, multipart problems than single individuals. However, when dealing with certain kinds of single-issue problems, individual problem solvers produced a greater number of solutions than interacting groups, and those solutions were judged to be more original and useful.

Consistent with the findings of Brophy (2006) , many scholars acknowledge that creative discoveries in the real world such as solving the problems of cutting-edge science—which are usually complex and multipart—are influenced or even stimulated by social interaction among experts. The common image of the lone scientist in the laboratory experiencing a flash of creative inspiration is probably a myth from earlier days. As a case in point, the science historian Mara Beller analyzed the social processes that underlay some of the major discoveries of early twentieth-century quantum physics. Close examination of successive drafts of publications by members of the Copenhagen group revealed a remarkable degree of influence and collaboration among 10 or more colleagues, although many of these papers were published under the name of a single author ( Beller, 1999 ). Sociologists Bruno Latour and Steve Woolgar's study ( Latour and Woolgar, 1986 ) of a neuroendocrinology laboratory at the Salk Institute for Biological Studies make the related point that social interactions among the participating scientists determined to a remarkable degree what discoveries were made and how they were interpreted. In the laboratory, researchers studied the chemical structure of substances released by the brain. By analysis of the Salk scientists' verbalizations of concepts, theories, formulas, and results of their investigations, Latour and Woolgar showed that the structures and interpretations that were agreed upon, that is, the discoveries announced by the laboratory, were mediated by social interactions and power relationships among members of the laboratory group. By studying the discovery process in other fields of the natural sciences, sociologists and anthropologists have provided more cases that further illustrate how social and cultural dimensions affect scientific insights (for a thoughtful review, see Knorr Cetina, 1995 ).

In sum, when an individual experiences an aha moment that feels like a singular creative act, it may rather have resulted from a multicomponent process, under the influence of group interactions and social context. The process that led up to what may be sensed as a sudden insight will probably have included at least three diverse, but testable elements: 1) divergent thinking, including ideational fluency or cognitive flexibility, which is the cognitive executive function that underlies the ability to visualize and accept many ideas related to a problem; 2) convergent thinking or the application of inhibitory control to focus and mentally evaluate ideas; and 3) analogical thinking, the ability to understand a novel idea in terms of one that is already familiar.

LITERATURE REVIEW

What do we know about how to teach creativity.

The possibility of teaching for creative problem solving gained credence in the 1960s with the studies of Jerome Bruner, who argued that children should be encouraged to “treat a task as a problem for which one invents an answer, rather than finding one out there in a book or on the blackboard” ( Bruner, 1965 , pp. 1013–1014). Since that time, educators and psychologists have devised programs of instruction designed to promote creativity and inventiveness in virtually every student population: pre–K, elementary, high school, and college, as well as in disadvantaged students, athletes, and students in a variety of specific disciplines (for review, see Scott et al. , 2004 ). Smith (1998) identified 172 instructional approaches that have been applied at one time or another to develop divergent thinking skills.

Some of the most convincing evidence that elements of creativity can be enhanced by instruction comes from work with young children. Bodrova and Leong (2001) developed the Tools of the Mind (Tools) curriculum to improve all of the three core mental executive functions involved in creative problem solving: cognitive flexibility, working memory, and inhibitory control. In a year-long randomized study of 5-yr-olds from low-income families in 21 preschool classrooms, half of the teachers applied the districts' balanced literacy curriculum (literacy), whereas the experimenters trained the other half to teach the same academic content by using the Tools curriculum ( Diamond et al. , 2007 ). At the end of the year, when the children were tested with a battery of neurocognitive tests including a test for cognitive flexibility ( Durston et al. , 2003 ; Davidson et al. , 2006 ), those exposed to the Tools curriculum outperformed the literacy children by as much as 25% ( Diamond et al. , 2007 ). Although the Tools curriculum and literacy program were similar in academic content and in many other ways, they differed primarily in that Tools teachers spent 80% of their time explicitly reminding the children to think of alternative ways to solve a problem and building their executive function skills.

Teaching older students to be innovative also demands instruction that explicitly promotes creativity but is rigorously content-rich as well. A large body of research on the differences between novice and expert cognition indicates that creative thinking requires at least a minimal level of expertise and fluency within a knowledge domain ( Bransford et al. , 2000 ; Crawford and Brophy, 2006 ). What distinguishes experts from novices, in addition to their deeper knowledge of the subject, is their recognition of patterns in information, their ability to see relationships among disparate facts and concepts, and their capacity for organizing content into conceptual frameworks or schemata ( Bransford et al. , 2000 ; Sawyer, 2005 ).

Such expertise is often lacking in the traditional classroom. For students attempting to grapple with new subject matter, many kinds of problems that are presented in high school or college courses or that arise in the real world can be solved merely by applying newly learned algorithms or procedural knowledge. With practice, problem solving of this kind can become routine and is often considered to represent mastery of a subject, producing what Sternberg refers to as “pseudoexperts” ( Sternberg, 2003 ). But beyond such routine use of content knowledge the instructor's goal must be to produce students who have gained the HOCS needed to apply, analyze, synthesize, and evaluate knowledge ( Crowe et al. , 2008 ). The aim is to produce students who know enough about a field to grasp meaningful patterns of information, who can readily retrieve relevant knowledge from memory, and who can apply such knowledge effectively to novel problems. This condition is referred to as adaptive expertise ( Hatano and Ouro, 2003 ; Schwartz et al. , 2005 ). Instead of applying already mastered procedures, adaptive experts are able to draw on their knowledge to invent or adapt strategies for solving unique or novel problems within a knowledge domain. They are also able, ideally, to transfer conceptual frameworks and schemata from one domain to another (e.g., Schwartz et al. , 2005 ). Such flexible, innovative application of knowledge is what results in inventive or creative solutions to problems ( Crawford and Brophy, 2006 ; Crawford, 2007 ).

Promoting Creative Problem Solving in the College Classroom

In most college courses, instructors teach science primarily through lectures and textbooks that are dominated by facts and algorithmic processing rather than by concepts, principles, and evidence-based ways of thinking. This is despite ample evidence that many students gain little new knowledge from traditional lectures ( Hrepic et al. , 2007 ). Moreover, it is well documented that these methods engender passive learning rather than active engagement, boredom instead of intellectual excitement, and linear thinking rather than cognitive flexibility (e.g., Halpern and Hakel, 2003 ; Nelson, 2008 ; Perkins and Wieman, 2008 ). Cognitive flexibility, as noted, is one of the three core mental executive functions involved in creative problem solving ( Ausubel, 1963 , 2000 ). The capacity to apply ideas creatively in new contexts, referred to as the ability to “transfer” knowledge (see Mestre, 2005 ), requires that learners have opportunities to actively develop their own representations of information to convert it to a usable form. Especially when a knowledge domain is complex and fraught with ill-structured information, as in a typical introductory college biology course, instruction that emphasizes active-learning strategies is demonstrably more effective than traditional linear teaching in reducing failure rates and in promoting learning and transfer (e.g., Freeman et al. , 2007 ). Furthermore, there is already some evidence that inclusion of creativity training as part of a college curriculum can have positive effects. Hunsaker (2005) has reviewed a number of such studies. He cites work by McGregor (2001) , for example, showing that various creativity training programs including brainstorming and creative problem solving increase student scores on tests of creative-thinking abilities.

Model creativity—students develop creativity when instructors model creative thinking and inventiveness.

Repeatedly encourage idea generation—students need to be reminded to generate their own ideas and solutions in an environment free of criticism.

Cross-fertilize ideas—where possible, avoid teaching in subject-area boxes: a math box, a social studies box, etc; students' creative ideas and insights often result from learning to integrate material across subject areas.

Build self-efficacy—all students have the capacity to create and to experience the joy of having new ideas, but they must be helped to believe in their own capacity to be creative.

Constantly question assumptions—make questioning a part of the daily classroom exchange; it is more important for students to learn what questions to ask and how to ask them than to learn the answers.

Imagine other viewpoints—students broaden their perspectives by learning to reflect upon ideas and concepts from different points of view.

How Is Creativity Related to Critical Thinking and the Higher-Order Cognitive Skills?

It is not uncommon to associate creativity and ingenuity with scientific reasoning ( Sawyer, 2005 ; 2006 ). When instructors apply scientific teaching strategies ( Handelsman et al. , 2004 ; DeHaan, 2005 ; Wood, 2009 ) by using instructional methods based on learning research, according to Ebert-May and Hodder ( 2008 ), “we see students actively engaged in the thinking, creativity, rigor, and experimentation we associate with the practice of science—in much the same way we see students learn in the field and in laboratories” (p. 2). Perkins and Wieman (2008) note that “To be successful innovators in science and engineering, students must develop a deep conceptual understanding of the underlying science ideas, an ability to apply these ideas and concepts broadly in different contexts, and a vision to see their relevance and usefulness in real-world applications … An innovator is able to perceive and realize potential connections and opportunities better than others” (pp. 181–182). The results of Scott et al. (2004) suggest that nontraditional courses in science that are based on constructivist principles and that use strategies of scientific teaching to promote the HOCS and enhance content mastery and dexterity in scientific thinking ( Handelsman et al. , 2007 ; Nelson, 2008 ) also should be effective in promoting creativity and cognitive flexibility if students are explicitly guided to learn these skills.

Creativity is an essential element of problem solving ( Mumford et al. , 1991 ; Runco, 2004 ) and of critical thinking ( Abrami et al. , 2008 ). As such, it is common to think of applications of creativity such as inventiveness and ingenuity among the HOCS as defined in Bloom's taxonomy ( Crowe et al. , 2008 ). Thus, it should come as no surprise that creativity, like other elements of the HOCS, can be taught most effectively through inquiry-based instruction, informed by constructivist theory ( Ausubel, 1963 , 2000 ; Duch et al. , 2001 ; Nelson, 2008 ). In a survey of 103 instructors who taught college courses that included creativity instruction, Bull et al. (1995) asked respondents to rate the importance of various course characteristics for enhancing student creativity. Items ranking high on the list were: providing a social climate in which students feels safe, an open classroom environment that promotes tolerance for ambiguity and independence, the use of humor, metaphorical thinking, and problem defining. Many of the responses emphasized the same strategies as those advanced to promote creative problem solving (e.g., Mumford et al. , 1991 ; McFadzean, 2002 ; Treffinger and Isaksen, 2005 ) and critical thinking ( Abrami et al. , 2008 ).

In a careful meta-analysis, Scott et al. (2004) examined 70 instructional interventions designed to enhance and measure creative performance. The results were striking. Courses that stressed techniques such as critical thinking, convergent thinking, and constraint identification produced the largest positive effect sizes. More open techniques that provided less guidance in strategic approaches had less impact on the instructional outcomes. A striking finding was the effectiveness of being explicit; approaches that clearly informed students about the nature of creativity and offered clear strategies for creative thinking were most effective. Approaches such as social modeling, cooperative learning, and case-based (project-based) techniques that required the application of newly acquired knowledge were found to be positively correlated to high effect sizes. The most clear-cut result to emerge from the Scott et al. (2004) study was simply to confirm that creativity instruction can be highly successful in enhancing divergent thinking, problem solving, and imaginative performance. Most importantly, of the various cognitive processes examined, those linked to the generation of new ideas such as problem finding, conceptual combination, and idea generation showed the greatest improvement. The success of creativity instruction, the authors concluded, can be attributed to “developing and providing guidance concerning the application of requisite cognitive capacities … [and] a set of heuristics or strategies for working with already available knowledge” (p. 382).

Many of the scientific teaching practices that have been shown by research to foster content mastery and HOCS, and that are coming more widely into use, also would be consistent with promoting creativity. Wood (2009) has recently reviewed examples of such practices and how to apply them. These include relatively small modifications of the traditional lecture to engender more active learning, such as the use of concept tests and peer instruction ( Mazur, 1996 ), Just-in-Time-Teaching techniques ( Novak et al. , 1999 ), and student response systems known as “clickers” ( Knight and Wood, 2005 ; Crossgrove and Curran, 2008 ), all designed to allow the instructor to frequently and effortlessly elicit and respond to student thinking. Other strategies can transform the lecture hall into a workshop or studio classroom ( Gaffney et al. , 2008 ) where the teaching curriculum may emphasize problem-based (also known as project-based or case-based) learning strategies ( Duch et al. , 2001 ; Ebert-May and Hodder, 2008 ) or “community-based inquiry” in which students engage in research that enhances their critical-thinking skills ( Quitadamo et al. , 2008 ).

Another important approach that could readily subserve explicit creativity instruction is the use of computer-based interactive simulations, or “sims” ( Perkins and Wieman, 2008 ) to facilitate inquiry learning and effective, easy self-assessment. An example in the biological sciences would be Neurons in Action ( http://neuronsinaction.com/home/main ). In such educational environments, students gain conceptual understanding of scientific ideas through interactive engagement with materials (real or virtual), with each other, and with instructors. Following the tenets of scientific teaching, students are encouraged to pose and answer their own questions, to make sense of the materials, and to construct their own understanding. The question I pose here is whether an additional focus—guiding students to meet these challenges in a context that explicitly promotes creativity—would enhance learning and advance students' progress toward adaptive expertise?

Assessment of Creativity

To teach creativity, there must be measurable indicators to judge how much students have gained from instruction. Educational programs intended to teach creativity became popular after the Torrance Tests of Creative Thinking (TTCT) was introduced in the 1960s ( Torrance, 1974 ). But it soon became apparent that there were major problems in devising tests for creativity, both because of the difficulty of defining the construct and because of the number and complexity of elements that underlie it. Tests of intelligence and other personality characteristics on creative individuals revealed a host of related traits such as verbal fluency, metaphorical thinking, flexible decision making, tolerance of ambiguity, willingness to take risks, autonomy, divergent thinking, self-confidence, problem finding, ideational fluency, and belief in oneself as being “creative” ( Barron and Harrington, 1981 ; Tardif and Sternberg, 1988 ; Runco and Nemiro, 1994 ; Snyder et al. , 2004 ). Many of these traits have been the focus of extensive research of recent decades, but, as noted above, creativity is not defined by any one trait; there is now reason to believe that it is the interplay among the cognitive and affective processes that underlie inventiveness and the ability to find novel solutions to a problem.

Although the early creativity researchers recognized that assessing divergent thinking as a measure of creativity required tests for other underlying capacities ( Guilford, 1950 ; Torrance, 1974 ), these workers and their colleagues nonetheless believed that a high score for divergent thinking alone would correlate with real creative output. Unfortunately, no such correlation was shown ( Barron and Harrington, 1981 ). Results produced by many of the instruments initially designed to measure various aspects of creative thinking proved to be highly dependent on the test itself. A review of several hundred early studies showed that an individual's creativity score could be affected by simple test variables, for example, how the verbal pretest instructions were worded ( Barron and Harrington, 1981 , pp. 442–443). Most scholars now agree that divergent thinking, as originally defined, was not an adequate measure of creativity. The process of creative thinking requires a complex combination of elements that include cognitive flexibility, memory control, inhibitory control, and analogical thinking, enabling the mind to free-range and analogize, as well as to focus and test.

More recently, numerous psychometric measures have been developed and empirically tested (see Plucker and Renzulli, 1999 ) that allow more reliable and valid assessment of specific aspects of creativity. For example, the creativity quotient devised by Snyder et al. (2004) tests the ability of individuals to link different ideas and different categories of ideas into a novel synthesis. The Wallach–Kogan creativity test ( Wallach and Kogan, 1965 ) explores the uniqueness of ideas associated with a stimulus. For a more complete list and discussion, see the Creativity Tests website ( www.indiana.edu/∼bobweb/Handout/cretv_6.html ).

The most widely used measure of creativity is the TTCT, which has been modified four times since its original version in 1966 to take into account subsequent research. The TTCT-Verbal and the TTCT-Figural are two versions ( Torrance, 1998 ; see http://ststesting.com/2005giftttct.html ). The TTCT-Verbal consists of five tasks; the “stimulus” for each task is a picture to which the test-taker responds briefly in writing. A sample task that can be viewed from the TTCT Demonstrator website asks, “Suppose that people could transport themselves from place to place with just a wink of the eye or a twitch of the nose. What might be some things that would happen as a result? You have 3 min.” ( www.indiana.edu/∼bobweb/Handout/d3.ttct.htm ).

In the TTCT-Figural, participants are asked to construct a picture from a stimulus in the form of a partial line drawing given on the test sheet (see example below; Figure 1 ). Specific instructions are to “Add lines to the incomplete figures below to make pictures out of them. Try to tell complete stories with your pictures. Give your pictures titles. You have 3 min.” In the introductory materials, test-takers are urged to “… think of a picture or object that no one else will think of. Try to make it tell as complete and as interesting a story as you can …” ( Torrance et al. , 2008 , p. 2).

Figure 1. Sample figural test item from the TTCT Demonstrator website ( www.indiana.edu/∼bobweb/Handout/d3.ttct.htm ).

How would an instructor in a biology course judge the creativity of students' responses to such an item? To assist in this task, the TTCT has scoring and norming guides ( Torrance, 1998 ; Torrance et al. , 2008 ) with numerous samples and responses representing different levels of creativity. The guides show sample evaluations based upon specific indicators such as fluency, originality, elaboration (or complexity), unusual visualization, extending or breaking boundaries, humor, and imagery. These examples are easy to use and provide a high degree of validity and generalizability to the tests. The TTCT has been more intensively researched and analyzed than any other creativity instrument, and the norming samples have longitudinal validations and high predictive validity over a wide age range. In addition to global creativity scores, the TTCT is designed to provide outcome measures in various domains and thematic areas to allow for more insightful analysis ( Kaufman and Baer, 2006 ). Kim (2006) has examined the characteristics of the TTCT, including norms, reliability, and validity, and concludes that the test is an accurate measure of creativity. When properly used, it has been shown to be fair in terms of gender, race, community status, and language background. According to Kim (2006) and other authorities in the field ( McIntyre et al. , 2003 ; Scott et al. , 2004 ), Torrance's research and the development of the TTCT have provided groundwork for the idea that creative levels can be measured and then increased through instruction and practice.

SCIENTIFIC TEACHING TO PROMOTE CREATIVITY

How could creativity instruction be integrated into scientific teaching.

Guidelines for designing specific course units that emphasize HOCS by using strategies of scientific teaching are now available from the current literature. As an example, Karen Cloud-Hansen and colleagues ( Cloud-Hansen et al. , 2008 ) describe a course titled, “Ciprofloxacin Resistance in Neisseria gonorrhoeae .” They developed this undergraduate seminar to introduce college freshmen to important concepts in biology within a real-world context and to increase their content knowledge and critical-thinking skills. The centerpiece of the unit is a case study in which teams of students are challenged to take the role of a director of a local public health clinic. One of the county commissioners overseeing the clinic is an epidemiologist who wants to know “how you plan to address the emergence of ciprofloxacin resistance in Neisseria gonorrhoeae ” (p. 304). State budget cuts limit availability of expensive antibiotics and some laboratory tests to patients. Student teams are challenged to 1) develop a plan to address the medical, economic, and political questions such a clinic director would face in dealing with ciprofloxacin-resistant N. gonorrhoeae ; 2) provide scientific data to support their conclusions; and 3) describe their clinic plan in a one- to two-page referenced written report.

Throughout the 3-wk unit, in accordance with the principles of problem-based instruction ( Duch et al. , 2001 ), course instructors encourage students to seek, interpret, and synthesize their own information to the extent possible. Students have access to a variety of instructional formats, and active-learning experiences are incorporated throughout the unit. These activities are interspersed among minilectures and give the students opportunities to apply new information to their existing base of knowledge. The active-learning activities emphasize the key concepts of the minilectures and directly confront common misconceptions about antibiotic resistance, gene expression, and evolution. Weekly classes include question/answer/discussion sessions to address student misconceptions and 20-min minilectures on such topics as antibiotic resistance, evolution, and the central dogma of molecular biology. Students gather information about antibiotic resistance in N. gonorrhoeae , epidemiology of gonorrhea, and treatment options for the disease, and each team is expected to formulate a plan to address ciprofloxacin resistance in N. gonorrhoeae .

In this project, the authors assessed student gains in terms of content knowledge regarding topics covered such as the role of evolution in antibiotic resistance, mechanisms of gene expression, and the role of oncogenes in human disease. They also measured HOCS as gains in problem solving, according to a rubric that assessed self-reported abilities to communicate ideas logically, solve difficult problems about microbiology, propose hypotheses, analyze data, and draw conclusions. Comparing the pre- and posttests, students reported significant learning of scientific content. Among the thinking skill categories, students demonstrated measurable gains in their ability to solve problems about microbiology but the unit seemed to have little impact on their more general perceived problem-solving skills ( Cloud-Hansen et al. , 2008 ).

What would such a class look like with the addition of explicit creativity-promoting approaches? Would the gains in problem-solving abilities have been greater if during the minilectures and other activities, students had been introduced explicitly to elements of creative thinking from the Sternberg and Williams (1998) list described above? Would the students have reported greater gains if their instructors had encouraged idea generation with weekly brainstorming sessions; if they had reminded students to cross-fertilize ideas by integrating material across subject areas; built self-efficacy by helping students believe in their own capacity to be creative; helped students question their own assumptions; and encouraged students to imagine other viewpoints and possibilities? Of most relevance, could the authors have been more explicit in assessing the originality of the student plans? In an experiment that required college students to develop plans of a different, but comparable, type, Osborn and Mumford (2006) created an originality rubric ( Figure 2 ) that could apply equally to assist instructors in judging student plans in any course. With such modifications, would student gains in problem-solving abilities or other HOCS have been greater? Would their plans have been measurably more imaginative?

Figure 2. Originality rubric (adapted from Osburn and Mumford, 2006 , p. 183).

Answers to these questions can only be obtained when a course like that described by Cloud-Hansen et al. (2008) is taught with explicit instruction in creativity of the type I described above. But, such answers could be based upon more than subjective impressions of the course instructors. For example, students could be pretested with items from the TTCT-Verbal or TTCT-Figural like those shown. If, during minilectures and at every contact with instructors, students were repeatedly reminded and shown how to be as creative as possible, to integrate material across subject areas, to question their own assumptions and imagine other viewpoints and possibilities, would their scores on TTCT posttest items improve? Would the plans they formulated to address ciprofloxacin resistance become more imaginative?

Recall that in their meta-analysis, Scott et al. (2004) found that explicitly informing students about the nature of creativity and offering strategies for creative thinking were the most effective components of instruction. From their careful examination of 70 experimental studies, they concluded that approaches such as social modeling, cooperative learning, and case-based (project-based) techniques that required the application of newly acquired knowledge were positively correlated with high effect sizes. The study was clear in confirming that explicit creativity instruction can be successful in enhancing divergent thinking and problem solving. Would the same strategies work for courses in ecology and environmental biology, as detailed by Ebert-May and Hodder (2008) , or for a unit elaborated by Knight and Wood (2005) that applies classroom response clickers?

Finally, I return to my opening question with the fictional Dr. Dunne. Could a weekly brainstorming “invention session” included in a course like those described here serve as the site where students are introduced to concepts and strategies of creative problem solving? As frequently applied in schools of engineering ( Paulus and Nijstad, 2003 ), brainstorming provides an opportunity for the instructor to pose a problem and to ask the students to suggest as many solutions as possible in a brief period, thus enhancing ideational fluency. Here, students can be encouraged explicitly to build on the ideas of others and to think flexibly. Would brainstorming enhance students' divergent thinking or creative abilities as measured by TTCT items or an originality rubric? Many studies have demonstrated that group interactions such as brainstorming, under the right conditions, can indeed enhance creativity ( Paulus and Nijstad, 2003 ; Scott et al. , 2004 ), but there is little information from an undergraduate science classroom setting. Intellectual Ventures, a firm founded by Nathan Myhrvold, the creator of Microsoft's Research Division, has gathered groups of engineers and scientists around a table for day-long sessions to brainstorm about a prearranged topic. Here, the method seems to work. Since it was founded in 2000, Intellectual Ventures has filed hundreds of patent applications in more than 30 technology areas, applying the “invention session” strategy ( Gladwell, 2008 ). Currently, the company ranks among the top 50 worldwide in number of patent applications filed annually. Whether such a technique could be applied successfully in a college science course will only be revealed by future research.

• Abrami P. C., Bernard R. M., Borokhovski E., Wadem A., Surkes M. A., Tamim R., Zhang D. ( 2008 ). Instructional interventions affecting critical thinking skills and dispositions: a stage 1 meta-analysis . Rev. Educ. Res 78 , 1102-1134. Google Scholar
• Amabile T. M. ( 1996 ). Creativity in Context , Boulder, CO: Westview Press. Google Scholar
• Amabile T. M., Barsade S. G., Mueller J. S., Staw B. M. ( 2005 ). Affect and creativity at work . Admin. Sci. Q 50 , 367-403. Google Scholar
• Ausubel D. ( 1963 ). The Psychology of Meaningful Verbal Learning , New York: Grune and Stratton. Google Scholar
• Ausubel B. ( 2000 ). The Acquisition and Retention of Knowledge: A Cognitive View , Boston, MA: Kluwer Academic Publishers. Google Scholar
• Banaji S., Burn A., Buckingham D. ( 2006 ). The Rhetorics of Creativity: A Review of the Literature , accessed 29 December 2008 London: Centre for the Study of Children, Youth and Media, www.creativepartnerships.com/data/files/rhetorics-of-creativity-12.pdf . Google Scholar
• Barron F., Harrington D. M. ( 1981 ). Creativity, intelligence and personality . Ann. Rev. Psychol 32 , 439-476. Google Scholar
• Beller M. ( 1999 ). Quantum Dialogue: The Making of a Revolution , Chicago, IL: University of Chicago Press. Google Scholar
• Blair C., Razza R. P. ( 2007 ). Relating effortful control, executive function, and false belief understanding to emerging math and literacy ability in kindergarten . Child Dev 78 , 647-663. Medline ,  Google Scholar
• Bodrova E., Leong D. J. ( 2001 ). The Tool of the Mind: a case study of implementing the Vygotskian approach In: American Early Childhood and Primary Classrooms , Geneva, Switzerland: UNESCO International Bureau of Education. Google Scholar
• Bransford J. D.Brown A. L.Cocking R. R. ( 2000 ). How People Learn: Brain, Mind, Experience, and School , Washington, DC: National Academies Press. Google Scholar
• Brophy D. R. ( 2006 ). A comparison of individual and group efforts to creatively solve contrasting types of problems . Creativity Res. J 18 , 293-315. Google Scholar
• Bruner J. ( 1965 ). The growth of mind . Am. Psychol 20 , 1007-1017. Medline ,  Google Scholar
• Bull K. S., Montgomery D., Baloche L. ( 1995 ). Teaching creativity at the college level: a synthesis of curricular components perceived as important by instructors . Creativity Res. J 8 , 83-90. Google Scholar
• Burton R. ( 2008 ). On Being Certain: Believing You Are Right Even When You're Not , New York: St. Martin's Press. Google Scholar
• Cloud-Hanson K. A., Kuehner J. N., Tong L., Miller S., Handelsman J. ( 2008 ). Money, sex and drugs: a case study to teach the genetics of antibiotic resistance . CBE Life Sci. Educ 7 , 302-309. Medline ,  Google Scholar
• Craft A. ( 2000 ). Teaching Creativity: Philosophy and Practice , New York: Routledge. Google Scholar
• Crawford V. M. ( 2007 ). Adaptive expertise as knowledge building in science teachers' problem solving accessed 1 July 2008 Proceedings of the Second European Cognitive Science Conference Delphi, Greece http://ctl.sri.com/publications/downloads/Crawford_EuroCogSci07Proceedings.pdf . Google Scholar
• Crawford V. M., Brophy S. ( 2006 ). Adaptive Expertise: Theory, Methods, Findings, and Emerging Issues; September 2006 In: accessed 1 July 2008 Menlo Park, CA: SRI International, http://ctl.sri.com/publications/downloads/AESymposiumReportOct06.pdf . Google Scholar
• Crossgrove K., Curran K. L. ( 2008 ). Using clickers in nonmajors- and majors-level biology courses: student opinion, learning, and long-term retention of course material . CBE Life Sci. Educ 7 , 146-154. Link ,  Google Scholar
• Crowe A., Dirks C., Wenderoth M. P. ( 2008 ). Biology in bloom: implementing Bloom's taxonomy to enhance student learning in biology . CBE Life Sci. Educ 7 , 368-381. Link ,  Google Scholar
• Davidson M. C., Amso D., Anderson L. C., Diamond A. ( 2006 ). Development of cognitive control and executive functions from 4–13 years: evidence from manipulations of memory, inhibition, and task switching . Neuropsychologia 44 , 2037-2078. Medline ,  Google Scholar
• DeHaan R. L. ( 2005 ). The impending revolution in undergraduate science education . J. Sci. Educ. Technol 14 , 253-270. Google Scholar
• Diamond A., Barnett W. S., Thomas J., Munro S. ( 2007 ). Preschool program improves cognitive control . Science 318 , 1387-1388. Medline ,  Google Scholar
• Duch B. J., Groh S. E., Allen D. E. ( 2001 ). The Power of Problem-based Learning , Sterling, VA: Stylus Publishers. Google Scholar
• Durston S., Davidson M. C., Thomas K. M., Worden M. S., Tottenham N., Martinez A., Watts R., Ulug A. M., Caseya B. J. ( 2003 ). Parametric manipulation of conflict and response competition using rapid mixed-trial event-related fMRI . Neuroimage 20 , 2135-2141. Medline ,  Google Scholar
• Ebert-May D., Hodder J. ( 2008 ). Pathways to Scientific Teaching , Sunderland, MA: Sinauer. Google Scholar
• Finke R. A., Ward T. B., Smith S. M. ( 1996 ). Creative Cognition: Theory, Research and Applications , Boston, MA: MIT Press. Google Scholar
• Freeman S., O'Connor E., Parks J. W., Cunningham M., Hurley D., Haak D., Dirks C., Wenderoth M. P. ( 2007 ). Prescribed active learning increases performance in introductory biology . CBE Life Sci. Educ 6 , 132-139. Link ,  Google Scholar
• Gabora L. ( 2002 ). Hewett T.Kavanagh E. Cognitive mechanisms underlying the creative process Proceedings of the Fourth International Conference on Creativity and Cognition 2002 October 13–16 Loughborough University, United Kingdom 126-133. Google Scholar
• Gaffney J.D.H., Richards E., Kustusch M. B., Ding L., Beichner R. ( 2008 ). Scaling up education reform . J. Coll. Sci. Teach 37 , 48-53. Google Scholar
• Gardner H. ( 1993 ). Creating Minds: An Anatomy of Creativity Seen through the Lives of Freud, Einstein, Picasso, Stravinsky, Eliot, Graham, and Ghandi In: New York: Harper Collins. Google Scholar
• Gladwell M. ( 2008 ). In the air; who says big ideas are rare? The New Yorker accessed 19 May 2008 www.newyorker.com/reporting/2008/05/12/080512fa_fact_gladwell . Google Scholar
• Guilford J. P. ( 1950 ). Creativity . Am. Psychol 5 , 444-454. Medline ,  Google Scholar
• Hake R. ( 2005 ). The physics education reform effort: a possible model for higher education . Natl. Teach. Learn. Forum 15 , 1-6. Google Scholar
• Halpern D. E., Hakel M. D. ( 2003 ). Applying the science of learning to the university and beyond . Change 35 , 36-42. Google Scholar
• Handelsman J. ( 2004 ). Scientific teaching . Science 304 , 521-522. Medline ,  Google Scholar
• Handelsman J, Miller S., Pfund C. ( 2007 ). Scientific Teaching , New York: W. H. Freeman and Co. Google Scholar
• Haring-Smith T. ( 2006 ). Creativity research review: some lessons for higher education. Association of American Colleges and Universities . Peer Rev 8 , 23-27. Google Scholar
• Hatano G., Ouro Y. ( 2003 ). Commentary: reconceptualizing school learning using insight from expertise research . Educ. Res 32 , 26-29. Google Scholar
• Hrepic Z., Zollman D. A., Rebello N. S. ( 2007 ). Comparing students' and experts' understanding of the content of a lecture . J. Sci. Educ. Technol 16 , 213-224. Google Scholar
• Hunsaker S. L. ( 2005 ). Outcomes of creativity training programs . Gifted Child Q 49 , 292-298. Google Scholar
• Kaufman J. C., Baer J. ( 2006 ). Intelligent testing with Torrance . Creativity Res. J 18 , 99-102. Google Scholar
• Kaufman J. C., Beghetto R. A. ( 2008 , Ed. R. L. DeHaanK.M.V. Narayan , Exploring mini-C: creativity across cultures In: Education for Innovation: Implications for India, China and America , Rotterdam, The Netherlands: Sense Publishers, 165-180. Google Scholar
• Kaufman J. C., Sternberg R. J. ( 2007 ). Creativity . Change 39 , 55-58. Google Scholar
• Kim K. H. ( 2006 ). Can we trust creativity tests: a review of the Torrance Tests of Creative Thinking (TTCT) . Creativity Res. J 18 , 3-14. Google Scholar
• Knight J. K., Wood W. B. ( 2005 ). Teaching more by lecturing less . Cell Biol. Educ 4 , 298-310. Link ,  Google Scholar
• Cetina Knorr K. ( 1995 , Ed. S. JasanoffG. MarkleJ. PetersenT. Pinch , Laboratory studies: the cultural approach to the study of science In: Handbook of Science and Technology Studies , Thousand Oaks, CA: Sage Publications, 140-166. Google Scholar
• Koestler A. ( 1964 ). The Act of Creation , New York: Macmillan. Google Scholar
• Latour B., Woolgar S. ( 1986 ). Laboratory Life: The Construction of Scientific Facts , Princeton, NJ: Princeton University Press. Google Scholar
• MacKinnon D. W. ( 1978 , Ed. D. W. MacKinnon , What makes a person creative? In: In Search of Human Effectiveness , New York: Universe Books, 178-186. Google Scholar
• Martindale C. ( 1999 , Ed. R. J. Sternberg , Biological basis of creativity In: Handbook of Creativity , Cambridge, United Kingdom: Cambridge University Press, 137-152. Google Scholar
• Mazur E. ( 1996 ). Peer Instruction: A User's Manual , Upper Saddle River, NJ: Prentice Hall. Google Scholar
• McFadzean E. ( 2002 ). Developing and supporting creative problem-solving teams: Part 1—a conceptual model . Manage. Decis 40 , 463-475. Google Scholar
• McGregor G. D. ( 2001 ). Creative thinking instruction for a college study skills program: a case study. . Dissert Abstr. Intl 62 , 3293A UMI No. AAT 3027933. Google Scholar
• McIntyre F. S., Hite R. E., Rickard M. K. ( 2003 ). Individual characteristics and creativity in the marketing classroom: exploratory insights . J. Mark. Educ 25 , 143-149. Google Scholar
• Mestre J. P. ( 2005 ). Transfer of Learning: From a Modern Multidisciplinary Perspective , Greenwich, CT: Information Age Publishing. Google Scholar
• Mumford M. D., Mobley M. I., Uhlman C. E., Reiter-Palmon R., Doares L. M. ( 1991 ). Process analytic models of creative capacities . Creativity Res. J 4 , 91-122. Google Scholar
• National Research Council ( 2007 ). Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future, Committee on Science, Engineering and Public Policy In: Washington, DC: National Academies Press. Google Scholar
• Neisser U. ( 1963 ). The multiplicity of thought . Br. J. Psychol 54 , 1-14. Medline ,  Google Scholar
• Nelson C. E. ( 2008 ). Teaching evolution (and all of biology) more effectively: strategies for engagement, critical reasoning, and confronting misconceptions Integrative and Comparative Biology Advance Access accessed 15 September 2008 http://icb.oxfordjournals.org/cgi/reprint/icn027v1.pdf . Google Scholar
• Novak G, Gavrin A., Christian W, Patterson E. ( 1999 ). Just-in-Time Teaching: Blending Active Learning with Web Technology , San Francisco, CA: Pearson Benjamin Cummings. Google Scholar
• Osborn A. F. ( 1948 ). Your Creative Power , New York: Scribner. Google Scholar
• Osborn A. F. ( 1979 ). Applied Imagination , New York: Scribner. Google Scholar
• Osburn H. K., Mumford M. D. ( 2006 ). Creativity and planning: training interventions to develop creative problem-solving skills . Creativity Res. J 18 , 173-190. Google Scholar
• Paulus P. B., Nijstad B. A. ( 2003 ). Group Creativity: Innovation through Collaboration , New York: Oxford University Press. Google Scholar
• Perkins K. K., Wieman C. E. ( 2008 , Ed. R. L. DeHaanK.M.V. Narayan , Innovative teaching to promote innovative thinking In: Education for Innovation: Implications for India, China and America , Rotterdam, The Netherlands: Sense Publishers, 181-210. Google Scholar
• Plucker J. A., Renzulli J. S. ( 1999 , Ed. R. J. Sternberg , Psychometric approaches to the study of human creativity In: Handbook of Creativity , Cambridge, United Kingdom: Cambridge University Press, 35-61. Google Scholar
• Quitadamo I. J., Faiola C. L., Johnson J. E., Kurtz M. J. ( 2008 ). Community-based inquiry improves critical thinking in general education biology . CBE Life Sci. Educ 7 , 327-337. Link ,  Google Scholar
• Runco M. A. ( 2004 ). Creativity . Annu. Rev. Psychol 55 , 657-687. Medline ,  Google Scholar
• Runco M. A., Nemiro J. ( 1994 ). Problem finding, creativity, and giftedness . Roeper Rev 16 , 235-241. Google Scholar
• Sawyer R. K. ( 2005 ). Educating for Innovation Thinking Skills Creativity accessed 13 August 2008 1 41-48 www.artsci.wustl.edu/∼ksawyer/PDFs/Thinkjournal.pdf . Google Scholar
• Sawyer R. K. ( 2006 ). Explaining Creativity: The Science of Human Innovation , New York: Oxford University Press. Google Scholar
• Schwartz D. L., Bransford J. D., Sears D. ( 2005 , Ed. J. P. Mestre , Efficiency and innovation in transfer In: Transfer of Learning from a Modern Multidisciplinary Perspective , Greenwich, CT: Information Age Publishing, 1-51. Google Scholar
• Scott G., Leritz L. E., Mumford M. D. ( 2004 ). The effectiveness of creativity training: a quantitative review . Creativity Res. J 16 , 361-388. Google Scholar
• Simonton D. K. ( 1975 ). Sociocultural context of individual creativity: a transhistorical time-series analysis . J. Pers. Soc. Psychol 32 , 1119-1133. Medline ,  Google Scholar
• Simonton D. K. ( 2004 ). Creativity in Science: Chance, Logic, Genius, and Zeitgeist , Oxford, United Kingdom: Cambridge University Press. Google Scholar
• Sloman S. ( 1996 ). The empirical case for two systems of reasoning . Psychol. Bull 9 , 3-22. Google Scholar
• Smith G. F. ( 1998 ). Idea generation techniques: a formulary of active ingredients . J. Creative Behav 32 , 107-134. Google Scholar
• Snyder A., Mitchell J., Bossomaier T., Pallier G. ( 2004 ). The creativity quotient: an objective scoring of ideational fluency . Creativity Res. J 16 , 415-420. Google Scholar
• Sternberg R. J. ( 2003 ). What is an “expert student?” . Educ. Res. 32 , 5-9. Google Scholar
• Sternberg R., Williams W. M. ( 1998 ). Teaching for creativity: two dozen tips accessed 25 March 2008 www.cdl.org/resource-library/articles/teaching_creativity.php . Google Scholar
• Tardif T. Z., Sternberg R. J. ( 1988 , Ed. R. J. Sternberg , What do we know about creativity? In: The Nature of Creativity , New York: Cambridge University Press, 429-440. Google Scholar
• Torrance E. P. ( 1974 ). Norms and Technical Manual for the Torrance Tests of Creative Thinking , Bensenville, IL: Scholastic Testing Service. Google Scholar
• Torrance E. P. ( 1998 ). The Torrance Tests of Creative Thinking Norms—Technical Manual Figural (Streamlined) Forms A and B , Bensenville, IL: Scholastic Testing Service. Google Scholar
• Torrance E. P., Ball O. E., Safter H. T. ( 2008 ). Torrance Tests of Creative Thinking: Streamlined Scoring Guide for Figural Forms A and B , Bensenville, IL: Scholastic Testing Service. Google Scholar
• Treffinger D. J., Isaksen S. G. ( 2005 ). Creative problem solving: the history, development, and implications for gifted education and talent development . Gifted Child Q 49 , 342-357. Google Scholar
• Vandervert L. R., Schimpf P. H., Liu H. ( 2007 ). How working memory and the cerebellum collaborate to produce creativity and innovation . Creativity Res. J 9 , 1-18. Google Scholar
• Wallach M. A., Kogan N. ( 1965 ). Modes of Thinking in Young Children: A Study of the Creativity-Intelligence Distinction , New York: Holt, Rinehart and Winston. Google Scholar
• Wood W. B. ( 2009 ). Innovations in undergraduate biology teaching and why we need them . Annu. Rev. Cell Dev. Biol in press. Medline ,  Google Scholar
• ChatGPT improves creative problem-solving performance in university students: An experimental study Computers & Education, Vol. 215
• Searching for creativity: How people search to generate new ideas 1 December 2023 | Journal of the Association for Information Science and Technology, Vol. 75, No. 4
• Fostering creativity in low-engagement students through socratic dialogue: An experiment in an operations class The International Journal of Management Education, Vol. 22, No. 1
• Cognitive flexibility and academic performance: Individual and cross-national patterns among adolescents in 57 countries Personality and Individual Differences, Vol. 217
• Are we teaching novice instructional designers to be creative? A qualitative case study 16 January 2024 | Instructional Science, Vol. 4
• What's in a word? Student beliefs and understanding about green chemistry 1 January 2024 | Chemistry Education Research and Practice, Vol. 25, No. 1
• Preservice Physical Education Teachers’ Resistance to Change: The Importance of Occupational Socialization Experiences 26 October 2023 | Trends in Higher Education, Vol. 2, No. 4
• How can we measure metacognition in creative problem-solving? Standardization of the MCPS scale Thinking Skills and Creativity, Vol. 49
• TOWARDS ENHANCING CREATIVITY AND INNOVATION IN EDUCATION SYSTEM FOR YOUTH IN HAIL REGION 1 August 2023 | Advanced Education, Vol. 10, No. 22
• Investigating the impact of innovation competence instruction in higher engineering education 12 June 2023 | European Journal of Engineering Education, Vol. 10
• Regaining creativity in science: insights from conversation 17 May 2023 | Royal Society Open Science, Vol. 10, No. 5
• How transdisciplinary integration, creativity and student motivation interact in three STEAM projects for gifted education? 29 March 2023 | Gifted Education International, Vol. 39, No. 2
• Does creative coursework predict educational, career, and community engagement outcomes for arts alumni? 15 November 2022 | Creativity Research Journal, Vol. 35, No. 2
• Make science disruptive again 27 March 2023 | Nature Biotechnology, Vol. 41, No. 4
• Promoting Creativity in Undergraduate Recreation and Leisure Services Classrooms: An Overview 19 March 2021 | SCHOLE: A Journal of Leisure Studies and Recreation Education, Vol. 38, No. 1
• A Positive Association between Working Memory Capacity and Human Creativity: A Meta-Analytic Evidence 13 January 2023 | Journal of Intelligence, Vol. 11, No. 1
• Students Creativity Through Digital Mind Map 26 July 2023
• Pedagogical and School Practices to Foster Key Competences and Domain-General Literacy 23 August 2023
• Coping with Challenges and Uncertainty in Scientific Research Asia-Pacific Science Education, Vol. 8, No. 2
• Teaching design thinking as a tool to address complex public health challenges in public health students: a case study 12 April 2022 | BMC Medical Education, Vol. 22, No. 1
• ‘Allowing them to dream’: fostering creativity in mathematics undergraduates 26 May 2022 | Journal of Further and Higher Education, Vol. 46, No. 10
• Interaction with metaphors enhances creative potential 1 July 2022 | Journal of Poetry Therapy, Vol. 35, No. 4
• Creative problem solving in knowledge-rich contexts Trends in Cognitive Sciences, Vol. 26, No. 10
• Teaching Protein–Ligand Interactions Using a Case Study on Tau in Alzheimer’s Disease 1 July 2022 | Journal of Chemical Education, Vol. 99, No. 8
• Student approaches to creative processes when participating in an open-ended project in science 30 June 2022 | International Journal of Science Education, Vol. 44, No. 10
• Arts, Machines, and Creative Education
• Perceived Learning Effectiveness and Student Satisfaction
• A Contribution to Scientific Creativity: A Validation Study Measuring Divergent Problem Solving Ability 3 September 2021 | Creativity Research Journal, Vol. 34, No. 2
• Problem Solving and Digital Transformation: Acquiring Skills through Pretend Play in Kindergarten 28 January 2022 | Education Sciences, Vol. 12, No. 2
• Growing Innovation and Collaboration Through Assessment and Feedback: A Toolkit for Assessing and Developing Students’ Soft Skills in Biological Experimentation 12 May 2022
• The Role of Creativity in Teaching Mathematics Online 1 December 2022
• Breathing Life Into Marketing Scholarship Through Creativity Learning and Teaching
• Mobilizing Research-Based Learning (RBL) in Higher Education
• Trends and opportunities by fostering creativity in science and engineering: a systematic review 2 September 2021 | European Journal of Engineering Education, Vol. 46, No. 6
• The effect of a scientific board game on improving creative problem solving skills Thinking Skills and Creativity, Vol. 41
• Create Teaching Creativity through Training Management, Effectiveness Training, and Teacher Quality in the Covid-19 Pandemic 6 August 2021 | Journal of Ethnic and Cultural Studies, Vol. 8, No. 4
• Creativity and technology in teaching and learning: a literature review of the uneasy space of implementation  11 January 2021 | Educational Technology Research and Development, Vol. 69, No. 4
• Üstün Yetenekli Öğrencilerin Bilimsel Yaratıcılık ve Bilimsel Problem Çözme ile İlgili Öz Değerlendirmeleri 15 July 2021 | Yuzunci Yil Universitesi Egitim Fakultesi Dergisi
• Cultivating creative thinking in engineering student teams: Can a computer‐mediated virtual laboratory help? 23 November 2020 | Journal of Computer Assisted Learning, Vol. 37, No. 2
• Entrepreneurial competencies of undergraduate students: The case of universities in Nigeria The International Journal of Management Education, Vol. 19, No. 1
• Promoting Creativity in General Education Mathematics Courses 5 August 2019 | PRIMUS, Vol. 31, No. 1
• Methodological Considerations for Understanding Students’ Problem Solving Processes and Affective Trajectories During Game-Based Learning: A Data Fusion Approach 3 July 2021
• The main trends in the process of building the creative potential of engineering students 18 June 2021 | E3S Web of Conferences, Vol. 274
• Research on the Present Situation and Countermeasures of Cultivating Graduate Students’ Innovation Ability under Cooperative Innovation Environment Creative Education Studies, Vol. 09, No. 02
• Innovation Centers and the Information Schools: The Influence of LIS Faculty Journal of Education for Library and Information Science, Vol. 61, No. 4
• Biomimetics: teaching the tools of the trade 28 September 2020 | FEBS Open Bio, Vol. 10, No. 11
• STEM academic teachers’ experiences of undertaking authentic assessment-led reform: a mixed method approach 21 March 2019 | Studies in Higher Education, Vol. 45, No. 9
• Problems of forming marketing competencies in the digital economy IOP Conference Series: Materials Science and Engineering, Vol. 940, No. 1
• Culturally Responsive Assessment of Physical Science Skills and Abilities: Development, Field Testing, Implementation, and Results 2 June 2020 | Journal of Advanced Academics, Vol. 31, No. 3
• Culturally Responsive Assessment of Life Science Skills and Abilities: Development, Field Testing, Implementation, and Results 3 June 2020 | Journal of Advanced Academics, Vol. 31, No. 3
• Curriculum Differentiation’s Capacity to Extend Gifted Students in Secondary Mixed-ability Science Classes 27 June 2020 | Talent, Vol. 10, No. 1
• Student Motivation from and Resistance to Active Learning Rooted in Essential Science Practices 23 December 2017 | Research in Science Education, Vol. 50, No. 1
• Integrating Entrepreneurship and Art to Improve Creative Problem Solving in Fisheries Education 27 February 2020 | Fisheries, Vol. 45, No. 2
• Concepts Re-imagined: Relational Signs Beyond Definitional Rigidity 15 August 2020
• Promoting Student Creativity and Inventiveness in Science and Engineering 24 October 2020
• Teaching for Leadership, Innovation, and Creativity
• Problem Çözme Becerileri Eğitim Programının Çocukların Karar Verme Becerileri Üzerindeki Etkisi 30 December 2019 | Erzincan Üniversitesi Eğitim Fakültesi Dergisi, Vol. 21, No. 3
• Mento’s change model in teaching competency-based medical education 27 December 2019 | BMC Medical Education, Vol. 19, No. 1
• The Effects of Individual Preparations on Group Creativity 21 January 2020
• The Value of Creativity for Enhancing Translational Ecologies, Insights, and Discoveries 9 July 2019 | Frontiers in Psychology, Vol. 10
• Using the International Classification of Functioning, Disability, and Health to Guide Students' Clinical Approach to Aging With Pathology Topics in Geriatric Rehabilitation, Vol. 35, No. 3
• Impact of Brainstorming Strategy in Dealing With Knowledge Retention Skill: An Insight Into Special Learners' Needs In Saudi Arabia 1 January 2021 | MIER Journal of Educational Studies Trends & Practices
• The potential of students’ creative disposition as a perspective to develop creative teaching and learning for senior high school biological science 12 March 2019 | Journal of Physics: Conference Series, Vol. 1157
• Evaluating Remote Experiment from a Divergent Thinking Point of View 25 July 2018
• Exploring Creative Education Practices and Implications: A Case study of National Chengchi University, Taiwan 4 April 2022 | Journal of Business and Economic Analysis, Vol. 02, No. 02
• Exploring Creative Education Practices and Implications: A Case study of National Chengchi University, Taiwan 1 January 2020 | Journal of Business and Economic Analysis, Vol. 02, No. 02
• Diverging from the Dogma: A Call to Train Creative Thinkers in Science 14 September 2018 | The Bulletin of the Ecological Society of America, Vol. 100, No. 1
• Comparison of German and Japanese student teachers’ views on creativity in chemistry class 16 May 2018 | Asia-Pacific Science Education, Vol. 4, No. 1
• The use of humour during a collaborative inquiry 27 August 2018 | International Journal of Science Education, Vol. 40, No. 14
• Connecting creative coursework exposure and college student engagement across academic disciplines 29 August 2019 | Gifted and Talented International, Vol. 33, No. 1-2
• Views of German chemistry teachers on creativity in chemistry classes and in general 1 January 2018 | Chemistry Education Research and Practice, Vol. 19, No. 3
• Embedding Critical and Creative Thinking in Chemical Engineering Practice
• Introducing storytelling to educational robotic activities
• Investigating Undergraduates’ Perceptions of Science in Courses Taught Using the CREATE Strategy Journal of Microbiology & Biology Education, Vol. 19, No. 1
• Teachers’ learning on the workshop of STS approach as a way of enhancing inventive thinking skills
• Creativity Development Through Inquiry-Based Learning in Biomedical Sciences
• The right tool for the right task: Structured techniques prove less effective on an ill-defined problem finding task Thinking Skills and Creativity, Vol. 26
• The influential factors and hierarchical structure of college students’ creative capabilities—An empirical study in Taiwan Thinking Skills and Creativity, Vol. 26
• Teacher perceptions of professional role and innovative teaching at elementary schools in Taiwan 10 November 2017 | Educational Research and Reviews, Vol. 12, No. 21
• Evaluation of creative problem-solving abilities in undergraduate structural engineers through interdisciplinary problem-based learning 28 July 2016 | European Journal of Engineering Education, Vol. 42, No. 6
• What Shall I Write Next? 19 September 2017
• Inquiry-based Laboratory Activities on Drugs Analysis for High School Chemistry Learning 3 October 2017 | Journal of Physics: Conference Series, Vol. 895
• BARRIERS TO STUDENTS’ CREATIVE EVALUATION OF UNEXPECTED EXPERIMENTAL FINDINGS 25 June 2017 | Journal of Baltic Science Education, Vol. 16, No. 3
• Kyle J. Frantz ,
• Melissa K. Demetrikopoulos ,
• Shari L. Britner ,
• Laura L. Carruth ,
• Brian A. Williams ,
• John L. Pecore ,
• Robert L. DeHaan , and
• Christopher T. Goode
• Elizabeth Ambos, Monitoring Editor
• A present absence: undergraduate course outlines and the development of student creativity across disciplines 3 October 2016 | Teaching in Higher Education, Vol. 22, No. 2
• Exploring differences in creativity across academic majors for high-ability college students 16 February 2018 | Gifted and Talented International, Vol. 32, No. 1
• Creativity in chemistry class and in general – German student teachers’ views 1 January 2017 | Chemistry Education Research and Practice, Vol. 18, No. 2
• IMPORTANCE OF CREATIVITY IN ENTREPRENEURSHIP 1 January 2017
• Accessing the Finest Minds
• Science and Innovative Thinking for Technical and Organizational Development
• Learning High School Biology in a Social Context Creative Education, Vol. 08, No. 15
• Possibilities and limitations of integrating peer instruction into technical creativity education 6 September 2016 | Instructional Science, Vol. 44, No. 6
• Creative Cognitive Processes in Higher Education 20 November 2014 | The Journal of Creative Behavior, Vol. 50, No. 4
• An Evidence-Based Review of Creative Problem Solving Tools 6 April 2016 | Human Resource Development Review, Vol. 15, No. 2
• Case-based exams for learning and assessment: Experiences in an information systems course
• Case exams for assessing higher order learning: A comparative social media analytics usage exam
• Beyond belief: Structured techniques prove more effective than a placebo intervention in a problem construction task Thinking Skills and Creativity, Vol. 19
• A Belief System at the Core of Learning Science
• Student Research Work and Modeled Situations in Order to Bridge the Gap between Basic Science Concepts and Those from Preventive and Clinical Practice. Meaningful Learning and Informed beneficience Creative Education, Vol. 07, No. 07
• FOSTERING FIFTH GRADERS’ SCIENTIFIC CREATIVITY THROUGH PROBLEM-BASED LEARNING 25 October 2015 | Journal of Baltic Science Education, Vol. 14, No. 5
• Scaffolding for Creative Product Possibilities in a Design-Based STEM Activity 16 November 2014 | Research in Science Education, Vol. 45, No. 5
• Intuition and insight: two concepts that illuminate the tacit in science education 18 June 2015 | Studies in Science Education, Vol. 51, No. 2
• Arts and crafts as adjuncts to STEM education to foster creativity in gifted and talented students 28 March 2015 | Asia Pacific Education Review, Vol. 16, No. 2
• Initiatives Towards an Education for Creativity Procedia - Social and Behavioral Sciences, Vol. 180
• Brian A. Couch ,
• Tanya L. Brown ,
• Tyler J. Schelpat ,
• Mark J. Graham , and
• Jennifer K. Knight
• Michèle Shuster, Monitoring Editor
• Kim Quillin , and
• Stephen Thomas
• Mary Lee Ledbetter, Monitoring Editor
• The Design of IdeaWorks: Applying Social Learning Networks to Support Tertiary Education 21 July 2015
• Video Games and Malevolent Creativity
• Modelling a Laboratory for Ideas as a New Tool for Fostering Engineering Creativity Procedia Engineering, Vol. 100
• “Development of Thinking Skills” Course: Teaching TRIZ in Academic Setting Procedia Engineering, Vol. 131
• Leadership in the Future Experts’ Creativity Development with Scientific Research Activities 4 November 2014
• Developing Deaf Children's Conceptual Understanding and Scientific Argumentation Skills: A Literature Review 3 January 2014 | Deafness & Education International, Vol. 16, No. 3
• Leslie M. Stevens , and
• Sally G. Hoskins
• Nancy Pelaez, Monitoring Editor
• Cortex, Vol. 51
• A Sociotechnological Theory of Discursive Change and Entrepreneurial Capacity: Novelty and Networks SSRN Electronic Journal, Vol. 3
• GEOverse: An Undergraduate Research Journal: Research Dissemination Within and Beyond the Curriculum 1 August 2013
• Learning by Practice, High-Pressure Student Ateliers 2 August 2013
• Relating Inter-Individual Differences in Verbal Creative Thinking to Cerebral Structures: An Optimal Voxel-Based Morphometry Study 5 November 2013 | PLoS ONE, Vol. 8, No. 11
• 21st Century Biology: An Interdisciplinary Approach of Biology, Technology, Engineering and Mathematics Education Procedia - Social and Behavioral Sciences, Vol. 102
• Reclaiming creativity in the era of impact: exploring ideas about creative research in science and engineering Studies in Higher Education, Vol. 38, No. 9
• An Evaluation of Alternative Ways of Computing the Creativity Quotient in a Design School Sample Creativity Research Journal, Vol. 25, No. 3
• A.-M. Hoskinson ,
• M. D. Caballero , and
• J. K. Knight
• Eric Brewe, Monitoring Editor
• Understanding, attitude and environment International Journal for Researcher Development, Vol. 4, No. 1
• Promoting Student Creativity and Inventiveness in Science and Engineering
• Building creative thinking in the classroom: From research to practice International Journal of Educational Research, Vol. 62
• A Demonstration of a Mastery Goal Driven Learning Environment to Foster Creativity in Engineering Design SSRN Electronic Journal, Vol. 111
• The development of creative cognition across adolescence: distinct trajectories for insight and divergent thinking 8 October 2012 | Developmental Science, Vol. 16, No. 1
• A CROSS-NATIONAL STUDY OF PROSPECTIVE ELEMENTARY AND SCIENCE TEACHERS’ CREATIVITY STYLES 10 September 2012 | Journal of Baltic Science Education, Vol. 11, No. 3
• Evaluation of fostering students' creativity in preparing aided recalls for revision courses using electronic revision and recapitulation tools 2.0 Behaviour & Information Technology, Vol. 31, No. 8
• Scientific Creativity: The Missing Ingredient in Slovenian Science Education 15 April 2012 | European Journal of Educational Research, Vol. volume-1-2012, No. volume1-issue2.html
• Could the ‘evolution’ from biology to life sciences prevent ‘extinction’ of the subject field? 6 March 2012 | Suid-Afrikaanse Tydskrif vir Natuurwetenskap en Tegnologie, Vol. 31, No. 1
• Mobile innovations, executive functions, and educational developments in conflict zones: a case study from Palestine 1 October 2011 | Educational Technology Research and Development, Vol. 60, No. 1
• Informing Pedagogy Through the Brain-Targeted Teaching Model Journal of Microbiology & Biology Education, Vol. 13, No. 1
• Teaching Creative Science Thinking Science, Vol. 334, No. 6062
• Sally G. Hoskins ,
• David Lopatto , and
• Leslie M. Stevens
• Diane K. O'Dowd, Monitoring Editor
• Embedding Research-Based Learning Early in the Undergraduate Geography Curriculum Journal of Geography in Higher Education, Vol. 35, No. 3
• Jared L. Taylor ,
• Karen M. Smith ,
• Adrian P. van Stolk , and
• George B. Spiegelman
• Debra Tomanek, Monitoring Editor
• IFAC Proceedings Volumes, Vol. 43, No. 17
• Critical and Creative Thinking Activities for Engaged Learning in Graphics and Visualization Course
• Creativity Development through Inquiry-Based Learning in Biomedical Sciences

Submitted: 31 December 2008 Revised: 14 May 2009 Accepted: 28 May 2009

© 2009 by The American Society for Cell Biology

Effective Multicultural Teams: Theory and Practice pp 239–274 Cite as

Problem Solving and Decision Making

• Linda Drake Gobbo 3  

7302 Accesses

Part of the book series: Advances in Group Decision and Negotiation ((AGDN,volume 3))

Problem solving and decision making in multicultural work teams are the last of the skill areas to be covered in this book. This topic will be discussed from the cultural, individual, and organizational levels of multicultural team development, building on the frameworks that have been presented in previous chapters. Many theorists consider problem solving and decision making as synonymous-all decisions are made in response to a problem or opportunity. Simply stated, if problem solving is the process used to find a solution to the problem, challenge, or opportunity. However, how one solves problems can be quite varied. An individual can use analytical tools based on logic, deduction, or induction, or intuition based on an understanding of principles, or creative thinking. Problem-solving abilities and approaches may vary considerably, actually using different paradigms or frameworks. In this chapter one approach, with the steps and methods to do problem solving in work teams, will be presented.

There are six steps to the problem-solving model described and demonstrated in this chapter. Several of those steps within the model are used for decisionmaking, and are covered as well. How a team makes the decision, and who on the team makes it are important elements and will also be discussed. As prior chapters have noted, membership of multicultural teams varies greatly. The procedures each member follows, the different value orientations guiding their behavior (Smith et al. 2002), the nature of the tasks they must complete, and the communication tools they employ (face-to-face and/or technology-based) all impact how they approach problem solving and decision making. When done effectively, problem solving, which includes decision making, moves through all the steps described here equally, engaging the knowledge and skills of all team members.

This chapter will first present theoretical frameworks for problem solving, then define the steps that comprise problem solving and decision making within them. This will be followed by a discussion of the cultural variations, and impact of individual styles and societal assumptions on decision-making. Shared mental models and consensus are offered as methods to equalize participation in team decision making, and an overview of other methods provided. The last section will look at ways to coordinate the stages of team development with the variety of problemsolving and decision-making techniques in order to maximize a team’s effectiveness.

• Team Member
• Team Development
• Power Distance
• Virtual Team

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, log in via an institution .

Buying options

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Unable to display preview.  Download preview PDF.

Adler, N.J. (2002). International Dimensions of Organizational Behavior (4th ed.). Cincinnati, OH: South-Western.

Google Scholar  

Cooperrider, D.L., Whitney, D. and Stavros, J.M. (2005). Appreciative Inquiry: The First in a Series of AI Workbooks for Leaders of Change. Brunswick, OH: Crown Custom Publishing.

Enayati, J. (2001). The research: Effective communication and decision-making in diverse groups. In Hemmati, M. (Ed.), Multi-Stakeholder Processes for Governance and Sustainability-Beyond Deadlock and Conflict. London, England: Earthscan.

Gardenswartz, L. and Rowe, A. (2003). In L. Gardenswartz and A. Rowe (Eds.), Diverse Teams at Work: Capitalizing on the Power of Diversity (1st ed.). Alexandria, VA: Society for Human Resource Management.

Halverson, C.B. (2004). Effective Multicultural Teams (5th ed.). Brattleboro, VT: School for International Training.

Harrington-Macklin, D. (1994). The Team Building Tool Kit: Tips, Tactics, and Rules for Effective Workplace Teams. New York: American Management Association.

Hofstede, G. (1980). Culture’s Consequences: International Differences in Work-related Values. Thousand Oaks, CA: Sage.

Janis, I. (1982). Groupthink: Psychological Studies of Policy Decisions and Fiascos (2nd ed.). Boston, MA: Houghton Mifflin.

Jeffery, A.B., Maes, J.D. and Bratton-Jeffery, M.F. (2005). Improving team decision-making performance with collaborative modeling. [Electronic version]. Team Performance Management, 11 (1/2), 40–50. Retrieved December 20, 2005, from the Emerald In sight database.

Article   Google Scholar  

Kayser, T.A. (1994). Building Team Power: How to Unleash the Collaborative Genius of Work Teams. New York: Irwin.

Kelly, K.P. (1994). Team Decision Making Techniques. Irvine, CA: Richard Chang Associates.

Kline, T. (1999). In M. Holt, D. Ullius and P. Berkman (Eds.), Remaking Teams: The Revolutionary Research-based Guide That Puts Theory into Practice. San Francisco, CA: Jossey-Bass/Pfeiffer.

Magruder Watkins, J. and Mohr, B.J. (2001). Appreciative Inquiry, Change at the Speed of Imagination. San Francisco, CA: Jossey-Bass/Pfieffer.

Mathieu, J., Heffner, T., Goodwin, G., Cannon-Bowers, J. and Salas, E. (2005). Scaling the quality of teammates’ mental models: equifinality and normative comparisons. Journal of Organizational Behavior, 26, 37–56.

McFadzean, E. (2002). Developing and supporting creative problem-solving teams: Part 1—a conceptual model. [Electronic version]. Management Decision, 40(5), 463–475. Retrieved December 20, 2005, from the Emerald Insight database.

McFadzean, E. (2002). Developing and supporting creative problem solving teams: Part 2-facili-tator competencies. [Electronic version]. Management Decision, 40(6), 537–551. Retrieved December 20, 2005, from the Emerald Insight database.

McKenna, R.J. and Martin-Smith, B. (2005). Decision making as a simplification process: New conceptual perspectives.[Electronic version]. Management Decision, 43(6), 821–836. Retrieved December 20, 2005, from the Emerald Insight database.

Sagie, A. and Akcan, Z. (2003). A cross-cultural analysis of participative decision-making in organizations. Human Relations, 56(4), 453–473.

Selart, M. (2005). Understanding the role of locus of control in consultative decision-making: A case study. [Electronic version]. Management Decision, 43(3), 397–412. Retrieved December 20, 2005, from the Emerald Insight database.

Simon, T., Pelled, L.H. and Smith, K.A. (1999). Making use of difference: diversity, debate, and decision comprehensiveness in top management teams. Academy of Management Journal, 42 (6), 662–673.

Smith, P.B., Peterson, M.F. and Schwartz, S.H. (2002). Cultural values, sources of guidance, and their relevance to managerial behavior. [Electronic version]. Journal of Cross-Cultural Psychology, 33(2), 188–208.

Tomlinson, S. (1999). Comparison of consensus Japanese style and Quaker style. [Electronic version]. Retrieved July 1, 2006, from http://www.earlham.edu/~consense/scott2.shtml

Watkins, J.M. and Mohr, B.J. (2001). Appreciative Inquiry: Change at the Speed of Imagination. San Francisco, CA: Jossey-Bass/Pfeiffer.

Whitney, D. and Trosten-Bloom, A. (2003). The Power of Appreciative Inquiry: A Practical Guide to Positive Change. San Francisco, CA: Berrett-Koehler.

Download references

Author information

Authors and affiliations.

School for International Training, Brattleboro, VT, USA

Linda Drake Gobbo

You can also search for this author in PubMed   Google Scholar

Editor information

Editors and affiliations.

Claire B. Halverson  & S. Aqeel Tirmizi  & 

Rights and permissions

Reprints and permissions

Copyright information

© 2008 Springer Science + Business Media B.V

About this chapter

Cite this chapter.

Gobbo, L.D. (2008). Problem Solving and Decision Making. In: Halverson, C.B., Tirmizi, S.A. (eds) Effective Multicultural Teams: Theory and Practice. Advances in Group Decision and Negotiation, vol 3. Springer, Dordrecht. https://doi.org/10.1007/978-1-4020-6957-4_9

Download citation

DOI : https://doi.org/10.1007/978-1-4020-6957-4_9

Publisher Name : Springer, Dordrecht

Print ISBN : 978-1-4020-6956-7

Online ISBN : 978-1-4020-6957-4

eBook Packages : Business and Economics Business and Management (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research