research articles on software testing

• Search by keyword
• Search by citation

Page 1 of 2

Metric-centered and technology-independent architectural views for software comprehension

The maintenance of applications is a crucial activity in the software industry. The high cost of this process is due to the effort invested on software comprehension since, in most of cases, there is no up-to-...

• View Full Text

Back to the future: origins and directions of the “Agile Manifesto” – views of the originators

In 2001, seventeen professionals set up the manifesto for agile software development. They wanted to define values and basic principles for better software development. On top of being brought into focus, the ...

Investigating the effectiveness of peer code review in distributed software development based on objective and subjective data

Code review is a potential means of improving software quality. To be effective, it depends on different factors, and many have been investigated in the literature to identify the scenarios in which it adds qu...

On the benefits and challenges of using kanban in software engineering: a structured synthesis study

Kanban is increasingly being used in diverse software organizations. There is extensive research regarding its benefits and challenges in Software Engineering, reported in both primary and secondary studies. H...

Challenges on applying genetic improvement in JavaScript using a high-performance computer

Genetic Improvement is an area of Search Based Software Engineering that aims to apply evolutionary computing operators to the software source code to improve it according to one or more quality metrics. This ...

Actor’s social complexity: a proposal for managing the iStar model

Complex systems are inherent to modern society, in which individuals, organizations, and computational elements relate with each other to achieve a predefined purpose, which transcends individual goals. In thi...

Investigating measures for applying statistical process control in software organizations

The growing interest in improving software processes has led organizations to aim for high maturity, where statistical process control (SPC) is required. SPC makes it possible to analyze process behavior, pred...

An approach for applying Test-Driven Development (TDD) in the development of randomized algorithms

TDD is a technique traditionally applied in applications with deterministic algorithms, in which the input and the expected result are known. However, the application of TDD with randomized algorithms have bee...

Supporting governance of mobile application developers from mining and analyzing technical questions in stack overflow

There is a need to improve the direct communication between large organizations that maintain mobile platforms (e.g. Apple, Google, and Microsoft) and third-party developers to solve technical questions that e...

Working software over comprehensive documentation – Rationales of agile teams for artefacts usage

Agile software development (ASD) promotes working software over comprehensive documentation. Still, recent research has shown agile teams to use quite a number of artefacts. Whereas some artefacts may be adopt...

Development as a journey: factors supporting the adoption and use of software frameworks

From the point of view of the software framework owner, attracting new and supporting existing application developers is crucial for the long-term success of the framework. This mixed-methods study explores th...

Applying user-centered techniques to analyze and design a mobile application

Techniques that help in understanding and designing user needs are increasingly being used in Software Engineering to improve the acceptance of applications. Among these techniques we can cite personas, scenar...

A measurement model to analyze the effect of agile enterprise architecture on geographically distributed agile development

Efficient and effective communication (active communication) among stakeholders is thought to be central to agile development. However, in geographically distributed agile development (GDAD) environments, it c...

A survey of search-based refactoring for software maintenance

This survey reviews published materials related to the specific area of Search-Based Software Engineering that concerns software maintenance and, in particular, refactoring. The survey aims to give a comprehen...

Guest editorial foreword for the special issue on automated software testing: trends and evidence

Similarity testing for role-based access control systems.

Access control systems demand rigorous verification and validation approaches, otherwise, they can end up with security breaches. Finite state machines based testing has been successfully applied to RBAC syste...

An algorithm for combinatorial interaction testing: definitions and rigorous evaluations

Combinatorial Interaction Testing (CIT) approaches have drawn attention of the software testing community to generate sets of smaller, efficient, and effective test cases where they have been successful in det...

How diverse is your team? Investigating gender and nationality diversity in GitHub teams

Building an effective team of developers is a complex task faced by both software companies and open source communities. The problem of forming a “dream”

Investigating factors that affect the human perception on god class detection: an analysis based on a family of four controlled experiments

Evaluation of design problems in object oriented systems, which we call code smells, is mostly a human-based task. Several studies have investigated the impact of code smells in practice. Studies focusing on h...

On the evaluation of code smells and detection tools

Code smells refer to any symptom in the source code of a program that possibly indicates a deeper problem, hindering software maintenance and evolution. Detection of code smells is challenging for developers a...

On the influence of program constructs on bug localization effectiveness

Software projects often reach hundreds or thousands of files. Therefore, manually searching for code elements that should be changed to fix a failure is a difficult task. Static bug localization techniques pro...

DyeVC: an approach for monitoring and visualizing distributed repositories

Software development using distributed version control systems has become more frequent recently. Such systems bring more flexibility, but also greater complexity to manage and monitor multiple existing reposi...

A genetic algorithm based framework for software effort prediction

Several prediction models have been proposed in the literature using different techniques obtaining different results in different contexts. The need for accurate effort predictions for projects is one of the ...

Elaboration of software requirements documents by means of patterns instantiation

Studies show that problems associated with the requirements specifications are widely recognized for affecting software quality and impacting effectiveness of its development process. The reuse of knowledge ob...

ArchReco: a software tool to assist software design based on context aware recommendations of design patterns

This work describes the design, development and evaluation of a software Prototype, named ArchReco, an educational tool that employs two types of Context-aware Recommendations of Design Patterns, to support us...

On multi-language software development, cross-language links and accompanying tools: a survey of professional software developers

Non-trivial software systems are written using multiple (programming) languages, which are connected by cross-language links. The existence of such links may lead to various problems during software developmen...

SoftCoDeR approach: promoting Software Engineering Academia-Industry partnership using CMD, DSR and ESE

The Academia-Industry partnership has been increasingly encouraged in the software development field. The main focus of the initiatives is driven by the collaborative work where the scientific research work me...

Issues on developing interoperable cloud applications: definitions, concepts, approaches, requirements, characteristics and evaluation models

Among research opportunities in software engineering for cloud computing model, interoperability stands out. We found that the dynamic nature of cloud technologies and the battle for market domination make clo...

Game development software engineering process life cycle: a systematic review

Software game is a kind of application that is used not only for entertainment, but also for serious purposes that can be applicable to different domains such as education, business, and health care. Multidisc...

Correlating automatic static analysis and mutation testing: towards incremental strategies

Traditionally, mutation testing is used as test set generation and/or test evaluation criteria once it is considered a good fault model. This paper uses mutation testing for evaluating an automated static anal...

A multi-objective test data generation approach for mutation testing of feature models

Mutation approaches have been recently applied for feature testing of Software Product Lines (SPLs). The idea is to select products, associated to mutation operators that describe possible faults in the Featur...

An extended global software engineering taxonomy

In Global Software Engineering (GSE), the need for a common terminology and knowledge classification has been identified to facilitate the sharing and combination of knowledge by GSE researchers and practition...

A systematic process for obtaining the behavior of context-sensitive systems

Context-sensitive systems use contextual information in order to adapt to the user’s current needs or requirements failure. Therefore, they need to dynamically adapt their behavior. It is of paramount importan...

Distinguishing extended finite state machine configurations using predicate abstraction

Extended Finite State Machines (EFSMs) provide a powerful model for the derivation of functional tests for software systems and protocols. Many EFSM based testing problems, such as mutation testing, fault diag...

Extending statecharts to model system interactions

Statecharts are diagrams comprised of visual elements that can improve the modeling of reactive system behaviors. They extend conventional state diagrams with the notions of hierarchy, concurrency and communic...

On the relationship of code-anomaly agglomerations and architectural problems

Several projects have been discontinued in the history of the software industry due to the presence of software architecture problems. The identification of such problems in source code is often required in re...

An approach based on feature models and quality criteria for adapting component-based systems

Feature modeling has been widely used in domain engineering for the development and configuration of software product lines. A feature model represents the set of possible products or configurations to apply i...

Patch rejection in Firefox: negative reviews, backouts, and issue reopening

Writing patches to fix bugs or implement new features is an important software development task, as it contributes to raise the quality of a software system. Not all patches are accepted in the first attempt, ...

Investigating probabilistic sampling approaches for large-scale surveys in software engineering

Establishing representative samples for Software Engineering surveys is still considered a challenge. Specialized literature often presents limitations on interpreting surveys’ results, mainly due to the use o...

Characterising the state of the practice in software testing through a TMMi-based process

The software testing phase, despite its importance, is usually compromised by the lack of planning and resources in industry. This can risk the quality of the derived products. The identification of mandatory ...

Self-adaptation by coordination-targeted reconfigurations

A software system is self-adaptive when it is able to dynamically and autonomously respond to changes detected either in its internal components or in its deployment environment. This response is expected to ensu...

Templates for textual use cases of software product lines: results from a systematic mapping study and a controlled experiment

Use case templates can be used to describe functional requirements of a Software Product Line. However, to the best of our knowledge, no efforts have been made to collect and summarize these existing templates...

F3T: a tool to support the F3 approach on the development and reuse of frameworks

Frameworks are used to enhance the quality of applications and the productivity of the development process, since applications may be designed and implemented by reusing framework classes. However, frameworks ...

NextBug: a Bugzilla extension for recommending similar bugs

Due to the characteristics of the maintenance process followed in open source systems, developers are usually overwhelmed with a great amount of bugs. For instance, in 2012, approximately 7,600 bugs/month were...

Assessing the benefits of search-based approaches when designing self-adaptive systems: a controlled experiment

The well-orchestrated use of distilled experience, domain-specific knowledge, and well-informed trade-off decisions is imperative if we are to design effective architectures for complex software-intensive syst...

Revealing influence of model structure and test case profile on the prioritization of test cases in the context of model-based testing

Test case prioritization techniques aim at defining an order of test cases that favor the achievement of a goal during test execution, such as revealing failures as earlier as possible. A number of techniques ...

A metrics suite for JUnit test code: a multiple case study on open source software

The code of JUnit test cases is commonly used to characterize software testing effort. Different metrics have been proposed in literature to measure various perspectives of the size of JUnit test cases. Unfort...

Designing fault-tolerant SOA based on design diversity

Over recent years, software developers have been evaluating the benefits of both Service-Oriented Architecture (SOA) and software fault tolerance techniques based on design diversity. This is achieved by creat...

Method-level code clone detection through LWH (Light Weight Hybrid) approach

Many researchers have investigated different techniques to automatically detect duplicate code in programs exceeding thousand lines of code. These techniques have limitations in finding either the structural o...

The problem of conceptualization in god class detection: agreement, strategies and decision drivers

The concept of code smells is widespread in Software Engineering. Despite the empirical studies addressing the topic, the set of context-dependent issues that impacts the human perception of what is a code sme...

• Editorial Board
• Sign up for article alerts and news from this journal

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

A comprehensive bibliometric assessment on software testing (2016–2021).

1. Introduction

2. related work, 3. methodology, 3.1. creation of two distinguished datasets for two different time spans, 3.2. research questions for the analysis of datasets, 4. research findings, 4.1. year-wise scientific production, 4.2. top 20 publication venues, 4.3. types of documents, 4.4. top 20 web of science categories based on the publications count, 4.5. top 20 research areas in accordance with the record count of publications, 4.6. leading 20 institutions/organizations based on the frequency of publications, 4.7. the top 20 most actively contributing countries based on the frequency of publications, 4.8. continent-wise research contribution, 4.9. language of the publications, 4.10. collaboration network amongst countries, 4.11. correlation of documents on the basis of co-words, 4.12. research themes/topics, 5. future work and limitations of the research study, 5.1. future work, 5.2. limitations of the study.

• Limited Time Frame: We have included the research publications for the six-year timeframes of the WoS database 2016–2021. Therefore, the paper does not include the research studies for the time duration before 2016.
• Limitations of sub-domain of SE: We have a limited or bibliometric assessment on Software Testing only. However, there are many other sub-domains of Software Engineering that need to be analyzed in future works.
• Use of ISI Web of Science (WoS): We have used one of the most commonly used and highly privileged databases, which is ISI Web of Science. Other databases can also be used.
• Twelve research questions: Analysis on the basis of 12 research questions can be enhanced to include other bibliometric assessment parameters.

6. Conclusions

Author contributions, conflicts of interest.

• Garousi, V. A bibliometric analysis of the Turkish software engineering research community. Scientometrics 2015 , 105 , 23–49. [ Google Scholar ] [ CrossRef ]
• Galler, B.A. ACM president’s letter: NATO and software engineering? Commun. ACM 1969 , 12 , 301. [ Google Scholar ] [ CrossRef ]
• Johnson, P.; Ekstedt MJacobson, I. Where’s the theory for software engineering? IEEE Softw. 2012 , 29 , 96. [ Google Scholar ] [ CrossRef ]
• Alam, S.; Zardari, S.; Bano, M. Software engineering and 12 prominent sub-areas: Comprehensive bibliometric assessment on 13 years (2007–2019). IET Softw. 2021 , 16 , 125–145. [ Google Scholar ] [ CrossRef ]
• Roger, S.P.; Bruce, R.M. Software Engineering: A Practitioner’s Approach ; McGraw-Hill Education: New York, NY, USA, 2005. [ Google Scholar ]
• Wasserman, A.I. Software engineering issues for mobile application development. In Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research, Santa Fe, NM, USA, 7–8 November 2010; Association for Computing Machinery: New York, NY, USA, 2010; pp. 397–400. [ Google Scholar ]
• Gregg, D.G.; Kulkarni, U.R.; Vinzé, A.S. Understanding the Philosophical Underpinnings of Software Engineering Research in Information Systems. Inf. Syst. Front. 2001 , 3 , 169–183. [ Google Scholar ] [ CrossRef ]
• Singh, S.K.; Singh, A. Software Testing ; Vandana Publications: Lucknow, India, 2012. [ Google Scholar ]
• Garousi, V.; Zhi, J. A survey of software testing practices in Canada. J. Syst. Softw. 2013 , 86 , 1354–1376. [ Google Scholar ] [ CrossRef ]
• Jindal, T. Importance of Testing in SDLC. Int. J. Eng. Appl. Comput. Sci. 2016 , 1 , 54–56. [ Google Scholar ] [ CrossRef ]
• Tan, T.B.; Cheng, W.K. December. Software testing levels in internet of things (IoT) architecture. In International Computer Symposium ; Springer: Singapore, 2018; pp. 385–390. [ Google Scholar ]
• Hamza, Z.; Hammad, M. Testing Approaches for Web and Mobile Applications: An Overview. Int. J. Comput. Digit. Syst. 2020 , 9 , 657–664. [ Google Scholar ] [ CrossRef ]
• Chauhan, R.K.; Singh, I. Latest research and development on software testing techniques and tools. Int. J. Curr. Eng. Technol. 2014 , 4 , 2368–2372. [ Google Scholar ]
• Jayakumar, A.V.; Gautham, S.; Kuhn, R.; Simons, B.; Collins, A.; Dirsch, T.; Kacker, R.; Elks, C. Systematic software testing of critical embedded digital devices in nuclear power applications. In Proceedings of the 2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), Coimbra, Portugal, 12–15 October 2020; IEEE: Piscataway Township, NJ, USA, 2020; pp. 85–90. [ Google Scholar ]
• Nurul, M.; Quadri, S.M.K. Software Testing Approach for Cloud Applications (STACA)–Methodology, Techniques & Tools. In Proceedings of the 2019 9th International Conference on Cloud Computing, Data Science & Engineering (Confluence), Noida, India, 10–11 January 2019; IEEE: Piscataway Township, NJ, USA, 2019; pp. 19–25. [ Google Scholar ]
• Sanchez-Gomez, N.; Torres-Valderrama, J.; Garcia-Garcia, J.A.; Gutierrez, J.J.; Escalona, M.J. Model-Based Software Design and Testing in Blockchain Smart Contracts: A Systematic Literature Review. IEEE Access 2020 , 8 , 164556–164569. [ Google Scholar ] [ CrossRef ]
• Murad, G.; Badarneh, A.; Qusef, A.; Almasalha, F. Software testing techniques in iot. In Proceedings of the 2018 8th International Conference on Computer Science and Information Technology (CSIT), Amman, Jordan, 11–12 July 2018; IEEE: Piscataway Township, NJ, USA, 2018; pp. 17–21. [ Google Scholar ]
• Górski, T. The 1+5 Architectural Views Model in Designing Blockchain and IT System Integration Solutions. Symmetry 2021 , 13 , 2000. [ Google Scholar ] [ CrossRef ]
• Shahin, M.; Babar, M.A.; Zhu, L. Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices. IEEE Access 2017 , 5 , 3909–3943. [ Google Scholar ] [ CrossRef ]
• Górski, T. Continuous Delivery of Blockchain Distributed Applications. Sensors 2021 , 22 , 128. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Schermann, G.; Schöni, D.; Leitner, P.; Gall, H.C. Bifrost: Supporting continuous deployment with automated enactment of multi-phase live testing strategies. In Proceedings of the 17th International Middleware Conference, Trento, Italy, 12–16 December 2016; pp. 1–14. [ Google Scholar ]
• Merigó, J.M.; Yang, J.-B. A bibliometric analysis of operations research and management science. Omega 2017 , 73 , 37–48. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Alam, S.; Zardari, S.; Shamsi, J. Comprehensive three-phase bibliometric assessment on the blockchain (2012–2020). Libr. Hi Tech 2022 . [ Google Scholar ] [ CrossRef ]
• Tse, T.; Chen, T.; Glass, R.L. An assessment of systems and software engineering scholars and institutions (2000–2004). J. Syst. Softw. 2006 , 79 , 816–819. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Wohlin, C. An analysis of the most cited articles in software engineering journals—1999. Inf. Softw. Technol. 2005 , 47 , 957–964. [ Google Scholar ] [ CrossRef ]
• Wong, W.E.; Tse, T.; Glass, R.L.; Basili, V.R.; Chen, T. An assessment of systems and software engineering scholars and institutions (2002–2006). J. Syst. Softw. 2009 , 82 , 1370–1373. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Hamadicharef, B. Scientometric study of the IEEE transactions on software engineering 1980–2010. In Proceedings of the 2011 2nd International Congress on Computer Applications and Computational Science, Bali, Indonesia, 15–17 November 2011; Springer: Berlin, Heidelberg, 2012; pp. 101–106. [ Google Scholar ]
• Freitas, F.G.D.; Souza, J.T.D. Ten years of search based software engineering: A bibliometric analysis. In International Symposium on Search Based Software Engineering ; Springer: Berlin/Heidelberg, Germany, 2011; pp. 18–32. [ Google Scholar ]
• Garousi, V.; Mäntylä, M.V. Citations, research topics and active countries in software engineering: A bibliometrics study. Comput. Sci. Rev. 2016 , 19 , 56–77. [ Google Scholar ] [ CrossRef ]
• Karanatsiou, D.; Li, Y.; Arvanitou, E.-M.; Misirlis, N.; Wong, W.E. A bibliometric assessment of software engineering scholars and institutions (2010–2017). J. Syst. Softw. 2018 , 147 , 246–261. [ Google Scholar ] [ CrossRef ]
• Almaliki, M. Software Engineering in Saudi Arabia: A Bibliometric Assessment. IEEE Access 2021 , 9 , 17245–17255. [ Google Scholar ] [ CrossRef ]
• Wong, W.E.; Mittas, N.; Arvanitou, E.M.; Li, Y. A bibliometric assessment of software engineering themes, scholars and institutions (2013–2020). J. Syst. Softw. 2021 , 180 , 111029. [ Google Scholar ] [ CrossRef ]
• Mikki, S. Comparing Google Scholar and ISI Web of Science for Earth Sciences. Scientometrics 2009 , 82 , 321–331. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Van Eck, N.J.; Waltman, L. VOSviewer Manual ; Univeristeit Leiden: Leiden, The Netherlands, 2013; Volume 1, pp. 1–53. [ Google Scholar ]
• Ravikumar, S.; Agrahari, A.; Singh, S.N. Mapping the intellectual structure of scientometrics: A co-word analysis of the journal Scientometrics (2005–2010). Scientometrics 2014 , 102 , 929–955. [ Google Scholar ] [ CrossRef ]
• Cobo, M.J.; López-Herrera, A.G.; Herrera-Viedma, E.; Herrera, F. Science mapping software tools: Review, analysis, and cooperative study among tools. J. Am. Soc. Inf. Sci. Technol. 2011 , 62 , 1382–1402. [ Google Scholar ] [ CrossRef ]
• Wang, J.; Li, X.; Wang, P.; Liu, Q. Bibliometric analysis of digital twin literature: A review of influencing factors and conceptual structure. Technol. Anal. Strat. Manag. 2022 , 1–15. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Zardari, S.; Alam, S.; Al Salem, H.A.; Al Reshan, M.S.; Shaikh, A.; Malik, A.F.K.; Masood ur Rehman, M.; Mouratidis, H. A Comprehensive Bibliometric Assessment on Software Testing (2016–2021). Electronics 2022 , 11 , 1984. https://doi.org/10.3390/electronics11131984

Zardari S, Alam S, Al Salem HA, Al Reshan MS, Shaikh A, Malik AFK, Masood ur Rehman M, Mouratidis H. A Comprehensive Bibliometric Assessment on Software Testing (2016–2021). Electronics . 2022; 11(13):1984. https://doi.org/10.3390/electronics11131984

Zardari, Shehnila, Sana Alam, Hamad Abosaq Al Salem, Mana Saleh Al Reshan, Asadullah Shaikh, Aneeq Fayyaz Karim Malik, Muhammad Masood ur Rehman, and Haralambos Mouratidis. 2022. "A Comprehensive Bibliometric Assessment on Software Testing (2016–2021)" Electronics 11, no. 13: 1984. https://doi.org/10.3390/electronics11131984

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Software Testing

• First Online: 12 February 2019

Cite this chapter

• Gordon Fraser 4 &
• José Miguel Rojas 5  

3930 Accesses

16 Citations

6 Altmetric

Any nontrivial program contains some errors in the source code. These “bugs” are annoying for users if they lead to application crashes and data loss, and they are worrisome if they lead to privacy leaks and security exploits. The economic damage caused by software bugs can be huge, and when software controls safety critical systems such as automotive software, then bugs can kill people. The primary tool to reveal and eliminate bugs is software testing: Testing a program means executing it with a selected set of inputs and checking whether the program behaves in the expected way; if it does not, then a bug has been detected. The aim of testing is to find as many bugs as possible, but it is a difficult task as it is impossible to run all possible tests on a program. The challenge of being a good tester is thus to identify which are the best tests that help us find bugs, and to execute them as efficiently as possible. In this chapter, we explore different ways to measure how “good” a set of tests is, as well as techniques to generate good sets of tests.

All authors have contributed equally to this chapter.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Unable to display preview.  Download preview PDF.

Similar content being viewed by others

Just Test What You Cannot Verify!

Second Competition on Software Verification

Status Report on Software Verification

Acree, A.T.: On mutation. PhD thesis, Georgia Institute of Technology, Atlanta, GA (1980)

Google Scholar  

Afzal, W., Torkar, R., Feldt, R.: A systematic review of search-based testing for non-functional system properties. Inf. Softw. Technol. 51 (6), 957–976 (2009)

Article   Google Scholar  

Ali, S., Briand, L.C., Hemmati, H., Panesar-Walawege, R.K.: A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Softw. Eng. 36 (6), 742–762 (2010)

Ammann, P., Offutt, J.: Introduction to Software Testing. Cambridge University Press, Cambridge (2016)

Book   Google Scholar  

Ammann, P., Offutt, J., Huang, H.: Coverage criteria for logical expressions. In: IEEE International Symposium on Software Reliability Engineering (ISSRE), pp. 99–107 (2003)

Andrews, J.H., Briand, L.C., Labiche, Y.: Is mutation an appropriate tool for testing experiments? In: ACM/IEEE International Conference on Software Engineering (ICSE), pp. 402–411 (2005)

Arcuri, A.: It really does matter how you normalize the branch distance in search-based software testing. Softw. Test. Verification Reliab. 23 (2), 119–147 (2013)

Arcuri, A., Briand, L.: Adaptive random testing: an illusion of effectiveness? In: ACM International Symposium on Software Testing and Analysis (ISSTA), pp. 265–275 (2011)

Avritzer, A., Weyuker, E.: The automatic generation of load test suites and the assessment of the resulting software. IEEE Trans. Softw. Eng. 21 (9), 705–716 (1995)

Barr, E.T., Harman, M., McMinn, P., Shahbaz, M., Yoo, S.: The oracle problem in software testing: a survey. IEEE Trans. Softw. Eng. 41 (5), 507–525 (2015)

Beck, K.: Kent Beck’s Guide to Better Smalltalk: A Sorted Collection, vol. 14. Cambridge University Press, Cambridge (1999)

Beck, K.: Test-Driven Development: By Example. Addison-Wesley Professional, Upper Saddle River (2003)

Bell, J., Kaiser, G.: Unit test virtualization with VMVM. In: ACM/IEEE International Conference on Software Engineering (ICSE), pp. 550–561. ACM, New York (2014)

Bird, D.L., Munoz, C.U.: Automatic generation of random self-checking test cases. IBM Syst. J. 22 (3), 229–245 (1983)

Briand, L.C., Labiche, Y., He, S.: Automating regression test selection based on UML designs. Inf. Softw. Technol. 51 (1), 16–30 (2009)

Broekman, B., Notenboom, E.: Testing Embedded Software. Pearson Education, Boston (2003)

Bron, A., Farchi, E., Magid, Y., Nir, Y., Ur, S.: Applications of synchronization coverage. In: Proceedings of the Tenth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP), pp. 206–212. ACM, New York (2005)

Bühler, O., Wegener, J.: Evolutionary functional testing. Comput. Oper. Res. 35 (10), 3144–3160 (2008)

Cadar, C., Dunbar, D., Engler, D.: KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In: Symposium on Operating Systems Design and Implementation (USENIX), pp. 209–224 (2008)

Cadar, C., Godefroid, P., Khurshid, S., Pǎsǎreanu, C.S., Sen, K., Tillmann, N., Visser, W.: Symbolic execution for software testing in practice: preliminary assessment. In: ACM/IEEE International Conference on Software Engineering (ICSE), pp. 1066–1071 (2011)

Chen, T.Y., Cheung, S.C., Yiu, S.M.: Metamorphic testing: a new approach for generating next test cases. Department of Computer Science, Hong Kong University of Science and Technology, Technical Report HKUST-CS98-01 (1998)

Chen, T.Y., Lau, M.F., Yu, Y.T.: Mumcut: a fault-based strategy for testing boolean specifications. In: Sixth Asia Pacific Software Engineering Conference (APSEC), pp. 606–613. IEEE, New York (1999)

Chen, T.Y., Leung, H., Mak, I.K.: Adaptive random testing. In: Advances in Computer Science, pp. 320–329 (2004)

Chapter   Google Scholar  

Chilenski, J.J.: An investigation of three forms of the modified condition decision coverage (MCDC) criterion. Technical Report, DTIC Document (2001)

Chilenski, J.J., Miller, S.P.: Applicability of modified condition/decision coverage to software testing. Softw. Eng. J. 9 (5), 193–200 (1994)

Choudhary, S.R., Gorla, A., Orso, A.: Automated test input generation for android: are we there yet?(e). In: IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 429–440. IEEE, New York (2015)

Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. 4 (3), 178 (1978)

Article   MATH   Google Scholar  

Ciupa, I., Leitner, A., Oriol, M., Meyer, B.: Experimental assessment of random testing for object-oriented software. In: ACM International Symposium on Software Testing and Analysis (ISSTA), pp. 84–94 (2007)

Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. ACM Sigplan Not. 46 (4), 53–64 (2011)

Clarke, D., Lee, I.: Testing real-time constraints in a process algebraic setting. In: ACM/IEEE International Conference on Software Engineering (ICSE), pp. 51–60. ACM, New York (1995)

Clarke, L.A., Rosenblum, D.S.: A historical perspective on runtime assertion checking in software development. ACM SIGSOFT Softw. Eng. Notes 31 (3), 25–37 (2006)

Clarke, E.M., Grumberg, O., Peled, D.: Model Checking. MIT Press, Cambridge (1999)

Cohen, D.M., Dalal, S.R., Parelius, J., Patton, G.C.: The combinatorial design approach to automatic test generation. IEEE Softw. 13 (5), 83 (1996)

Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23 (7), 437–444 (1997)

Cohen, M., Gibbons, P., Mugridge, W., Colbourn, C.: Constructing test suites for interaction testing. In: ACM/IEEE International Conference on Software Engineering (ICSE), pp. 38–48 (2003)

Commission, I.E., et al.: IEC 61508: functional safety of electrical. Electronic/Programmable Electronic Safety-Related Systems (1999)

DeMillo, R.A., Offutt, A.J.: Constraint-based automatic test data generation. IEEE Trans. Softw. Eng. 17 (9), 900–910 (1991)

DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. Computer 11 (4), 34–41 (1978)

Derderian, K., Hierons, R.M., Harman, M., Guo, Q.: Automated unique input output sequence generation for conformance testing of FSMs. Comput. J. 49 (3), 331–344 (2006)

Di Lucca, G.A., Fasolino, A.R.: Testing web-based applications: the state of the art and future trends. Inf. Softw. Technol. 48 (12), 1172–1186 (2006)

Dowson, M.: The Ariane 5 software failure. ACM SIGSOFT Softw. Eng. Notes 22 (2), 84 (1997)

Duran, J.W., Ntafos, S.C.: An evaluation of random testing. IEEE Trans. Softw. Eng. SE-10 (4), 438–444 (1984)

Edelstein, O., Farchi, E., Nir, Y., Ratsaby, G., Ur, S.: Multithreaded java program test generation. IBM Syst. J. 41 (1), 111–125 (2002)

Elbaum, S., Malishevsky, A.G., Rothermel, G.: Test case prioritization: a family of empirical studies. IEEE Trans. Softw. Eng. 28 (2), 159–182 (2002)

Faught, D.R.: Keyword-driven testing. Sticky minds. https://www.stickyminds.com/article/keyword-driven-testing [online]. Retrieved 28.10.2018

Ferguson, R., Korel, B.: The chaining approach for software test data generation. ACM Trans. Softw. Eng. Methodol. 5 (1), 63–86 (1996)

Forrester, J.E., Miller, B.P.: An empirical study of the robustness of windows nt applications using random testing. In: Proceedings of the 4th Conference on USENIX Windows Systems Symposium - Volume 4, WSS’00, p. 6. USENIX Association, Berkeley (2000)

Fosdick, L.D., Osterweil, L.J.: Data flow analysis in software reliability. ACM Comput. Surv. 8 (3), 305–330 (1976)

Article   MathSciNet   MATH   Google Scholar  

Fraser, G., Arcuri, A.: Whole test suite generation. IEEE Trans. Softw. Eng. 39 (2), 276–291 (2013)

Fraser, G., Walkinshaw, N.: Assessing and generating test sets in terms of behavioural adequacy. Softw. Test. Verification Reliab. 25 (8), 749–780 (2015)

Fraser, G., Wotawa, F.: Redundancy Based Test-Suite Reduction, pp. 291–305. Springer, Heidelberg (2007)

Fraser, G., Zeller, A.: Mutation-driven generation of unit tests and oracles. IEEE Trans. Softw. Eng. 28 (2), 278–292 (2012)

Fraser, G., Wotawa, F., Ammann, P.E.: Testing with model checkers: a survey. Softw. Test. Verification Reliab. 19 (3), 215–261 (2009)

Fraser, G., Staats, M., McMinn, P., Arcuri, A., Padberg, F.: Does automated unit test generation really help software testers? a controlled empirical study. ACM Trans. Softw. Eng. Methodol. 24 (4), 23 (2015)

Fujiwara, S., Bochmann, G.V., Khendek, F., Amalou, M., Ghedamsi, A.: Test selection based on finite state models. IEEE Trans. Softw. Eng. 17 (6), 591–603 (1991)

Gambi, A., Bell, J., Zeller, A.: Practical test dependency detection. In: 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), Västerås, Sweden, pp. 1–11 (2018)

Gargantini, A., Heitmeyer, C.: Using model checking to generate tests from requirements specifications. In: ACM Symposium on the Foundations of Software Engineering (FSE), pp. 146–162. Springer, Berlin (1999)

Godefroid, P., Klarlund, N., Sen, K.: DART: directed automated random testing. SIGPLAN Not. 40 (6), 213–223 (2005)

Godefroid, P., Kiezun, A., Levin, M.Y.: Grammar-based whitebox fuzzing. ACM Sigplan Not. 43 (6), 206–215 (2008a)

Godefroid, P., Levin, M., Molnar, D.: Automated whitebox fuzz testing. In: Network and Distributed System Security Symposium (NDSS), pp. 1–16 (2008b)

Goodenough, J.B., Gerhart, S.L.: Toward a theory of test data selection. IEEE Trans. Softw. Eng. SE-1 (2), 156–173 (1975)

Article   MathSciNet   Google Scholar  

Gotlieb, A., Botella, B., Rueher, M.: Automatic test data generation using constraint solving techniques. ACM SIGSOFT Softw. Eng. Notes 23 (2), 53–62 (1998)

Graves, T.L., Harrold, M.J., Kim, J.M., Porter, A., Rothermel, G.: An empirical study of regression test selection techniques. ACM Trans. Softw. Eng. Methodol. 10 (2), 184–208 (2001)

Grechanik, M., Fu, C., Xie, Q.: Automatically finding performance problems with feedback-directed learning software testing. In: Proceedings of the 34th International Conference on Software Engineering, pp. 156–166. IEEE Press, New York (2012)

Grindal, M., Offutt, J., Andler, S.: Combination testing strategies: a survey. Softw. Test. Verification Reliab. 15 (3), 167–199 (2005)

Gross, F., Fraser, G., Zeller, A.: Search-based system testing: high coverage, no false alarms. In: ACM International Symposium on Software Testing and Analysis (2012)

Gutjahr, W.J.: Partition testing vs. random testing: the influence of uncertainty. IEEE Trans. Softw. Eng. 25 (5), 661–674 (1999)

Gyori, A., Shi, A., Hariri, F., Marinov, D.: Reliable testing: detecting state-polluting tests to prevent test dependency. In: Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA), pp. 223–233. ACM, New York (2015)

Hamlet, R.: Random testing. In: Encyclopedia of Software Engineering, pp. 970–978. Wiley, New York (1994)

Hamlet, D., Taylor, R.: Partition testing does not inspire confidence. IEEE Trans. Softw. Eng. 16 (12), 1402–1411 (1990)

Hammoudi, M., Rothermel, G., Tonella, P.: Why do record/replay tests of web applications break? In: IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 180–190. IEEE, New York (2016)

Hanford, K.V.: Automatic generation of test cases. IBM Syst. J. 9 (4), 242–257 (1970)

Hao, S., Li, D., Halfond, W.G., Govindan, R.: Estimating mobile application energy consumption using program analysis. In: 35th International Conference on Software Engineering, pp. 92–101. IEEE, New York (2013)

Harman, M., Clark, J.: Metrics are fitness functions too. In: Proceedings. 10th International Symposium on, Software Metrics, pp. 58–69. IEEE, New York (2004)

Harman, M., Jones, B.F.: Search-based software engineering. Inf. Softw. Technol. 43 (14), 833–839 (2001)

Harrold, M.J., Rothermel, G.: Performing data flow testing on classes. ACM SIGSOFT Softw. Eng. Notes 19 (5), 154–163 (1994)

Harrold, M.J., Gupta, R., Soffa, M.L.: A methodology for controlling the size of a test suite. ACM Trans. Softw. Eng. Methodol. 2 (3), 270–285 (1993)

Herzig, K., Nagappan, N.: Empirically detecting false test alarms using association rules. In: ICSE SEIP (2015)

Herzig, K., Greiler, M., Czerwonka, J., Murphy, B.: The art of testing less without sacrificing quality. In: ACM/IEEE International Conference on Software Engineering (ICSE), pp. 483–493. IEEE Press, New York (2015)

Holzmann, G.J.: The model checker spin. IEEE Trans. Softw. Eng. 23 (5), 279–295 (1997)

Holzmann, G.J., H Smith, M.: Software model checking: extracting verification models from source code. Softw. Test. Verification Reliab. 11 (2), 65–79 (2001)

Howden, W.E.: Reliability of the path analysis testing strategy. IEEE Trans. Softw. Eng. SE-2 (3), 208–215 (1976)

Hsu, H., Orso, A.: MINTS: a general framework and tool for supporting test-suite minimization. In: ACM/IEEE International Conference on Software Engineering (ICSE), pp. 419–429 (2009)

Huang, J.C.: An approach to program testing. ACM Comput. Surv. 7 (3), 113–128 (1975)

Humble, J., Farley, D.: Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Pearson Education, Boston (2010)

Inozemtseva, L., Holmes, R.: Coverage is not strongly correlated with test suite effectiveness. In: ACM/IEEE International Conference on Software Engineering (ICSE), pp. 435–445. ACM, New York (2014)

Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. Technical Report TR-09-06, CREST Centre, King’s College London, London (2009)

Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37 (5), 649–678 (2011)

Jiang, B., Zhang, Z., Chan, W., Tse, T.: Adaptive random test case prioritization. In: IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 233–244 (2009)

Johnson, L.A., et al.: Do-178b, software considerations in airborne systems and equipment certification. Crosstalk, October 1998

Jones, B.F., Sthamer, H.H., Eyres, D.E.: Automatic structural testing using genetic algorithms. Softw. Eng. J. 11 (5), 299–306 (1996)

Just, R., Schweiggert, F., Kapfhammer, G.: Major: an efficient and extensible tool for mutation analysis in a Java compiler. In: IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 612–615 (2011)

Just, R., Jalali, D., Inozemtseva, L., Ernst, M., Holmes, R., Fraser, G.: Are mutants a valid substitute for real faults in software testing? In: ACM Symposium on the Foundations of Software Engineering (FSE), pp. 654–665 (2014)

Kane, S., Liberman, E., DiViesti, T., Click, F.: Toyota Sudden Unintended Acceleration. Safety Research & Strategies, Rehoboth (2010)

Kaner, C.: Exploratory testing. In: Quality Assurance Institute Worldwide Annual Software Testing Conference (2006)

King, J.C.: Symbolic execution and program testing. Commun. ACM 19 (7), 385–394 (1976)

Korel, B.: Automated software test data generation. IEEE Trans. Softw. Eng. 16 , 870–879 (1990)

Korel, B., Tahat, L.H., Vaysburg, B.: Model based regression test reduction using dependence analysis. In: IEEE International Conference on Software Maintenance (ICSM), pp. 214–223 (2002)

Korel, B., Tahat, L.H., Harman, M.: Test prioritization using system models. In: IEEE International Conference on Software Maintenance (ICSM), pp. 559–568 (2005)

Krichen, M., Tripakis, S.: Conformance testing for real-time systems. Formal Methods Syst. Des. 34 (3), 238–304 (2009)

Kuhn, D., Wallace, D., Gallo, A. Jr.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. 30 (6), 418–421 (2004)

Labuschagne, A., Inozemtseva, L., Holmes, R.: Measuring the cost of regression testing in practice: a study of java projects using continuous integration. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE, pp. 821–830. ACM, New York (2017)

Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines-a survey. Proc. IEEE 84 (8), 1090–1123 (1996)

Leitner, A., Oriol, M., Zeller, A., Ciupa, I., Meyer, B.: Efficient unit test case minimization. In: IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 417–420 (2007)

Leveson, N.G., Turner, C.S.: An investigation of the Therac-25 accidents. Computer 26 (7), 18–41 (1993)

Li, Z., Harman, M., Hierons, R.M.: Search algorithms for regression test case prioritization. IEEE Trans. Softw. Eng. 33 (4), 225–237 (2007)

Lieberman, H., Paternò, F., Klann, M., Wulf, V.: End-user development: an emerging paradigm. In: End User Development, pp. 1–8. Springer, Berlin (2006)

Liggesmeyer, P., Trapp. M.: Trends in embedded software engineering. IEEE Softw. 26 (3), 19–25 (2009)

Lu, S., Jiang, W., Zhou, Y.: A study of interleaving coverage criteria. In: The 6th Joint Meeting on European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering: Companion Papers, pp. 533–536. ACM, New York (2007)

Lu, S., Park, S., Seo, E., Zhou, Y.: Learning from mistakes: a comprehensive study on real world concurrency bug characteristics. ACM Sigplan Not. 43 (3), 329–339 (2008)

Luo, Q., Hariri, F., Eloussi, L., Marinov, D.: An empirical analysis of flaky tests. In: ACM Symposium on the Foundations of Software Engineering (FSE), pp. 643–653. ACM, New York (2014)

Mansour, N., Bahsoon, R., Baradhi, G.: Empirical comparison of regression test selection algorithms. J. Syst. Softw. 57 (1), 79–90 (2001)

McMillan, K.L.: Symbolic model checking. In: Symbolic Model Checking, pp. 25–60. Springer, Berlin (1993)

McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verification Reliab. 14 (2), 105–156 (2004)

Memon, A.M., Soffa, M.L., Pollack, M.E.: Coverage criteria for gui testing. ACM SIGSOFT Softw. Eng. Notes 26 (5), 256–267 (2001)

Memon, A., Banerjee, I., Nagarajan, A.: What test oracle should i use for effective gui testing? In: 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings, pp. 164–173. IEEE, New York (2003a)

Memon, A.M., Banerjee, I., Nagarajan, A.: Gui ripping: reverse engineering of graphical user interfaces for testing. In: WCRE, vol. 3, p. 260 (2003b)

Memon, A., Gao, Z., Nguyen, B., Dhanda, S., Nickell, E., Siemborski, R., Micco, J.: Taming Google-scale continuous testing. In: 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), pp. 233–242 (2017)

Meyer, B.: Applying’design by contract’. Computer 25 (10), 40–51 (1992)

Michael, C.C., McGraw, G., Schatz, M.A.: Generating software test data by evolution. IEEE Trans. Softw. Eng. 27 (12), 1085–1110 (2001)

Miller, J.C., Maloney, C.J.: Systematic mistake analysis of digital computer programs. Commun. ACM 6 (2), 58–63 (1963)

Miller, W., Spooner, D.L.: Automatic generation of floating-point test data. IEEE Trans. Softw. Eng. 2 (3), 223–226 (1976)

Miller, B.P., Fredriksen, L., So, B.: An empirical study of the reliability of unix utilities. Commun. ACM 33 (12), 32–44 (1990)

Mirarab, S., Akhlaghi, S., Tahvildari, L.: Size-constrained regression test case selection using multicriteria optimization. IEEE Trans. Softw. Eng. 38 (4), 936–956 (2012)

Musuvathi, M., Qadeer, S., Ball, T., Basler, G., Nainar, P.A., Neamtiu, I.: Finding and reproducing Heisenbugs in concurrent programs. In: Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI), pp. 267–280. USENIX Association, Berkeley (2008)

Myers, G.: The Art of Software Testing. Wiley, New York (1979)

Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43 (2), 1–29 (2011). Article 11

Nielsen, B., Skou, A.: Automated test generation from timed automata. In: International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pp. 343–357. Springer, Berlin (2001)

Nistor, A., Luo, Q., Pradel, M., Gross, T.R., Marinov, D.: BALLERINA: automatic generation and clustering of efficient random unit tests for multithreaded code. In: ACM/IEEE International Conference on Software Engineering (ICSE), pp. 727–737 (2012)

North, D.: Behavior modification: the evolution of behavior-driven development. Better Softw. 8 (3) (2006). https://www.stickyminds.com/better-software-magazine-volume-issue/2006-03

Ntafos, S.C.: A comparison of some structural testing strategies. IEEE Trans. Softw. Eng. 14 (6), 868 (1988)

Offutt, J., Abdurazik, A.: Generating tests from UML specifications. In: International Conference on the Unified Modeling Language: Beyond the Standard (UML), pp. 416–429. Springer, Berlin (1999)

Offutt, A.J., Untch, R.H.: Mutation 2000: uniting the orthogonal. In: Wong, W.E. (ed.) Mutation Testing for the New Century, pp. 34–44. Kluwer Academic Publishers, Boston (2001)

Offutt, A.J, Rothermel, G., Zapf, C.: An experimental evaluation of selective mutation. In: ACM/IEEE International Conference on Software Engineering (ICSE), pp. 100–107 (1993)

Osterweil, L.J., Fosdick, L.D.: Dave–a validation error detection and documentation system for fortran programs. Softw. Pract. Exp. 6 (4), 473–486 (1976)

Ostrand, T.J., Balcer, M.J.: The category-partition method for specifying and generating functional tests. Commun. ACM 31 (6), 676–686 (1988)

Pacheco, C., Ernst, M.D.: Randoop: feedback-directed random testing for Java. In: ACM SIGPLAN Conference on Object-Oriented Programming Systems and Application (OOPSLA Companion), pp. 815–816. ACM, New York (2007)

Pacheco, C., Lahiri, S.K., Ernst, M.D., Ball, T.: Feedback-directed random test generation. In: ACM/IEEE International Conference on Software Engineering (ICSE), pp. 75–84 (2007)

Panichella, A., Oliveto, R., Penta, M.D., Lucia, A.D.: Improving multi-objective test case selection by injecting diversity in genetic algorithms. IEEE Trans. Softw. Eng. 41 (4), 358–383 (2015)

Pargas, R.P., Harrold, M.J., Peck, R.: Test-data generation using genetic algorithms. Softw. Test. Verification Reliab. 9 (4), 263–282 (1999)

Potter, B., McGraw, G.: Software security testing. IEEE Secur. Priv. 2 (5), 81–85 (2004)

Rapps, S., Weyuker, E.J.: Selecting software test data using data flow information. IEEE Trans. Softw. Eng. (4), 367–375 (1985). https://doi.org/10.1109/TSE.1985.232226

Rothermel, G., Harrold, M.J.: A safe, efficient regression test selection technique. ACM Trans. Softw. Eng. Methodol. 6 (2), 173–210 (1997)

Rothermel, G., Harrold, M.J., Ostrin, J., Hong, C.: An empirical study of the effects of minimization on the fault detection capabilities of test suites. In: IEEE International Conference on Software Maintenance (ICSM), pp. 34–43 (1998)

Rothermel, G., Untch, R.H., Chu, C., Harrold, M.J.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27 (10), 929–948 (2001)

Sabnani, K., Dahbura, A.: A protocol test generation procedure. Comput Netw. ISDN Syst. 15 (4), 285–297 (1988)

Segura, S., Fraser, G., Sanchez, A., Ruiz-Cortes, A.: A survey on metamorphic testing. IEEE Trans. Softw. Eng. (TSE) 42 (9), 805–824 (2016)

Sen, K., Marinov, D., Agha, G.: CUTE: a concolic unit testing engine for C. In: ACM Symposium on the Foundations of Software Engineering, pp. 263–272. ACM, New York (2005)

Shull, F., Melnik, G., Turhan, B., Layman, L., Diep, M., Erdogmus, H.: What do we know about test-driven development? IEEE Softw. 27 (6), 16–19 (2010). https://doi.org/10.1109/MS.2010.152

Sirer, E.G., Bershad, B.N.: Using production grammars in software testing. In: ACM SIGPLAN Notices, vol. 35, pp. 1–13. ACM, New York (1999)

Steenbuck, S., Fraser, G.: Generating unit tests for concurrent classes. In: IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 144–153. IEEE, New York (2013)

Stocks, P., Carrington, D.: A framework for specification-based testing. IEEE Trans. Softw. Eng. 22 (11), 777–793 (1996)

Sutton, M., Greene, A., Amini, P.: Fuzzing: Brute Force Vulnerability Discovery. Pearson Education, Boston (2007)

Tassey, G.: The economic impacts of inadequate infrastructure for software testing, final report. National Institute of Standards and Technology (2002)

Tikir, M.M., Hollingsworth, J.K.: Efficient instrumentation for code coverage testing. ACM SIGSOFT Softw. Eng. Notes 27 (4), 86–96 (2002)

Tonella, P.: Evolutionary testing of classes. In: ACM International Symposium on Software Testing and Analysis (ISSTA), pp. 119–128 (2004)

Tracey, N., Clark, J., Mander, K., McDermid, J.A.: An automated framework for structural test-data generation. In: IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 285–288 (1998)

Tretmans, J.: Test generation with inputs, outputs and repetitive quiescence (1996)

Tretmans, J., Brinksma, E.: TorX: automated model-based testing. In: First European Conference on Model-Driven Software Engineering, pp. 31–43 (2003)

Untch, R.H., Offutt, A.J., Harrold, M.J.: Mutation analysis using mutant schemata. ACM SIGSOFT Softw. Eng. Notes 18 (3), 139–148 (1993)

Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann, San Francisco (2010)

Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22 (5), 297–312 (2012)

Visser, W., Havelund, K., Brat, G., Park, S., Lerda, F.: Model checking programs. Autom. Softw. Eng. 10 (2), 203–232 (2003)

Visser, W., Pasareanu, C.S., Khurshid, S.: Test input generation with Java PathFinder. ACM SIGSOFT Softw. Eng. Notes 29 (4), 97–107 (2004)

Wegener, J., Baresel, A., Sthamer, H.: Evolutionary test environment for automatic structural testing. Inf. Softw. Technol. 43 (14), 841–854 (2001)

Weyuker, E.J.: On testing non-testable programs. Comput. J. 25 (4), 465–470 (1982)

Weyuker, E.J.: Assessing test data adequacy through program inference. ACM Trans. Program. Lang. Syst. 5 (4), 641–655 (1983)

Weyuker, E., Goradia, T., Singh, A.: Automatically generating test data from a boolean specification. IEEE Trans. Softw. Eng. 20 (5), 353 (1994)

White, L.J., Cohen, E.I.: A domain strategy for computer program testing. IEEE Trans. Softw. Eng. 6 (3), 247–257 (1980)

Whittaker, J.A., Arbon, J., Carollo, J.: How Google Tests Software. Addison-Wesley, Upper Saddle River (2012)

Wong, W.E., Horgan, J.R., London, S., Mathur, A.P.: Effect of test set minimization on fault detection effectiveness. Softw. Pract. Exper. 28 (4), 347–369 (1998)

Xanthakis, S., Ellis, C., Skourlas, C., Le Gall, A., Katsikas, S., Karapoulios, K.: Application of genetic algorithms to software testing. In: Proceedings of the 5th International Conference on Software Engineering and Applications, pp. 625–636 (1992)

Yang, C.S.D., Souter, A.L., Pollock, L.L.: All-du-path coverage for parallel programs. ACM SIGSOFT Softw. Eng. Notes 23 (2), 153–162 (1998)

Yoo, S., Harman, M.: Pareto efficient multi-objective test case selection. In: ACM International Symposium on Software Testing and Analysis (ISSTA), pp. 140–150. ACM, New York (2007)

Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab. 22 (2), 67–120 (2012)

Young, M., Pezze, M.: Software Testing and Analysis: Process, Principles and Techniques. Wiley, New York (2005)

Yuan, X., Cohen, M.B., Memon, A.M.: GUI interaction testing: incorporating event context. IEEE Trans. Softw. Eng. 37 (4), 559–574 (2011)

Zhang, P., Elbaum, S., Dwyer, M.B.: Automatic generation of load tests. In: Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, pp. 43–52. IEEE Computer Society, Washington (2011)

Zhang, S., Jalali, D., Wuttke, J., Muşlu, K., Lam, W., Ernst, M.D., Notkin, D.: Empirically revisiting the test independence assumption. In: ACM International Symposium on Software Testing and Analysis (ISSTA), pp. 385–396. ACM, New York (2014)

Zhu, H., Hall, P.A.V., May, J.H.R.: Software unit test coverage and adequacy. ACM Comput. Surv. 29 (4), 366–427 (1997)

Download references

Author information

Authors and affiliations.

University of Passau, Passau, Germany

Gordon Fraser

University of Leicester, Leicester, UK

José Miguel Rojas

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Gordon Fraser .

Editor information

Editors and affiliations.

College of Informatics, Korea University, Seoul, Korea (Republic of)

Sungdeok Cha

University of California, Irvine, CA, USA

Richard N. Taylor

Professor Emeritus, POSTECH, Pohang, Korea (Republic of)

Kyochul Kang

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Fraser, G., Rojas, J.M. (2019). Software Testing. In: Cha, S., Taylor, R., Kang, K. (eds) Handbook of Software Engineering. Springer, Cham. https://doi.org/10.1007/978-3-030-00262-6_4

Download citation

DOI : https://doi.org/10.1007/978-3-030-00262-6_4

Published : 12 February 2019

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-00261-9

Online ISBN : 978-3-030-00262-6

eBook Packages : Computer Science Computer Science (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Search-based software test data generation: a survey: Research Articles

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, view options.

• Xiao D Guo Y Li Y Chen L (2024) Optimizing Search-Based Unit Test Generation with Large Language Models: An Empirical Study Proceedings of the 15th Asia-Pacific Symposium on Internetware 10.1145/3671016.3674813 (71-80) Online publication date: 24-Jul-2024 https://dl.acm.org/doi/10.1145/3671016.3674813
• Alshahwan N Blasi A Bojarczuk K Ciancone A Gucevska N Harman M Krolikowski M Rojas R Martac D Schellaert S Ustiuzhanina K Harper I Jia Y Lewis W Roychoudhury A Paiva A Abreu R Storey M Aniche M Nagappan N (2024) Enhancing Testing at Meta with Rich-State Simulated Populations Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice 10.1145/3639477.3639729 (1-12) Online publication date: 14-Apr-2024 https://dl.acm.org/doi/10.1145/3639477.3639729
• Auer M Diner D Fraser G Li X Handl J (2024) Search-based Crash Reproduction for Android Apps Proceedings of the Genetic and Evolutionary Computation Conference 10.1145/3638529.3654034 (1426-1434) Online publication date: 14-Jul-2024 https://dl.acm.org/doi/10.1145/3638529.3654034
• Show More Cited By

Index Terms

Software and its engineering

Software creation and management

Software verification and validation

Software defect analysis

Software testing and debugging

Recommendations

Search based software test data generation for structural testing: a perspective.

Software testing is an important and expensive phase of the software development life cycle. Over the past few decades, there has been an ongoing research to automate the process of software testing but the attempts have been constrained by the size and ...

An extensive evaluation of search-based software testing: a review

In recent years, search-based software testing (SBST) is the active research topic in software testing. SBST is the process of generating test cases that use metaheuristics for optimization of a task in the framework of software testing to solve ...

Evolutionary algorithms for the multi-objective test data generation problem

Automatic test data generation is a very popular domain in the field of search-based software engineering. Traditionally, the main goal has been to maximize coverage. However, other objectives can be defined, such as the oracle cost, which is the cost ...

Information

Published in.

John Wiley and Sons Ltd.

United Kingdom

Publication History

Author tags.

• automated software test data generation
• evolutionary algorithms
• evolutionary testing
• metaheuristic search
• search-based software engineering
• simulated annealing

Contributors

Other metrics, bibliometrics, article metrics.

• 395 Total Citations View Citations
• 0 Total Downloads
• Downloads (Last 12 months) 0
• Downloads (Last 6 weeks) 0
• Mallissery S Wu Y (2023) Demystify the Fuzzing Methods: A Comprehensive Survey ACM Computing Surveys 10.1145/3623375 56 :3 (1-38) Online publication date: 5-Oct-2023 https://dl.acm.org/doi/10.1145/3623375
• Zhang M Belhadi A Arcuri A (2023) JavaScript SBST Heuristics to Enable Effective Fuzzing of NodeJS Web APIs ACM Transactions on Software Engineering and Methodology 10.1145/3593801 32 :6 (1-29) Online publication date: 24-Apr-2023 https://dl.acm.org/doi/10.1145/3593801
• Bui T Bui Q Do V Silva S Paquete L (2023) A Novel Fitness Function for Automated Software Test Case Generation Based on Nested Constraint Hardness Proceedings of the Companion Conference on Genetic and Evolutionary Computation 10.1145/3583133.3590727 (791-794) Online publication date: 15-Jul-2023 https://dl.acm.org/doi/10.1145/3583133.3590727
• Gereziher T Gebrekrstos S Gay G Silva S Paquete L (2023) Search-Based Test Generation Targeting Non-Functional Quality Attributes of Android Apps Proceedings of the Genetic and Evolutionary Computation Conference 10.1145/3583131.3590449 (1518-1526) Online publication date: 15-Jul-2023 https://dl.acm.org/doi/10.1145/3583131.3590449
• Feldmeier P Fraser G Silva S Paquete L (2023) Learning by Viewing: Generating Test Inputs for Games by Integrating Human Gameplay Traces in Neuroevolution Proceedings of the Genetic and Evolutionary Computation Conference 10.1145/3583131.3590448 (1508-1517) Online publication date: 15-Jul-2023 https://dl.acm.org/doi/10.1145/3583131.3590448
• Guizzo G Panichella S (2023) Fuzzing vs SBST ACM SIGSOFT Software Engineering Notes 10.1145/3573074.3573102 48 :1 (105-107) Online publication date: 17-Jan-2023 https://dl.acm.org/doi/10.1145/3573074.3573102
• Zohdinasab T Riccio V Gambi A Tonella P (2023) Efficient and Effective Feature Space Exploration for Testing Deep Learning Systems ACM Transactions on Software Engineering and Methodology 10.1145/3544792 32 :2 (1-38) Online publication date: 29-Mar-2023 https://dl.acm.org/doi/10.1145/3544792

View options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Share this publication link.

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

Artificial Intelligence in Software Test Automation: A Systematic Literature Review

International Journal of Emerging Technologies and Innovative Research (www.jetir.org | UGC and issn Approved), ISSN:2349-5162, Vol.6, Issue 12, page no. pp1329-1332, December-2019, Available at : http://www.jetir.org/papers/JETIR1912176.pdf

4 Pages Posted: 26 Jan 2022

Dhaya Sindhu Battina

Independent

Date Written: December 12, 2019

The main aim of this paper was to review how artificial intelligence works in software test automation.When it comes to software engineering, artificial intelligence (AI) has had a significant influence, and software testing is no exception. With artificial intelligence(AI), the goal of software test automation may be closer than ever before. To some extent, the paradigm has changed during the previous two decades [1]. Everything about the testing process has been a positive experience, starting with manual testing and progressing to automated testing, where Selenium is acknowledged to be one of the best test automation tools. As a result, in today's high-speed IT landscape software testing must come up with fresh testing approaches that are based on solid research. The emergence of AI-based testing has been very beneficial for this aim [1]. A computer's ability to learn without human involvement may be fully simulated by AI algorithms and machine learning (ML). While AI and ML entail the construction of distinct and unique algorithms to access data and learn from it by identifying patterns to make conclusions, these predictions are intended to be employed in software testing to their full potential [1].

Keywords: Artificial intelligence, automation, Software test automation, software engineering, AI systems

Suggested Citation: Suggested Citation

Dhaya Sindhu Battina (Contact Author)

Independent ( email ), do you have a job opening that you would like to promote on ssrn, paper statistics, related ejournals, sustainable technology ejournal.

Subscribe to this fee journal for more curated articles on this topic

Artificial Intelligence eJournal

Libraries & information technology ejournal, anthropology of science & technology studies ejournal.