Internship: An Empirical Study of Bugs
Discription
Test code can have bugs, too! Who tests the tests? What is the prevalence of such bugs, and what is their impact? The paper "An Empirical Study of Bugs in Test Code" (Paper at IEEE International Conference on Software Maintenance 2015) finds that such bugs cause tests to fail incorrectly, thus causing false alarms. Can you replicate the results presented in the paper on another dataset, or in the settings of an industrial project?
Paper abstract
Testing aims at detecting (regression) bugs in production code. However, testing code is just as likely to contain bugs as the code it tests. Buggy test cases can silently miss bugs in the production code or loudly ring false alarms when the production code is correct. We present the first empirical study of bugs in test code to characterize their prevalence and root cause categories. We mine the bug repositories and version control systems of 211 Apache Software Foundation (ASF) projects and find 5,556 test-related bug reports. We (1) compare properties of test bugs with production bugs, such as active time and fixing effort needed, and (2) qualitatively study 443 randomly sampled test bug reports in detail and categorize them based on their impact and root causes. Our results show that (1) around half of all the projects had bugs in their test code; (2) the majority of test bugs are false alarms, i.e., test fails while the production code is correct, while a minority of these bugs result in silent horrors, i.e., test passes while the production code is incorrect; (3) incorrect and missing assertions are the dominant root cause of silent horror bugs; (4) semantic (25%), flaky (21%), environment-related (18%) bugs are the dominant root cause categories of false alarms; (5) the majority of false alarm bugs happen in the exercise portion of the tests, and (6) developers contribute more actively to fixing test bugs and test bugs are fixed sooner compared to production bugs. In addition, we evaluate whether existing bug detection tools can detect bugs in test code.
You will be embedded in the Research team of the Software Improvement Group. One of SIG's researchers will be appointed as a daily supervisor. Apart from daily supervision, you will interact with the other researchers on a regular basis. SIG is a dynamic, demanding, and rewarding working environment.
Students are expected to perform solid scientific work that is at the same time relevant for practitioners. You will get ample support and supervision and in return we expect you to learn fast and take responsibility for obtaining excellent results.
Apply directly on the site.