Mitigating Challenges in Experimentation with Machine Learning-Based Vulnerability Detectors by Eric Bodden

Abstract:
In recent years, a growing number of publications have reported promising results in statically detecting software vulnerabilities using machine learning. However, more
recent work challenges these findings, suggesting that many of the reported successes may not generalize to practical scenarios and could even originate purely from methodological shortcomings. In this paper, we systematically examine the literature on machine learning-based software vulnerability prediction, showcasing  recurring methodological issues in an example-driven approach. For each issue, we discuss its manifestation in published studies, its impact on result validity, and offer practical guidance for researchers seeking to avoid such pitfalls. To support this investigation, we conducted a systematic literature review based on established best practices, ultimately uncovering and analyzing nearly one hundred publications in the field.

Also a brief bio:
Eric Bodden is one of the leading experts on secure software engineering, with a specialty in building highly precise tools for automated program analysis. He is Professor for Secure Software Engineering at Paderborn University and director for Software Engineering and IT-Security at Fraunhofer IEM, where he is collaborating with the leading national and international software development companies. Prof. Bodden's research was awarded numerous times. For instance, in 2024 he received an ERC Advanced Grant, and in 2019, Bodden was named ACM Distinguished Member. At the German IT-Security Price, his group scored 1st place in 2016 and 2nd place in 2014. In 2014, the DFG awarded Bodden the Heinz Maier-Leibnitz-Preis, Germany's highest honour for young scientists. Prof. Bodden's research has received six ACM Distinguished Paper Awards and two Most Influential Paper Awards in different communities.