Malware Research: If You Cannot Replicate it, You Will Not Use It

Why aren't malware analysis practitioners making more use academic research results? In this webcast, we suggest that one reason is the general difficulty of replicating and reproducing research results in this field. We randomly selected 100 papers on "malware classification" from Google Scholar results and attempted to replicate each one. We were only able to find released code for 6 of these 100 papers, and what's worse, only 6 of the 88 remaining papers contained a specific listing for the algorithm. We offer suggestions for improving the state of the field and end with a call to action for researchers to improve their methods so that their work will be useful for everyone.

What Attendees Will Learn:

Why replication and reproduction are important topics in malware research
The problems with data in malware research and suggestions to fix them
The basics of using science in the field

Speaker and Presenter Information

Leigh Metcalf, PhD, is a Senior Network Security Research Analyst at the Carnegie Mellon University Software Engineering Institute�s CERT Division. Before joining SEI, Leigh spent more than 10 years in industry working as a systems engineer, architect, and security specialist. Dr. Metcalf has presented research at numerous conferences. She is the co-author (with William Casey) of the book Cybersecurity and Applied Mathematics, as the co-author (with Jonathan Spring) of Using Science in Cybersecurity, as well as co-author (with Gene Spafford and Josiah Dykstra) of the award winning book Cybersecurity: Myths and Misconceptions. She is also founding Co-Editor-in-Chief (with Arun Lakhotia) of the ACM journal Digital Threats: Research and Practice (DTRAP).

Edward J. Schwartz is a Senior Researcher at Carnegie Mellon University's Software Engineering Institute, where he has focused on binary analysis for the past eight years. Prior to that, he earned his PhD from Carnegie Mellon University's CyLab in 2014, where he maintained the Binary Analysis Platform for performing semantic analysis of executables. His dissertation examined the performance benefits of recovering abstractions from executables.

Dr. Schwartz publishes regularly in academic computer security conferences and has received several awards for his publications.

More recently, he has been collaborating with other researchers at CMU on a multidisciplinary effort to apply advances in machine learning to reverse engineering and binary analysis.