Mozilla says 271 vulnerabilities found by Mythos have “almost no false positives”

As noted earlier, Mozilla’s characterization of AI-assisted vulnerability discovery as a game changer has been met with massive, vocal skepticism in many quarters. Critics initially scoffed when Mozilla didn’t obtain CVE designations for any of the 271 vulnerabilities. Like many developers, however, Mozilla doesn’t obtain CVE listings for internally discovered security bugs. Instead, they are bundled into a single patch. Normally, Bugzilla reports detailing these “rollups” are hidden for several months after being fixed to protect those who are slow to patch. Now that Mozilla has revealed a dozen of them, the same critics will surely claim they too were cherry-picked and conceal less accurate results.

Of the 271 bugs found using Mythos, 180 were sec-high, Mozilla’s highest designation for internally reported vulnerabilities. These types of vulnerabilities can be exploited through normal user behavior, such as browsing to a web page. (The only higher rating, sec-critical, is reserved for zero-days.) Another 80 were sec-moderate, and 11 were sec-low.

The critics are right to keep pushing back. Hype is a key method for inflating the already high puffed-up valuations of AI companies. Given the extensive praise Mozilla has given to Mythos, it’s easy for even more trusting people to wonder: What’s it getting in return? Far from settling the debate, Thursday’s elaborations are likely to only further stoke the controversy.

To hear Grinstead tell it, however, the details are clear evidence of the usefulness of AI-assisted discovery, and Mozilla’s motivation is simple.

“People are a bit burned from the last year of these slop commits so we felt it was important to show some of our work, open up some of the bugs, and talk about it in a little more detail as a way to hopefully spur some action or continue the conversation,” he said. “There’s no sort of marketing angle here. Our team has completely bought in on this approach. We are trying to get a message out about this technique in general and not any specific model provider, company, or anything like that.”