When a security scan flags a vulnerability, your team still needs to assess its impact and decide how to address it. We wanted to understand whether GPT-6 Cyber could help our existing ensemble find more issues worth investigating, and what that would mean for the reports developers receive.
We evaluated GPT-6 Cyber in CodeRabbit Secure, comparing our model ensemble with and without it on the same 33 benchmark cases. With Cyber, the ensemble caught 22 selected issues versus 19 without it and reported 66% more high- and medium-severity findings. The 66% increase counts issues flagged in the final reports. We still need to assess how many of the additional findings are real vulnerabilities.
CodeRabbit Secure scans your codebase for potential vulnerabilities and traces the code paths behind its findings. This evaluation extends our September code review evaluation to security scans.
66% more high- and medium-severity findings (65.6%, rounded). The known-issue catch rate increased by 9.1%. Some investigations required recovery runs.
Try Deep Scan
Catching vulnerabilities with a Deep Scan helps your team address security risks before they become very costly incidents.
What is GPT-6 Cyber?
GPT-6 Cyber is the cybersecurity variant of OpenAI’s GPT-6 models. Cyber models are trained for security research, including finding vulnerabilities and testing whether they can be exploited, with fewer refusals on authorized security tasks.
We received access for this evaluation through the OpenAI Daybreak Defense Network. Cyber models are part of OpenAI Daybreak, with access subject to approval, including additional model-specific approval in some cases.
How we evaluated GPT-6 Cyber in our security scans
We compared CodeRabbit’s model ensemble with and without GPT-6 Cyber on the same 33 cases. Each case had a known security issue we wanted the scan to find. The cases covered Python, JavaScript, TypeScript, Go, PHP, Rust, Ruby, C, and C#.
We evaluated what reached the developer in the final report, after the scan’s verification and reporting stages. A potential issue raised during investigation did not earn credit unless it appeared in that report with an explanation of how the code could fail.
We checked the final reports for two things:
- Known issues caught: We checked whether the report identified the issue selected for that case and explained its cause and security consequence. Each case could contribute one catch to this score. Flagging a risky function or finding a different issue did not count as catching the selected issue.
- Reported findings: We counted all findings across the 33 final reports for each configuration, including findings about issues other than the selected benchmark issue. We also counted the high- and medium-severity findings separately.
A scan could report several potential problems and still miss the known issue. Conversely, several findings about the selected issue would still count as only one benchmark catch. The catch score tells us how many of the known issues were found, while the findings count tells us how much output developers received.
For each case, we compared one final report from each configuration. The investigation stage used high reasoning effort to trace potential vulnerabilities through the code. We did not compare results at different effort settings.
Results
| Metric | Ensemble with GPT-6 Cyber | Ensemble without GPT-6 Cyber | Change |
| Known issues caught | 22/33 · 66.7% | 19/33 · 57.6% | +9.1% |
| High- and medium-severity reported findings | 361 | 218 | +65.6% |
| Reported findings at all severities | 979 | 592 | +65.4% |
The ensemble caught three more selected issues, a 15.8% relative increase, and reported 143 additional high- and medium-severity findings. We still need to assess how many of the additional findings describe valid vulnerabilities.
Where coverage changed
5 additional catches − 2 lost catches = 3 more overall. Each square represents one benchmark case. Squares show counts, not identifiable cases.
Five additional catches and two lost catches produced the net gain of three. Nine selected issues remained undetected by either configuration. These cases give us concrete targets for improving coverage while preserving existing catches.
Reported findings increased in all six selected categories. These are output counts, not category-specific detection rates.
Injection and server-side request forgery had the largest percentage increases. Denial of service added the most findings (98), followed by authorization bypass (77).
Discussion
The evaluation records a net gain of three selected issues with GPT-6 Cyber: five catches the baseline missed and two baseline catches lost. The additions and losses show the difference in coverage between the configurations. Both the additional detections and the lost catches matter when assessing the model’s contribution to the ensemble. We can then test where to use the model selectively, with the goal of adding useful detections while preserving the ensemble’s existing results.
Our integration work focuses on making those additional findings useful to developers. The larger report volume makes verification and prioritization especially important. Additional issues worth fixing, preservation of existing catches, and manageable reports are the criteria for that work. The benchmark provides concrete cases for that assessment.
Choosing the reasoning effort
We used high reasoning effort for investigation, where the model traces potential vulnerabilities through the code. Without a matched lower-effort run, we can’t isolate the effect of that setting.
Our next comparison will change effort at one stage at a time, using the same cases and scoring criteria. We will compare known issues retained in final reports, complete-scan time, and cost, including verification. Lower effort could be more practical if it preserves useful catches. Higher effort would earn its place where it adds findings worth investigating.
Where GPT-6 Cyber adds value
GPT-6 Cyber shows a concrete improvement for deeper security scans in CodeRabbit. In our evaluation, the ensemble with GPT-6 Cyber included 66% more high- and medium-severity findings in its final reports and caught approximately 16% more of the selected benchmark issues than the ensemble without it.
If your team’s main problem is already too many alerts, report volume alone gives you little reason to change models. The useful measure is how many additional issues deserve a fix and how much work it takes to establish that. The evaluation provides measured coverage and report-volume results, while report accuracy and assessment effort remain priorities for the integration work.
We recommend evaluating GPT-6 Cyber for security-sensitive scans, keeping the issues the existing ensemble already catches in the regression set. For developers using CodeRabbit, our integration work aims to turn that additional detection into useful findings before code ships. These results give us a concrete improvement to build on.
What CodeRabbit Secure Deep Scan is finding in practice
From October 23 to November 3, teams ran over 13,000 Deep Scans across more than 2,700 organizations.
Deep Scan identified critical and high-severity findings, with authorization bypass, IDOR, and cross-site scripting among the most common.
These are product-wide Deep Scan results. The evaluation above examines GPT-6 Cyber’s contribution to the model ensemble.
