How NetSPI brings code quality back in line with AI-driven throughput with CodeRabbit

With CodeRabbit, NetSPI restored its earlier quality standards while sustaining the higher development throughput created by AI coding tools.

CodeRabbit estimates 1.22 weeks of reviewer time saved during the last 90 days.

NetSPI helps organizations discover, prioritize, and remediate their most consequential security vulnerabilities. Its proactive security platform combines expert-led penetration testing with purpose-built AI to deliver continuous cybersecurity testing across applications, networks, cloud environments, and AI systems. Built on more than 20 years of real-world security experience, NetSPI gives enterprises greater visibility into complex attack surfaces and helps their security teams accelerate detection, prioritization, and remediation.

For the engineers building that platform, the balance between human expertise and AI acceleration had also become an internal priority. AI coding tools enabled NetSPI’s development teams to produce code at a markedly faster rate. The company needed its quality safeguards to advance at the same speed.

NetSPI embraced AI coding tools early. The payoff appeared quickly. Engineering teams produced substantially more code, and individual teams gained the capacity to move more work through development.

The new pace also placed intense pressure on the company’s traditional review model. Senior engineers owned important parts of the codebase and upheld engineering standards through manual review, supported by linters, formatters, tests, and other established controls. As AI-generated output climbed, those senior engineers spent an increasing share of their time examining pull requests and repeatedly flagging basic mistakes.

“The volume of code that the teams were producing greatly increased,” said Andrew Graham, Vice President of Engineering at NetSPI, and the company’s senior-most technical lead. “It was a burden on our senior engineers to keep up with all the reviews at a quality that we expected.”

The effect reached beyond the hours consumed by review. Engineers worked through misconfigurations, typos, unused variables, and other familiar problems before reaching the subtle questions that demanded their experience. By then, the mental fatigue had accumulated.

NetSPI continued delivering at the pace it wanted, yet a more consequential signal began to move in the wrong direction.

“We were achieving the throughput that we thought we wanted, but with an increasing defect escape rate, it wasn’t at the quality standard that we expected,” Graham said.

AI coding tools created another challenge for less-experienced developers. Their polished, fast output could encourage a degree of trust the generated code had not earned. Junior engineers were often more willing to accept what tools such as Cursor produced, even when those tools lacked the full business history, architectural context, and operational knowledge surrounding the code.

NetSPI wanted a review layer that could keep pace with the new output, give developers immediate and contextual guidance, and preserve senior attention for the decisions that required seasoned human judgment.

NetSPI placed the evaluation in the hands of its engineering review board, a grassroots group of the company’s most senior engineers and architects. Graham described them as “hackers at heart.” They approached the pilot accordingly.

The group used CodeRabbit on routine pull requests to see what it would find in day-to-day development. It also manufactured issues ranging from common mistakes to less-familiar failure patterns. The tests included typos, unused variables, inappropriate calls, infrastructure misconfigurations, and deliberately introduced security problems in areas including Terraform and GitHub Actions.

The goal was to learn where CodeRabbit performed well, where it struggled, and whether it could recognize the patterns most likely to consume reviewer time or create risk. Its comments were contextual and reasonable, and its performance on deliberately planted issues earned the review board’s confidence.

“It did quite well and passed the initial muster to the point that it got the confidence of that team to turn things on more broadly,” Graham said.

NetSPI evaluated Cursor’s Bugbot and several security-adjacent tools as part of its broader search. CodeRabbit fit the engineering workflow directly and addressed the quality-review problem NetSPI needed to solve.

CodeRabbit became NetSPI’s first review layer, catching recurring baseline issues before senior engineers had to spend their attention on them. Human reviewers could scan the automated feedback, confirm or challenge it, and move more quickly into questions about architecture, performance, scaling, service dependencies, and the wider implications of a change.

“CodeRabbit being able to come in and catch this baseline level of quality for us has broadly spared the senior devs from needing to go back through and find these simple issues,” Graham said. “Now they can spend their brain power on really interesting, hard contextual problems.”

That division of labor matters at NetSPI. Security expertise runs deep across the organization. CodeRabbit’s role is to lift the floor of everyday code quality so those experts can devote more energy to deep-rooted security problems and architectural decisions.

For junior engineers, CodeRabbit provides a second perspective on AI-generated work. It reviews the changes in the pull request, examines surrounding code, and helps developers understand implications that may be invisible from the generated code alone.

“CodeRabbit has acted as a counterweight to that type of behavior,” Graham said. Generation and review now serve “complementary adversarial roles,” with one system producing code and the other challenging it before humans make the final decision.

CodeRabbit has earned broad adoption across NetSPI’s engineering organization with an acceptance rate of 56.5%.

The strongest category acceptance rates appeared in areas with direct operational consequences. Developers accepted 64.7% of stability and availability findings and 62.7% of data integrity and integration findings. There was also a 58.3% acceptance rate of major-severity comments.

Over a recent 90-day review period, CodeRabbit estimates 1.22 weeks of reviewer time saved. Graham sees the more important result in what NetSPI achieved with its expanded capacity. AI coding tools had initially driven defect escapes and change failures upward alongside output. With CodeRabbit supporting review, both quality measures improved and returned to levels better aligned with NetSPI’s higher throughput.

CodeRabbit also produced an unexpected human outcome. NetSPI’s engineering review board tracked human interaction on pull requests and found that CodeRabbit prompted engineers to participate more actively. Human comments increased approximately two to three times compared with the period before CodeRabbit.

“With the rollout of CodeRabbit, I have more humans interacting in pull requests than I ever had pre-CodeRabbit,” Graham said. CodeRabbit’s findings create a starting point for engineers to add context, challenge assumptions, and deepen the discussion around a change.

One incident made the value especially vivid. CodeRabbit identified a boundary-related problem, but the developers involved did not understand or act on the finding. The issue later caused an incident. During the root-cause analysis, the team found that CodeRabbit had identified the underlying problem before the code moved forward. The episode reinforced both the tool’s ability to find consequential issues and the importance of thoughtful human engagement with CodeRabbit’s feedback.

NetSPI is also using CodeRabbit’s Learnings and path-based instructions to carry engineering context from one review into the next. Over the last three months, there were 140 new Learnings created and applied approximately 4,560 times, reaching 69.6% of reviewed pull requests. Path-based instructions covered 66.3% of pull requests and were applied approximately 12,200 times.

Human review remains central to the system. Senior engineers monitor the guidance, correct incomplete interpretations, and curate Learnings as the codebase and NetSPI’s standards evolve. The approach converts knowledge that once had to be repeated manually into guidance developers can receive inside the pull request.

That location is one of CodeRabbit’s defining advantages for NetSPI. Developers remain inside GitHub, where they already work. They can read a finding, ask questions, and resolve the issue without moving into another interface.

“It just is in the PRs they’re already in,” Graham said. “It’s conversational.”

CodeRabbit has grown with NetSPI’s adoption path. The company began conservatively, letting engineers decide where and how they wanted the automated reviewer to participate. As they identified new needs, they frequently discovered that CodeRabbit already offered the relevant configuration or capability.

The result is a development process capable of sustaining the faster pace created by AI while protecting the quality NetSPI expects. Senior engineers are regaining productive time. Junior developers receive a counterweight to generated code. Human collaboration inside pull requests has grown. Defect escape and change failure rates have improved at the new level of throughput.

Graham has already recommended CodeRabbit to technology leaders facing the same collision between AI-generated volume and finite review capacity. Its GitHub-native, conversational workflow makes it useful across engineering organizations of different sizes, he said.

His assessment of the investment is direct.

“I would absolutely do this one again. No problem,” Graham said.