Web3 security expert challenges AI for audit competition: Here’s the result

An audit challenge between Nirlin and Bunzz Audit highlighted the strengths and weaknesses of manual audits versus AI-assisted audits.

Web3 security auditing platforms ensure the integrity of smart contracts — self-executing digital agreements — by identifying vulnerabilities and mitigating potential risks. Some audit platforms utilize artificial intelligence (AI) to enhance their auditing processes, significantly increasing the speed and breadth of audits. However, the reliance on AI raises concerns about potentially missing subtle, critical vulnerabilities that a human auditor might detect.

This balance between AI efficiency and human expertise was put to the test in an audit challenge on X, where Nirlin, a prominent Web3 security expert, competed against Bunzz Audit, an AI-assisted auditing platform. The “Audit Challenge” kept everyone on their toes, raising questions about the future of AI in smart contract security.

Human auditor challenges AI-assisted audit firm

It all began with a tweet. Bunzz Audit announced it had launched an AI-assisted audit service designed to speed up and expand vulnerability checks on smart contracts.

📣📣The AI-based audit, Bunzz Audit has officially launched today.📣📣

LP: https://t.co/r81rcSAcZX
Press Release: https://t.co/9SUbcAM6ZH

1. What is Bunzz Audit?
It's an AI-based smart contract audit service. It compares the contracts under audit with Bunzz's uniquely… pic.twitter.com/BMM67TnSch

— Bunzz | Audit & Smart Contract Hub (@BunzzDev) April 2, 2024

Nirlin, a smart contract auditor, reacted to the tweet and expressed skepticism regarding the effectiveness of AI in smart contract audits.

Source: X

Proposing a head-to-head audit contest, Nirlin publicly challenged Bunzz Audit, sparking a viral debate on X (formerly Twitter).

Source: X

Bunzz Audit accepted the challenge.

Source: X

The challenge attracted the attention of 0xDjango, a judge from the developer competition platform Code4rena, who agreed to be the judge for this challenge.

Source: X

The smart contract that was audited

Nirlin chose the smart contract for the audit contest. As part of their regular audit duties, Nirlin had audited these contracts beforehand. Bunzz Audit, on the other hand, was given two hours to complete its analysis. Despite this disparity, the results provided valuable insights.

A clear distinction has emerged in the results:

• Bunzz Audit identified 43 vulnerabilities, demonstrating an ability to scan for a wide range of potential issues.
• Nirlin’s meticulous manual audit uncovered critical vulnerabilities that could cause significant trouble to the smart contract, which Bunzz Audit missed. From a risk-mitigation standpoint, Nirlin’s report provided more actionable insights.

The takeaway here is there’s no such thing as a one-size-fits-all audit report, Bunzz commented, adding that choosing the right service depends on specific needs.

For those prioritizing a wide-net vulnerability scan, Bunzz Audit might be the optimal choice. However, a human auditor like Nirlin could be invaluable if the focus is on pinpointing critical risks.

Following the contest, Bunzz Audit took pointers and invested in further research and development, leading to a significant update to its AI engine.

Enhancing detection and learning capabilities

Shortly after the competition, Bunzz Audit announced on X that it had successfully identified the vulnerabilities missed during the challenge.

Weeks ago, @0xnirlin challenged our AI’s ability to find critical smart contract vulnerabilities. He won that challenge, and it pushed us to improve!

Since then, Bunzz Audit has been enhanced, allowing us to detect the vulnerabilities missed in the challenge.

This demonstrates…

— Bunzz | Audit & Smart Contract Hub (@BunzzDev) May 24, 2024

The revelation came on the heels of a minor update to their AI engine, showcasing the learning capabilities of Bunzz Audit’s technology. Although Nirlin won the first contest, Bunzz Audit’s rapid progress indicates that its approach may hold significant promise. After all, AI can learn at an accelerated pace compared to human auditors.

Processing speed triples with GPT-4o

Further bolstering the case for AI-assisted audits, Bunzz Audit revealed the use of OpenAI’s GPT-4o language model, an advanced AI capable of analyzing complex data patterns and providing detailed insights with high accuracy. The development resulted in a threefold increase in processing speed.

Unlike human auditors, AI firms benefit from the continuous advancements of OpenAI technology at no additional cost. Experts predict a significant leap in vulnerability detection with the release of GPT-5, an advanced model set to enhance data analysis capabilities.

The future of smart contract auditing

The Audit Challenge served as more than just a contest; it displayed a captivating spectacle for many on X.

Source: X

Fueled by a healthy competitive spirit, Bunzz Audit commented:

“We aim to present a new auditing service enabled by AI-assisted auditing, beyond the conventional firm model. We are confident that if this “Bunzz Audit V2” concept materializes, it will be a game-changer in the audit industry.”

Bunzz Audit plans to introduce innovative security services that leverage the platform’s freedom from the limitations of human auditing.

The future of Web3 security appears promising, with a potential collaboration between human expertise and the ever-evolving power of artificial intelligence. This case study provides a valuable glimpse into the landscape, highlighting both approaches’ strengths and limitations. As AI progresses, it will be fascinating to witness how this dynamic unfolds, ultimately shaping the future of Web3 security.