ChatGPT was asked to do smart contract audit; Here’s how it went

The recent launch of the innovative text-based artificial intelligence (AI) project ChatGPT has reinvigorated the public’s interest in the technology as it demonstrated utility across multiple fields, leading one company to test its capabilities as a smart contract auditor.

Indeed, decentralized betting platform ZKasino has recently engaged in a pre-audit of its smart contract code with ChatGPT while undergoing a comprehensive audit by CertiK, the blockchain security company said in its blog post published on February 13.

According to CertiK’s analysis, the AI tool did manage to raise “several concerns that sounded valid on the surface,” demonstrating its capability to provide “valuable service to the Web3 security community,” but that there was still “quite a lot of room for improvement.”

AI’s blind spots

Specifically, ChatGPT failed to pinpoint certain serious security issues, including project-specific logic vulnerabilities, inaccurate math calculations and statistical models, and inconsistencies between implementation and design intention – in addition to reporting false positives for code that didn’t show any problems upon CertiK’s manual audit.

CertiK’s analysis of ChatGPT’s findings. Source: CertiK

All things considered, AI still seems a long way from being relied upon as the sole auditor of smart contract code due to its limitations in “fully understanding the complexities and nuances of code, as well as its lack of hands-on experience in real-world scenarios.”

These are the reasons why “it is important to supplement ChatGPT’s analysis with manual audits by experienced security experts to ensure accuracy,” the blockchain security platform stressed, highlighting the strengths and weaknesses of ChatGPT versus professional human auditors on various criteria.

ChatGPT vs. human professional in smart code auditing. Source: CertiK

Although by itself, it is still far from perfect, the AI bot can be useful in the cryptocurrency industry, as it is able to provide insights into specific cryptocurrencies, such as the possible price range of the XRP token in 2030, as well as to clarify concepts in an interactive and conversational way to aid crypto adoption.