New ChatGPT-4 instantly identified Ethereum smart contract flaws

The popularity of OpenAI’s text-based artificial intelligence (AI) platform ChatGPT has reinvigorated the public’s interest in the underlying technology due to its utility in multiple areas, and one cryptocurrency exchange executive has decided to test its abilities in finding flaws in Ethereum (ETH) smart contract.

Specifically, Conor Grogan, director of crypto trading platform Coinbase, has dumped a live Ethereum contract into the newest version of the popular chatbot, GPT-4, and it highlighted multiple security vulnerabilities and surface areas where the smart contract could be exploited, as he said in a Twitter post on March 14.

Additionally, Grogan posted screenshots of the AI bot’s analysis, which indeed seem to show that ChatGPT is capable of correctly identifying critical issues and vulnerabilities, as it concluded that the analyzed smart contract “should not be used, as it contains critical vulnerabilities and is based on an illegal scheme.”

I dumped a live Ethereum contract into GPT-4.

In an instant, it highlighted a number of security vulnerabilities and pointed out surface areas where the contract could be exploited. It then verified a specific way I could exploit the contract pic.twitter.com/its5puakUW
— Conor (@jconorgrogan) March 14, 2023

Regurgitating available data?

That being said, some disagreement arose in the comments regarding whether the new version of the AI tool was able to discover these smart contract vulnerabilities on its own or was just highlighting old information about it already available online.

Indeed, Grogan did specify that the contract in question was hacked in 2018 using the flaws that the AI tool was pointing out, which led several commenters to state it was simply listing the issues that had already been made public before its training data cutoff in September 2021, and that it might not be as accurate with an unseen smart contract that was never exploited before.

Just bear in mind that GPT-4 is aware of this exact contract, that it was hacked in 2018 and how.

It might not be so accurate if you give it a smart contract that it's never seen before, or has never had the exploit shared publicly before.
— TwoBags ⛩ (@ItsTwoBags) March 14, 2023

Regardless of whether ChatGPT was able to dig out the vulnerabilities in the smart contract on its own or was just regurgitating the information already available online, its capabilities are still significant and potentially useful in auditing smart contracts, as well as in other areas in the cryptocurrency sector, such as in making educated guesses about the future price of cryptocurrencies like Polygon (MATIC).

That said, some critics, including Tesla (NASDAQ: TSLA) CEO Elon Musk, have expressed their views that ChatGPT may be biased in discussing certain topics that are considered controversial, which has allegedly led Musk to start contemplating the possibility of creating a ChatGPT alternative as he joked about the need for “TruthGPT.”