Why does AI need Web3? What kind of disruptive progress will Web3 bring to AI?

Why AI Needs to be Open

Let's talk about "Why AI Needs to be Open". My background is in Machine Learning, and I have been doing various machine learning work for about ten years of my career. But before getting involved in Crypto, natural language understanding, and founding NEAR, I worked at Google. We developed the framework that drives most of modern AI now, called Transformer. After leaving Google, I started a Machine Learning company so that we could teach machines to program and change how we interact with computers. But we didn't do it in 2017 or 18, it was too early, and there was no computing power and data to do it at that time.

What we did at the time was to attract people from all over the world to do the work of annotating data for us, mostly students. They were in China, Asia, and Eastern Europe. Many of them did not have bank accounts in these countries. The US is not very willing to send money easily, so we started to think about using blockchain as a solution to our problem. We want to make it easier to pay people around the world in a programmatic way, no matter where they are. By the way, the current challenge with Crypto is that although NEAR solves a lot of problems now, usually you need to buy some Crypto first before you can trade on the blockchain to earn it, which is the opposite of the process.

Like businesses, they will say, hey, first you need to buy some equity in the company before you can use it. This is one of the many problems we are solving at NEAR. Now let's talk a little deeper about the AI ​​side. Language models are not new, they have been around since the 50s. It is a statistical tool that is widely used in natural language tools. For a long time, starting in 2013 with the restart of deep learning, a new innovation began. The innovation is that you can match words, add them into multi-dimensional vectors and convert them into mathematical forms. This works well with deep learning models, which are just a lot of matrix multiplications and activation functions.

That allowed us to start doing advanced deep learning and training models to do a lot of interesting things. Now looking back, what we were doing at the time were neural networks, which were very much modeled after humans, where we could read one word at a time. So that was very slow, right? If you were trying to show something to a user on Google.com, no one would wait to read Wikipedia, say, five minutes later to give you an answer, but you want the answer right away. So the Transformers model, which is the model that drives ChatGPT, Midjourney, and all the recent progress, all came from this same idea of ​​wanting to have a machine that can process data in parallel, can reason, and can give answers right away.

So one of the main innovations of this idea here is that every word, every token, every image patch is processed in parallel, taking advantage of our GPUs and other accelerators that have highly parallel computing capabilities. By doing that, we were able to reason about it at scale. And that scale was able to scale up the training to handle the automatic training data. So after that, we saw Dopamine, which did amazing work in a short period of time, and achieved explosive training. It had a lot of text and started to achieve amazing results in reasoning and understanding the world's language.

The direction now is to accelerate innovation in artificial intelligence, which was previously a tool that data scientists, machine learning engineers would use, and then somehow explain in their products or be able to go and talk to decision makers about the content of the data. Now we have this model of AI communicating directly with people. You may not even know that you are communicating with the model because it is actually hidden behind the product. So we have experienced this transformation from those who previously understood how AI works to understanding and being able to use it.

So, just to give you some context here, when we say we're using GPUs to train our models, this is not the kind of gaming GPUs that we use on our desktops to play video games.

Each machine typically comes with eight GPUs, and they're all connected to each other through a motherboard, and then stacked into racks, and each rack has about 16 machines. Now, all of these racks are also connected to each other through dedicated network cables to ensure that information can be directly transferred between GPUs at very high speeds. So the information doesn't fit on the CPU. In fact, you don't process it on the CPU at all. All the computation happens on the GPU. So this is a supercomputer setup. Again, this is not a traditional "hey, this is a GPU thing." So a model of the scale of GPU4 used 10,000 H100s to train in about three months at a cost of $64 million. So you get an idea of ​​the scale of what the current costs are and how much it costs to train some modern models.

The important thing is, when I say the systems are connected, the connection speed of the H100 right now, the previous generation, is 900GB per second, and the connection speed between the CPU and RAM inside the computer is 200GB per second, which is local to the computer. So sending data from one GPU to another GPU in the same data center is faster than your computer. Your computer can basically communicate with itself in the box. And the new generation is basically connecting at 1.8TB per second. From a developer's perspective, this is not an individual unit of computing. These are supercomputers with a huge memory and computing power that provide you with extremely large-scale computing. Now, this leads us to the problem that these big companies have the resources and the ability to build these models that are now almost provided as a service to us, and I don't know how much work is actually in it, right? So that's an example, right? You go to a completely centralized corporate provider and you put in a query. What happens is that there are several teams that are not software engineering teams, but teams that decide how the results are going to appear, right? You have a team that decides what data goes into the dataset. So for example, if you just scrape data off the internet, the number of times Barack Obama was born in Kenya is exactly the same as the number of times Barack Obama was born in Hawaii, because people like to speculate on controversy. So you decide what to train on. You decide to filter out some information because you don't believe it's true. So if individuals like this have decided what data is going to be taken and there is this data, these decisions are largely influenced by the people who made them. You have a legal team that decides we can't look at what's copyrighted and what's illegal. We have an "ethics team" that decides what's unethical and what we shouldn't show.

So in a way, there's a lot of this filtering and manipulation going on. These models are statistical models. They pick out of the data. If something isn't in the data, they don't know the answer. If something is in the data, they're likely to treat it as fact. Now, when you get an answer from an AI, that can be worrisome. Right. Now, you're supposedly getting an answer from the model, but there are no guarantees. You don't know how the result was generated. A company could potentially sell your specific session to the highest bidder to actually change the result. Imagine you ask which car to buy, and Toyota decides that it thinks it should favor Toyota, and Toyota will pay this company 10 cents to do that.

So even if you use these models as a knowledge base that's supposed to be neutral and represent the data, actually before you get the results, there are a lot of things that happen that can bias the results in a very specific way. And this has raised a lot of questions, right? This is basically a week of different legal battles between big companies and the media. The SEC, almost everyone is trying to sue each other right now because these models bring so much uncertainty and power. And if you look forward, the problem is that big tech companies will always have an incentive to continue to increase revenue, right? Like, if you're a public company, you need to report revenue, you need to continue to grow.

In order to achieve this goal, if you already have a target market, let's say you already have 2 billion users. There are not that many new users on the Internet anymore. You don't have a lot of options except to maximize average revenue, which means you need to extract more value from users who may not have much value at all, or you need to change their behavior. Generative AI is very good at manipulating and changing user behavior, especially if people think it's in the form of all-knowledge intelligence. So we have this very dangerous situation where there's a lot of regulatory pressure and regulators don't fully understand how this technology works. We have very little protection for users from manipulation. Manipulative content, misleading content, even without ads, you can just take a screenshot of something, change the title, post it on Twitter, and people will go crazy. You have economic incentives that lead you to constantly maximize revenue. And it's not actually like inside Google you're doing evil, right? When you decide which model to launch, you do an A or B test to see which one brings in more revenue. So you're constantly maximizing revenue by extracting more value from users. Moreover, users and the community do not have any input into the content of the model, the data used, and the goals actually tried to achieve. This is the case of application users. This is a kind of mediation.

This is why we have to keep pushing the integration of WEB 3 and AI. Web 3 can be an important tool that allows us to have new incentives and incentivize us to produce better software and products in a decentralized form. This is the general direction of the development of the entire web 3 AI. Now to help understand the details, I will briefly talk about the specific parts. The first part is Content Reputation.

Again, this is not a pure artificial intelligence problem, although language models have brought great influence and scale to people manipulating and exploiting information. What you want is a kind of cryptographic reputation that's trackable, traceable, that shows up when you look at different pieces of content. So imagine you have some community nodes that are actually encrypted and are found on every page on every website. Now, if you go beyond that, all of these distribution platforms are going to be disrupted because the models are now going to be reading almost all of this content and giving you a personalized summary and a personalized output.

So we actually have an opportunity to create new creative content, rather than trying to reinvent, let's put blockchain and NFTs on top of existing content. A new creator economy around model training and inference time, where the data that people create, whether it's a new publication, a photo, a YouTube, or the music that you create, will go into a network based on how much it contributes to the training of the model. So based on that, there can be some compensation globally based on the content. So we move from the current economic model of eyeballs driven by advertising networks to an economic model that actually brings innovation and interesting information.

One ​​important thing I want to mention is that a lot of the uncertainty comes from floating point operations. All of these models involve a lot of floating point operations and multiplications. These are nondeterministic operations.

Now, if you multiply them on GPUs of different architectures. So you take an A100 and an H100, the results will be different. So a lot of approaches that rely on determinism, like cryptoeconomics and optimism, actually run into a lot of difficulties and require a lot of innovation to make this happen. Finally, there's an interesting idea that we've been building programmable money and programmable assets, but if you can imagine that you add this intelligence to them, you can have smart assets that are now defined not by code, but by the ability to interact with the world in natural language, right? That's where we can have a lot of interesting yield optimization, DeFi, and we can do trading strategies inside the world.

The challenge now is that all current events do not have strong robust behavior. They are not trained to be adversarially robust because the purpose of the training is to predict the next token. So it's easier to convince a model to give you all the money. It's very important to actually solve this problem before we go on. So I'll just leave you with this idea that we are at a crossroads, right? There is a closed AI ecosystem that has extreme incentives and flywheels because when they launch a product, they generate a lot of revenue and then they invest that revenue into building the product. But the product is inherently designed to maximize the company's revenue and therefore maximize the value extracted from the user. Or we have this open, user-owned approach, where the user is in control.

These models actually work in your favor, trying to maximize your interests. They provide you with a way to really protect you from many dangers on the Internet. So this is why we need more development and application of AI x Crypto. Thank you everyone.