Dwarkesh Podcast · Andrej Karpathy — AGI is still a decade away

Today, I'm speaking with Andre Karpathy.

今天，我和安德烈·卡帕西交谈。

Andre, why do you say that this will be the decade of agents and not the year of agents?

安德烈，你为什么说这将是代理的十年，而不是代理之年？

Well, first of all, thank you for having me here.

嗯，首先，谢谢你邀请我来这里。

I'm excited to be here.

我很高兴能来这里。

So the quote that you've just mentioned, it's the decade of agents.

所以你刚才提到的这句话，是代理的十年。

That's actually a reaction to an existing, pre-existing quote, I should say,

这实际上是对一个已有的、先前存在的说法的回应，我应该这么说，

where I think a lot of some of the labs, I'm not actually sure who said this,

我认为一些实验室，其实我不确定是谁说的，

but they were alluding to this being the year of agents with respect to LLMs and how they were going to evolve.

但他们暗示这是LLM的代理之年，以及它们将如何演变。

And I think I was triggered by that

我想我是被那个触发了

because I feel like there's some over predictions going on in the industry.

因为我觉得行业里有一些过度预测。

And in my mind, this is really a lot more accurately described as the decade of agents.

在我看来，这更准确地描述为代理的十年。

And we have some very early agents that are actually like extremely impressive

我们有一些非常早期的代理，实际上非常令人印象深刻，

and that I use daily, you know, Cloud and Codex and so on.

我每天都用，你知道，Cloud和Codex等等。

But I still feel like there's so much work to be done.

但我仍然觉得还有很多工作要做。

And so I think my like my reaction is like, we'll be working with these things for a decade.

所以我觉得我的反应是，我们会和这些东西一起工作十年。

They're going to get better and it's going to be wonderful.

它们会变得更好，这会很棒。

but I think I was just reacting to the timelines, I suppose, of the implication.

但我认为我只是对时间线做出反应，我想，是对其含义的反应。

And what do you think will take a decade to accomplish?

你觉得什么需要十年才能完成？

What are the bottlenecks?

瓶颈是什么？

Well, actually make it work.

嗯，实际上是让它运作起来。

So in my mind, I mean, when you're talking about an agent, I guess,

所以在我看来，我的意思是，当你谈论一个智能体时，我猜，

or what the labs have in mind and what maybe I have in mind as well,

或者实验室们所设想的，以及也许我所设想的，

is you should think of it almost like an employee or like an intern that you would hire to work with you.

是你应该把它几乎看作一个员工，或者像一个你会雇来和你一起工作的实习生。

So for example, you work with some employees here.

所以举个例子，你在这里和一些员工一起工作。

When would you prefer to have an agent like Cloud or Codex do that work?

你希望什么时候让像Cloud或Codex这样的智能体来做那项工作？

Like currently, of course, they can't.

比如目前，当然，它们做不到。

What would it take for them to be able to do that?

它们需要什么才能做到那一点？

Why don't you do it today?

为什么你今天不做呢？

And the reason you don't do it today is because they just don't work.

而你今天不做的原因是因为它们就是不管用。

So like they don't have enough intelligence.

所以就像它们没有足够的智能。

They're not multimodal enough.

它们不够多模态。

They can't do computer use and all this kind of stuff.

它们不能做电脑操作以及所有这类事情。

And they don't do a lot of the things that you've alluded to earlier.

而且它们做不到很多你之前提到的事情。

You know, they don't have continual learning.

你知道，它们没有持续学习。

You can't just tell them something and they'll remember it.

你不能只是告诉它们一些事情，它们就会记住。

And they're just cognitively lacking and it's just not working.

而且它们只是认知上有所欠缺，就是不管用。

And I just think that it will take about a decade to work through all those issues.

而我只是觉得，解决所有这些问题大约需要十年。

Interesting. So as a professional podcaster and a viewer of AI from afar,

有意思。所以作为一个专业播客和远距离观察AI的观众，

it's easy to identify for me like, oh, here's what's lacking.

对我来说很容易识别出，哦，这里缺的是什么。

Continual learning is lacking or multimodality is lacking.

持续学习有所欠缺，或者说多模态能力有所欠缺。

But I don't really have a good way of trying to put a timeline on it.

但我真的没有一个好办法来给它定一个时间线。

Like if somebody's like, how long will continual learning take?

比如如果有人问，持续学习需要多长时间？

There's no like prior I have about like this is a project that should take five years, 10 years, 50 years.

我并没有一个先验判断，说这是一个应该花五年、十年还是五十年的项目。

Why a decade? Why not one year?

为什么是十年？为什么不是一年？

Why not 50 years?

为什么不是五十年？

Yeah, I guess this is where you get into like a bit of, I guess, my own intuition a little bit.

是的，我想这就是你开始有点进入，我猜，我自己直觉的部分。

And also just kind of doing a bit of an extrapolation with respect to my own experience in the field, right?

同时也只是根据我自己在这个领域的经验做一点外推，对吧？

So I guess I've been in AI for almost two decades.

所以我想我在人工智能领域已经快二十年了。

I mean, it's going to be maybe 15 years or so, not that long.

我是说，大概会是十五年左右，没那么长。

You had Richard Sutton here who was around, of course, for much longer.

这里有理查德·萨顿，当然，他待的时间要长得多。

But I do have about 15 years of experience of people making predictions of seeing how they actually turned out.

但我确实有大约十五年的经验，看人们做出预测，然后看它们实际结果如何。

And also I was in the industry for a while and I was in research and I've worked in the industry for a while.

而且我也在工业界待过一段时间，我做过研究，也在工业界工作过一段时间。

So I guess I kind of have just a general intuition that I have left from that.

所以我想我只是从这些经历中留下了一种大致的直觉。

And I feel like the problems are tractable.

而且我觉得这些问题是可解决的。

They're surmountable.

它们是可以克服的。

But they're still difficult.

但它们仍然很困难。

And if I just average it out, it just kind of feels like a ticket, I guess, to me.

如果我把它平均一下，对我来说，感觉就像一张入场券吧。

This is actually quite interesting.

这其实挺有意思的。

I want to, like, hear not only the history, but what people in the room felt was about to happen at various different breakthrough moments.

我想，呃，不仅听到历史，还想听到在那个房间里的人们，在各个不同的突破性时刻，觉得即将要发生什么。

What were the ways in which their feelings were either overly pessimistic or overly optimistic?

他们的感受在哪些方面要么过于悲观，要么过于乐观？

Should we just go through each of them one by one?

我们要不要就一个一个地过一遍？

Yeah, I mean, that's a giant question because, of course, you're talking about 15 years of stuff that happened.

是啊，我是说，这是个巨大的问题，因为当然，你谈的是15年间发生的事情。

I mean, AI is actually so wonderful because there have been a number of, I would say, seismic shifts that were like the entire field has sort of like suddenly looked a different way, right?

我是说，AI其实非常奇妙，因为有过一些，我会说，地震般的转变，就像是整个领域突然之间看起来不一样了，对吧？

And I guess I've maybe lived through two or three of those.

我猜我大概经历过其中两三次。

And I still think there will continue to be some because they come with some kind of like almost surprising irregularity.

而且我仍然认为还会继续有，因为它们伴随着某种几乎令人惊讶的不规律性。

Well, when my career began, of course, like when I started to work on deep learning, when I became interested in deep learning,

嗯，当我的职业生涯开始时，当然，就像当我开始研究深度学习时，当我对深度学习产生兴趣时，

this was just kind of like by chance of being right next to Jeff Hinton at the University of Toronto.

这其实只是碰巧，我就在多伦多大学的杰夫·辛顿旁边。

And Jeff Hinton, of course, is kind of like the godfather figure of AI.

而杰夫·辛顿，当然，算是AI的教父级人物。

And he was training all these neural networks and I thought it was incredible and interesting.

他在训练所有这些神经网络，我觉得这太不可思议了，很有意思。

But this was not like the main thing that everyone in AI was doing by far.

但这远不是AI领域每个人都在做的主要事情。

This was a niche little subject on the side.

这是一个小众的、边缘的小课题。

That's kind of maybe like the first like dramatic sort of seismic shift that came with the AlexNet and so on.

那大概算是第一次像戏剧性的、地震般的转变，随着AlexNet等等一起到来。

I would say like AlexNet sort of reoriented everyone and everyone started to train neural networks.

我会说，AlexNet某种程度上重新引导了所有人，大家开始训练神经网络。

But it was still like very like per task, per specific task.

但它仍然非常像是针对每个任务、每个具体任务的。

So maybe I have an image classifier or I have a neural machine translator or something like that.

所以也许我有一个图像分类器，或者我有一个神经机器翻译器，或者类似的东西。

And people became very slowly actually interested in basically kind of agents, I would say.

人们实际上非常缓慢地开始对基本上算是智能体的东西感兴趣，我会这么说。

And people started to think, okay, well, maybe we have a checkmark next to the visual cortex or something like that.

人们开始想，好吧，也许我们在视觉皮层旁边打了个勾，或者类似的东西。

But what about the other parts of the brain?

但大脑的其他部分呢？

How can we get an actual like full agent or a full entity that can actually interact in the world?

我们怎样才能得到一个真正的、完整的智能体，或者一个真正能在世界中互动的完整实体？

And I would say the Atari sort of deep reinforcement learning shift in 2013 or so was part of that early effort of agents, in my mind,

我会说，2013年左右Atari那种深度强化学习的转变，在我看来，是早期智能体努力的一部分，

because it was an attempt to try to get agents that not just perceive the world, but also take actions and interact and get rewards from environments.

因为它是一次尝试，试图让智能体不仅感知世界，还能采取行动、互动，并从环境中获得奖励。

And at the time, this was Atari games, right?

当时，这就是Atari游戏，对吧？

And I kind of feel like that was a misstep, actually.

我有点觉得那其实是一个失误。

And it was a misstep that actually even the early OpenAI

而这实际上是一个失误，甚至早期的OpenAI

that I was a part of, of course, kind of adopted.

我当时也是其中一员，当然，也多少采纳了这种做法。

Because at that time, the zeitgeist was reinforcement learning environments, games, game playing, beat games, get lots of different types of games.

因为当时，时代精神就是强化学习环境、游戏、玩游戏、打败游戏、搞很多不同类型的游戏。

And OpenAI was doing a lot of that.

而OpenAI当时也在做很多这方面的事。

So that was maybe like another like prominent part of, I would say, AI where maybe for two or three or four years, everyone was doing reinforcement learning on games.

所以那也许是AI另一个比较突出的部分，我会说，大概有两三年或四年，大家都在游戏上做强化学习。

And basically, that was a little bit of a misstep.

基本上，那算是一个小失误。

And what I was trying to do at OpenAI actually is like I was always a little bit suspicious of games as being like this thing that would actually lead to AGI

而我在OpenAI真正想做的事，其实是我一直有点怀疑游戏，觉得它真能通向AGI

because in my mind you want something like an accountant or like something that's actually interacting with the real world.

因为在我看来，你想要的是像会计师那样的东西，或者真正与现实世界互动的东西。

And I just didn't see how games kind of like add up to it.

而我就是看不出游戏怎么能累积成那样。

And so my project at OpenAI, for example, was within the scope of the Universe project on an agent that was using keyboard and mouse to operate web pages.

所以我在OpenAI的项目，比如，是在Universe项目的范围内，做一个用键盘和鼠标操作网页的智能体。

And I really wanted to have something that like interacts with, you know, the actual digital world that can do knowledge work.

而我真的很想有一个能与你所知的真实数字世界互动、能做知识工作的东西。

And it just so turns out that this was extremely early, way too early.

而事实证明，这实在是太早了，早得离谱。

So early that we shouldn't have been working on that, you know,

早到我们本不该在那上面下功夫，你知道，

because if you're just stumbling your way around and keyboard mashing and mouse clicking and trying to get rewards in these environments,

因为如果你只是跌跌撞撞地摸索，乱敲键盘、乱点鼠标，试图在这些环境里获得奖励，

your reward is too sparse and you just won't learn.

你的奖励太稀疏了，你根本学不会。

And you're going to burn a forest computing and you're never actually going to get something off the ground.

而且你会烧掉一片森林的算力，却永远无法真正让事情起步。

And so what you're missing is this power of representation in the neural network.

所以你缺少的是神经网络中这种表征的力量。

And so, for example, today people are training those computer-using agents,

所以，举个例子，今天人们在训练那些使用计算机的智能体，

but they're doing it on top of a large language model.

但他们是在大型语言模型的基础上做这件事。

And so you actually have to get the language model first.

所以你实际上必须先获得语言模型。

You have to get the representations first.

你必须先获得表征。

And you have to do that by all the pre-training and all the LLM stuff.

而你必须通过所有的预训练和所有的大语言模型工作来做到这一点。

So I kind of feel like maybe, loosely speaking, it was like people keep maybe trying to get the full thing too early a few times,

所以我有点觉得，也许，粗略地说，就像人们可能几次三番试图过早地获得完整的东西，

where people really try to go after agents too early, I would say.

人们真的试图过早地追求智能体，我会这么说。

And that was Atari and Universe, and even my own experience. And you actually have to do some things first before you sort of get to those agents.

那就是Atari和Universe，甚至我自己的经历。你实际上必须先做一些事情，然后才能达到那些智能体。

And maybe now the agents are a lot more competent,

也许现在智能体已经能干多了，

but maybe we're still missing sort of some parts of that stack.

但也许我们仍然缺少那个技术栈的某些部分。

But I would say maybe those are like the three major buckets of what people were doing.

但我会说，也许那些就是人们当时在做的三大类事情。

Training neural nets per tasks, trying to the first round of agents,

针对每个任务训练神经网络，尝试第一轮的智能体，

and then maybe the LLMs and actually seeking the representation power of the neural networks before you tack on everything else on top.

然后也许是大型语言模型，实际上是先寻求神经网络的表征能力，然后再在上面附加其他所有东西。

Interesting. Yeah, I guess if I were to steel man, the sort of a sudden perspective would be that humans actually can just take on everything at once, right?

有意思。是啊，我想如果我要为那种突然出现的观点做最强辩护，那就是人类其实可以一下子同时处理所有事情，对吧？

even animals can take on everything at once, right?

就连动物也能一下子同时处理所有事情，对吧？

Animals are maybe a better example because they don't even have the scaffold of language.

动物也许是更好的例子，因为它们甚至没有语言这个脚手架。

They just get thrown out into the world and they just have to make sense of everything without any labels.

它们只是被扔到这个世界里，就必须在没有任何标签的情况下理解一切。

And the vision for AGI then should just be something which like just looks at sensory data,

那么AGI的愿景就应该只是某种只看感官数据的东西，

looks at the computer screen, and it just like figures out what's going on from scratch.

看着电脑屏幕，然后就像从零开始弄清楚发生了什么。

I mean, if a human was put in a similar situation, that'd be trained from scratch.

我的意思是，如果把一个人放在类似的情境中，那也会是从零开始训练。

But I mean, this is like a human growing up or animal growing up.

但我的意思是，这就像人类成长或动物成长一样。

So why shouldn't that be the vision for AI rather than like this thing where we're doing millions of years of training?

那为什么这不该是AI的愿景，而不是像这种我们要进行数百万年训练的东西呢？

I think that's a really good question.

我觉得那真是个很好的问题。

And I think, I mean, so Sutton was in your podcast and I saw the podcast and I had a write-up about that podcast almost that gets into a little bit of how I see things.

而且我觉得，我是说，Sutton上过你的播客，我看了那期播客，我还写了一篇关于那期播客的文章，差不多讲了一点我是怎么看事情的。

And I kind of feel like I'm very careful to make analogies to animals because they came about by a very different optimization process.

我有点觉得，我很小心地去类比动物，因为动物是通过一个非常不同的优化过程产生的。

Animals are evolved and they actually come with a huge amount of hardware that's built in.

动物是进化而来的，它们实际上自带大量内置的硬件。

And when, for example, my example in the post was the zebra, a zebra gets born and a few minutes later it's running around and following its mother.

比如说，我文章里的例子是斑马，一只斑马出生后几分钟就能跑来跑去，跟着妈妈走。

That's an extremely complicated thing to do.

这是一件极其复杂的事情。

That's not reinforcement learning. That's something that's baked in.

那不是强化学习。那是天生就有的东西。

And evolution obviously has some way of encoding the weights of our neural nets in ATCGs.

而进化显然有某种方式，把我们的神经网络的权重编码在ATCG里。

And I have no idea how that works, but it apparently works.

我不知道那是怎么运作的，但显然它是有效的。

So I kind of feel like brains just came from a very different process.

所以我有点觉得，大脑就是来自一个非常不同的过程。

And I'm very hesitant to take inspiration from it because we're not actually running that process.

我很犹豫要不要从中汲取灵感，因为我们实际上并没有在运行那个过程。

So in my post, I kind of said, we're not actually building animals.

所以在我文章里，我有点像是在说，我们实际上不是在造动物。

We're building ghosts. or spirits or whatever people want to call it.

我们在造幽灵。或者说鬼魂，或者随便人们想怎么叫它。

Because we're not doing training by evolution.

因为我们不是通过进化来进行训练的。

We're doing training by basically imitation of humans and the data that they've put on the internet.

我们基本上是通过模仿人类以及他们放在互联网上的数据来进行训练的。

And so you end up with these like sort of ethereal spirit entities

所以最终你会得到这些有点像空灵的灵魂实体，

because they're fully digital and they're kind of like mimicking humans.

因为它们完全是数字化的，而且它们有点像在模仿人类。

And it's a different kind of intelligence.

这是一种不同的智能。

Like if you imagine a space of intelligences, we're starting off at a different point almost.

就像如果你想象一个智能的空间，我们几乎是从一个不同的起点开始的。

We're not really building animals.

我们并不是真的在造动物。

But I think it's also possible to make them a bit more animal-like over time.

但我认为随着时间推移，也有可能让它们更像动物一点。

And I think we should be doing that.

而且我认为我们应该这么做。

And so I kind of feel like, sorry, just I guess one more point is,

所以我觉得，抱歉，我想还有一点是，

I do feel like Sutton basically has a very, like his framework is like we want to build animals.

我确实觉得Sutton基本上有一个非常，就像他的框架是：我们想造动物。

And I actually think that would be wonderful.

而我真的觉得那会很棒。

If we can get that to work, that would be amazing.

如果我们能让那实现，那会非常了不起。

If there was a single like algorithm that you can just, you know, run on the internet and it learns everything,

如果有一个单一的算法，你只要，你知道，在互联网上运行它，它就能学会一切，

that would be incredible.

那会不可思议。

I almost suspect that I'm not actually sure that it exists.

我几乎怀疑，我其实不确定它是否存在。

And that's certainly actually not what animals do.

而那确实不是动物所做的事。

Because animals have this outer loop of evolution.

因为动物有进化这个外循环。

And a lot of what looks like learning is actually a lot more maturation of the brain.

而很多看起来像学习的东西，其实更多是大脑的成熟。

And I think there's actually very little reinforcement learning for animals.

而且我认为动物其实很少有强化学习。

And I think a lot of the reinforcement learning is actually more like motor tasks.

而我认为很多强化学习其实更像是运动任务。

It's not intelligence tasks.

它不是智能任务。

So I actually kind of think humans don't actually really use RL, roughly speaking, is what I would say.

所以其实我有点觉得，人类实际上并不真的使用强化学习，大致来说，这就是我要说的。

Can you repeat the last sentence?

你能重复一下最后那句话吗？

A lot of that intelligence is not motor tasks.

很多那种智能并不是运动任务。

That's what? Sorry.

那是什么？抱歉。

A lot of the reinforcement learning, in my perspective, would be things that are a lot more like motor-like.

很多强化学习，在我看来，会是更像运动类的那些事情。

like simple kind of like tasks, throwing a hoop, something like that.

就像简单的那种任务，扔圈，诸如此类。

But I don't think that humans use reinforcement learning for a lot of intelligence tasks, like problem solving and so on.

但我不认为人类在很多智能任务上使用强化学习，比如解决问题等等。

Interesting.

有意思。

That doesn't mean we shouldn't do that for research,

这并不意味着我们不应该为了研究去做那件事，

but I just feel like that's what animals do or don't.

但我就是觉得，那正是动物做或不做的。

I'm going to take a second to digest that because there's a lot of different ideas.

我要花一点时间消化一下，因为这里面有很多不同的想法。

Maybe one clarifying question I can ask to understand the perspective.

也许我可以问一个澄清性的问题来理解这个观点。

So I think you suggest that, look, evolution is doing the kind of thing that pre-training does in the sense of building something which can then understand the world.

所以我认为你是在说，你看，进化在做的那种事情，就像预训练所做的那样，是在构建某种之后能够理解世界的东西。

The difference, I guess, is that evolution has to be titrated, in the case of humans, through three gigabytes of DNA.

我想，区别在于，就人类而言，进化必须通过三十亿字节的DNA来逐步调节。

And so that's very unlike the weights of a model.

所以这和一个模型的权重非常不同。

I mean, literally, the weights of the model are a brain,

我的意思是，实际上，模型的权重就是一个大脑，

which obviously is not encoded in the sperm and the egg, or does not exist in the sperm and the egg.

而这个大脑显然并没有编码在精子和卵子里，或者说并不存在于精子和卵子中。

So it has to be grown.

所以它必须被生长出来。

And also the information for every single synapse in the brain simply cannot exist in the three gigabytes that exist in the DNA.

而且大脑中每一个突触的信息，根本不可能存在于DNA里的那三个千兆字节中。

Evolution seems closer to finding the algorithm, which then does the lifetime learning.

进化似乎更接近于找到那个算法，然后由这个算法来完成一生的学习。

Now, maybe the lifetime learning is not analogous to RL, to your point.

那么，也许一生的学习并不类似于强化学习，就像你说的那样。

Is that compatible with the thing you were saying, or would you disagree with that?

这和你之前说的观点一致吗，还是你会不同意？

I think so. I would agree with you that there's some miraculous compression going on,

我想是的。我同意你的看法，确实存在某种神奇的压缩，

because obviously the weights of the neural net are not stored in ATCGs.

因为显然神经网络的权重并不是存储在ATCG里。

There's some kind of dramatic compression, and there's some kind of learning algorithms and coding that take over and do some of the learning online.

存在某种剧烈的压缩，还有某种学习算法和编码接管并在线完成一部分学习。

So I definitely agree with you on that.

所以我绝对同意你这一点。

Basically, I would say I'm a lot more kind of like practically minded.

基本上，我会说我更偏向于务实。

I don't come at it from the perspective of like, let's build animals.

我不是从那种“让我们造动物”的角度来看待这个问题。

I come from the perspective of like, let's build useful things.

我是从“让我们造有用的东西”这个角度来看的。

So I have a hard hat on and I'm just observing that, look, we're not going to do evolution

所以我戴着安全帽，只是在观察，你看，我们不会去做进化，

because I don't know how to do that.

因为我不知道该怎么做。

But it does turn out we can build these ghost spirit-like entities by imitating internet documents.

但事实证明，我们可以通过模仿互联网文档来构建这些幽灵般、精神般的实体。

This works. And it's actually kind of like, it's a way to bring you up to something that has a lot of sort of built-in knowledge and intelligence in some way, similar to maybe what evolution has done.

这很有效。而且它实际上有点像，它是一种让你达到某种具有大量内置知识和智能的东西的方式，类似于进化所做的事情。

So that's why I kind of call pre-training this kind of like crappy evolution.

所以这就是为什么我把预训练称为这种有点像蹩脚的进化。

It's like the practically possible version with our technology and what we have available to us to get to a starting point where we can actually do things like reinforcement learning and so on.

这就像是用我们的技术和我们可用的东西所能实现的实用版本，以达到一个我们可以真正做强化学习等事情的起点。

Just to steel man the other perspective, because after doing this interview and thinking about it a bit, he has an important point here.

只是为了替另一种观点辩护，因为在做完这次采访并思考了一下之后，他在这里有一个重要的观点。

Evolution does not give us the knowledge, really, right? It gives us the algorithm to find the knowledge.

进化并没有真正给我们知识，对吧？它给我们的是找到知识的算法。

That seems different from pre-training. So if perhaps the perspective is that pre-training helps build the kind of entity which can learn better. It teaches meta-learning.

这似乎与预训练不同。所以也许观点是，预训练有助于构建那种能学得更好的实体。它教会元学习。

And therefore, it is similar to like finding an algorithm.

因此，它类似于找到一种算法。

But if it's like evolution gives us knowledge, pre-training gives us knowledge, that analogy seems to break down.

但如果像进化给我们知识，预训练给我们知识，那个类比似乎就不成立了。

So it's subtle, and I think you're right to push back on it.

所以这很微妙，我认为你对此提出质疑是对的。

But basically, the thing that pre-training is doing, so you're basically getting the next token predictor over the internet, and you're training that into a neural net.

但基本上，预训练在做的事情，就是你基本上是在互联网上得到一个下一个词元的预测器，然后你把它训练进一个神经网络里。

It's doing two things, actually, that are kind of like unrelated.

它实际上在做两件事，这两件事有点不相关。

Number one, it's picking up all this knowledge, as I call it.

第一，它在吸收所有这些知识，我这么称呼它。

Number two, it's actually becoming intelligent.

第二，它实际上正在变得智能。

By observing the algorithmic patterns in the internet, it actually kind of like boots up all these like little circuits and algorithms inside the neural net to do things like in-context learning and all this kind of stuff.

通过观察互联网中的算法模式，它实际上有点像是在神经网络内部启动所有这些小电路和算法，来做诸如上下文学习之类的事情。

And actually, you don't actually need or want the knowledge.

而实际上，你其实并不需要也不想要这些知识。

I actually think that's probably actually holding back the neural networks overall because it's actually like getting them to rely on the knowledge a little too much sometimes.

我实际上认为，这很可能实际上在整体上拖累了神经网络，因为它实际上有时会让它们有点过于依赖知识。

For example, I kind of feel like agents, one thing they're not very good at is going off the data manifold of what exists on the internet.

例如，我有点觉得，智能体不太擅长的一件事就是偏离互联网上存在的数据流形。

if they had less knowledge or less memory, actually maybe they would be better.

如果它们拥有更少的知识或更少的记忆，实际上也许它们会更好。

And so what I think we have to do kind of going forward, and this would be part of the research paradigms,

所以我认为我们今后必须做的事情，而这会成为研究范式的一部分，

is I actually think we need to start, we need to figure out ways to remove some of the knowledge and to keep what I call this cognitive core.

是我实际上认为我们需要开始，我们需要找出方法来移除一些知识，并保留我所说的这个认知核心。

It's this like intelligent entity that is kind of stripped from knowledge but contains the algorithms and contains the magic of intelligence and problem solving and the strategies of it and all this kind of stuff.

它就像是一个智能实体，被剥离了知识，但包含了算法，包含了智能和问题解决的魔力，以及它的各种策略，诸如此类的东西。

There's so much interesting stuff there.

那里有太多有趣的东西了。

Okay, so let's start with in-context learning.

好，那我们就从上下文学习开始吧。

This is an obvious point, but I think it's worth just like saying it explicitly and meditating on it.

这是一个显而易见的观点，但我觉得值得把它明确说出来，并好好琢磨一下。

The situation in which these models seem the most intelligent, in which they are like, I talk to them and I'm like, wow, there's really something on the other end that's responding to me thinking about things.

这些模型显得最智能的那种情境，就是那种，我跟它们说话，然后我会想，哇，另一端真的有某个东西在回应我思考事情。

If it like makes a mistake, it's like, oh, wait, that's actually the wrong way to think about it.

如果它犯了错，它就会像，哦，等等，其实那样想是错的。

I'm packing up.

我在收拾东西。

All that is happening in context.

所有这一切都发生在上下文中。

That's where I feel like the real intelligence you can like visibly see.

那就是我觉得你能真正明显看到真正智能的地方。

And that in context learning process is developed by gradient descent on pre-training, right?

而那个上下文学习过程是通过预训练时的梯度下降发展出来的，对吧？

Like it spontaneously meta-learns in-context learning.

就像它自发地元学习出了上下文学习。

But the in-context learning itself is not gradient descent.

但上下文学习本身并不是梯度下降。

In the same way that our lifetime intelligence as humans to be able to do things is conditioned by evolution.

就像我们人类一生中能够做事情的那种智能，是由进化所塑造的。

But our actual learning during our lifetime is like happening through some other process.

但我们在有生之年实际的学习，是通过某种其他过程发生的。

I actually don't fully agree with that, but you should continue with that.

其实我并不完全同意这一点，但你应该继续讲下去。

Okay.

好。

Actually, then I'm very curious to understand how that analogy breaks down.

其实，那我非常好奇想弄明白那个类比是怎么不成立的。

I think I'm hesitant to say that in-context learning is not doing gradient descent.

我觉得我不太愿意说上下文学习不是在做梯度下降。

Because, I mean, it's not doing explicit gradient descent, but I still think that,

因为，我的意思是，它并不是在做显式的梯度下降，但我仍然认为，

so in-context learning, basically, it's pattern completion within a token window, right?

所以上下文学习，基本上，就是在词元窗口内做模式补全，对吧？

And it just turns out that there's a huge amount of patterns on the internet.

而结果就是，互联网上有大量的模式。

And so you're right, the model kind of learns to complete the pattern.

所以你说得对，模型有点像是学会了补全这个模式。

And that's inside the weights.

而这就在权重里面。

The weights of the neural network are trying to discover patterns and complete the pattern.

神经网络的权重在试图发现模式并补全模式。

And there's some kind of an adaptation that happens inside the neural network, right?

而且神经网络内部会发生某种适应，对吧？

Which is kind of magical and just falls out from internet just because there's a lot of patterns.

这有点神奇，就只是因为有很多模式，它就从互联网中自然涌现出来了。

I will say that there have been some papers that I thought were interesting that actually look at the mechanisms behind in-context learning.

我要说，有一些论文我觉得挺有意思，它们实际上研究了上下文学习背后的机制。

And I do think it's possible that in-context learning actually runs a small gradient descent loop internally in the layers of the neural network.

而且我确实认为，上下文学习有可能实际上在神经网络各层内部运行一个小型的梯度下降循环。

And so I recall one paper in particular where they were doing linear regression, actually, using in-context learning.

所以我记得特别有一篇论文，他们实际上是在用上下文学习做线性回归。

So basically your inputs into the neural network are XY pairs, XY, XY, XY, that happen to be on the line.

所以基本上，你输入神经网络的是XY对，XY、XY、XY，它们恰好都在这条线上。

And then you do X and you expect the Y.

然后你输入X，期望得到Y。

And the neural network, when you train it in this way, actually does do linear regression.

而神经网络，当你以这种方式训练它时，实际上确实在做线性回归。

And normally when you would run linear regression, you have a small gradient descent optimizer that basically looks at x, y, looks at an error, calculates the gradient of the weights and does the update a few times.

而通常当你运行线性回归时，你有一个小的梯度下降优化器，它基本上查看x、y，查看误差，计算权重的梯度，并更新几次。

It just turns out that when they looked at the weights of that in-context learning algorithm, they actually found some analogies to gradient descent mechanics.

结果发现，当他们查看那个上下文学习算法的权重时，他们实际上发现了一些与梯度下降机制的类比。

In fact, I think even the paper was stronger because they actually hard-coded the weights of a neural network to do gradient descent through attention and all the internals of the neural network.

事实上，我认为这篇论文甚至更有力，因为他们实际上硬编码了一个神经网络的权重，通过注意力机制和神经网络的所有内部结构来进行梯度下降。

So I guess that's just my only pushback is that who knows how in-context learning works, but I actually think that it's probably doing a little bit of some kind of funky gradient descent internally, and that I think that that's possible.

所以我想我唯一的反驳是，谁知道上下文学习是如何工作的，但我实际上认为它可能在内部进行某种奇特的梯度下降，而且我认为这是可能的。

So I guess I was only pushing back on you're saying it's not doing in-context learning.

所以我想我只是在反驳你说它没有在进行上下文学习。

Who knows what it's doing, but it's probably maybe doing something similar to it, but we don't know.

谁知道它在做什么，但它可能在做类似的事情，但我们不知道。

So then it's worth thinking about, okay,

所以，这值得思考一下，好吧，

if both of them are implementing gradient descent, sorry, if in-context learning and pre-training are both implementing something like gradient descent,

如果两者都在实现梯度下降，抱歉，如果上下文学习和预训练都在实现类似梯度下降的东西，

why does it feel like in context learning,

为什么感觉上下文学习，

actually we're getting to this continual learning, real intelligence-like thing,

实际上我们正在达到这种持续学习、真正类似智能的东西，

whereas you don't get the analogous feeling just from pre-training.

而仅仅从预训练中你不会得到类似的感觉。

At least you could argue that.

至少你可以这么认为。

And so if it's the same algorithm, what could be different?

所以如果算法相同，可能有什么不同呢？

Well, one way you can think about it is

嗯，你可以这样想，

how much information does the model store per information it receives from training?

模型从训练中接收的每条信息存储了多少信息？

And if you look at pre-training, if you look at Llama 3, for example,

如果你看预训练，比如看Llama 3，

I think it's trained on 15 trillion tokens

我认为它是在15万亿个token上训练的

And if you look at the 70B model, that would be the equivalent of 0.07 bits per token

如果你看70B模型，那相当于每个token 0.07比特

in that it sees in pre-training in terms of the information in the weights of the model compared to the tokens it reads.

因为它在预训练中看到的，就模型权重中的信息而言，与它读取的token相比。

Whereas if you look at the KV cache and how it grows per additional token in in-context learning, it's like 320 kilobytes.

而如果你看KV缓存以及它在上下文学习中每增加一个token如何增长，大约是320千字节。

So that's a 35 million fold difference in how much information per token is assimilated by the model.

所以，模型每个token吸收的信息量相差3500万倍。

I wonder if that's relevant at all.

我想知道这是否有任何相关性。

I think I kind of agree.

我想我有点同意。

I mean, the way I usually put this is that anything that happens during the training of the neural network,

我的意思是，我通常这样表述：在神经网络训练期间发生的任何事情，

the knowledge is only kind of like a hazy recollection of what happened in the training time.

这些知识只是对训练期间所发生事情的一种模糊回忆。

And that's because the compression is dramatic.

那是因为压缩非常剧烈。

You're taking 15 trillion tokens and you're compressing it to just your final network of a few billion parameters.

你把15万亿个token压缩到最终只有几十亿参数的网络中。

So obviously, it's a massive amount of compression going on.

所以显然，这进行了大量的压缩。

So I kind of refer to it as like a hazy recollection of the internet documents,

所以我把它比作对互联网文档的模糊回忆，

whereas anything that happens in the context window of the neural network, you're plugging all the tokens and it's building up all this KV cache representation, is very directly accessible to the neural net.

而神经网络上下文窗口中发生的任何事情，你输入所有token，它建立起所有这些KV缓存表示，对神经网络来说是非常直接可访问的。

So I compare the KV cache and the stuff that happens at test time to more like a working memory.

所以我把KV缓存和测试时发生的事情比作更像工作记忆。

All the stuff that's in the context window is very directly accessible to the neural net.

上下文窗口中的所有内容对神经网络来说都是非常直接可访问的。

So there's always these almost surprising analogies between LLMs and humans.

所以LLM和人类之间总是有这些几乎令人惊讶的类比。

And I find them kind of surprising because we're not trying to build a human brain, of course, just directly.

我觉得它们有点令人惊讶，因为我们当然不是直接试图构建人脑。

We're just finding that this works and we're doing it.

我们只是发现这有效，然后就这么做了。

But I do think that anything that's in the weights, it's kind of like a hazy recollection of what you read a year ago.

但我确实认为，权重中的任何内容，都像是你一年前读过的内容的模糊回忆。

Anything that you give it as a context at test time is directly in the working memory.

你在测试时给它作为上下文的任何内容，都直接存在于工作记忆中。

And I think that's a very powerful analogy to think through things.

而且我觉得，这是一个非常有力的类比，可以用来思考问题。

So when you, for example, go to an LLM and you ask it about some book and what happened in it, like a Nick Lane's book or something like that,

所以，比如说，当你去问一个大语言模型某本书里发生了什么，比如尼克·莱恩的书之类的，

the LLM will often give you some stuff which is roughly correct.

大语言模型往往会给你一些大致正确的内容。

But if you give it the full chapter and ask it questions, you're going to get much better results because it's now loaded in the working memory of the model.

但如果你把整章内容给它，并向它提问，你会得到好得多的结果，因为它现在被加载到了模型的工作记忆中。

So I basically agree with your very long way of saying that I kind of agree, and that's why.

所以我基本上同意你那种非常冗长的说法，也就是我有点同意，这就是原因。

Stepping back, what is the part about human intelligence that we have most failed to replicate with these models?

退一步说，人类智能中我们最未能用这些模型复现的部分是什么？

I almost feel like just a lot of it.

我几乎觉得，其中很大一部分都是。

So maybe one way to think about it, I don't know if this is the best way, but I almost kind of feel like, again, making these analogies imperfect as they are.

所以也许思考这个问题的一种方式是，我不知道这是不是最好的方式，但我几乎有点觉得，再说一次，就让这些类比保持它们本来的不完美吧。

We've stumbled by with the transformer neural network, which is extremely powerful, very general.

我们偶然发现了Transformer神经网络，它极其强大，非常通用。

You can train transformers on audio or video or text or whatever you want, and it just learns patterns, and they're very powerful, and it works really well.

你可以在音频、视频、文本或任何你想要的数据上训练Transformer，它只是学习模式，而这些模式非常强大，效果也非常好。

That, to me, almost indicates that this is kind of like some piece of cortical tissue.

对我来说，这几乎表明这有点像某种皮层组织。

It's something like that.

差不多就是这样。

Because the cortex is famously very plastic as well.

因为皮层也以高度可塑性著称。

You can rewire, you know, parts of brains.

你知道，你可以重新连接大脑的某些部分。

And there was slightly gruesome experiments with rewiring like visual cortex to the auditory cortex and this animal like learn fine, etc.

还有一些有点残忍的实验，比如把视觉皮层重新连接到听觉皮层，然后这只动物也能学得很好，等等。

So I think that this is kind of like a cortical tissue.

所以我认为这有点像皮层组织。

I think when we're doing reasoning and planning inside the neural networks, so basically doing reasoning traces for thinking models, that's kind of like the prefrontal cortex.

我认为当我们在神经网络内部进行推理和规划时，也就是基本上为思考模型做推理轨迹，那有点像前额叶皮层。

And then I think maybe those are like little check marks,

然后我觉得也许那些就像小对勾，

but I still think there's many brain parts and nuclei that are not explored.

但我仍然认为有很多大脑部位和神经核团尚未被探索。

So maybe, for example, there's a basic ganglia doing a bit of reinforcement learning when we fine-tune the models on reinforcement learning.

所以也许，例如，当我们用强化学习微调模型时，基底神经节在做一点强化学习。

But, you know, whereas like the hippocampus, not obvious what that would be.

但是，你知道，而像海马体，那会是什么就不明显了。

Some parts are probably not important.

有些部分可能不重要。

Maybe the cerebellum is like not important to cognition, it's thought, so maybe we can skip some of it.

也许小脑对认知不重要，它是思维，所以也许我们可以跳过一些。

But I still think there's, for example, the amygdala, all the emotions and instincts.

但我仍然认为，例如，杏仁核，所有的情绪和本能。

And there's probably like a bunch of other nuclei in the brain that are very ancient that I don't think we've like really replicated.

而且大脑中可能还有一堆非常古老的神经核团，我认为我们还没有真正复制出来。

I don't actually know that we should be pursuing, you know, the building of an analog of a human brain.

我其实不知道我们是否应该追求，你知道，构建人脑的类似物。

I'm, again, an engineer mostly at heart.

我，再次说，本质上主要是个工程师。

But I still feel like maybe another way to answer the question is

但我还是觉得，也许回答这个问题的另一种方式是

you're not going to hire this thing as an intern

你不会把这个东西当作实习生来雇用

and it's missing a lot of, because it comes with a lot of these cognitive deficits

而且它缺少很多，因为它带有许多这样的认知缺陷

that we all intuitively feel when we talk to the models.

这些缺陷在我们与这些模型交谈时都能直觉地感受到。

And so it's just like not fully there yet.

所以它就像是还没有完全到位。

You can look at it as like not all the brain parts are checked off yet.

你可以把它看作大脑的各个部分还没有全部打勾完成。

This is maybe relevant to the question of thinking about how fast these issues will be solved.

这可能与思考这些问题能多快被解决有关。

So sometimes people will say about continual learning,

所以有时候人们会谈到持续学习，

look, actually, you could easily replicate this capability.

看，其实你可以很容易地复制这种能力。

Just as in-context learning emerged spontaneously as a result of pre-training,

正如上下文学习作为预训练的结果自发出现一样，

continual learning over longer horizons will emerge spontaneously

在更长时间跨度上的持续学习也会自发出现，

if the model is incentivized to recollect information over longer horizons

如果模型被激励在更长时间跨度上回忆信息，

or horizons longer than one session.

或者比一次会话更长的时间跨度。

So if there's some like outer loop RL, which has many sessions within that outer loop,

所以如果存在某种像外层循环强化学习，在那个外层循环内有很多次会话，

then like this continual learning where it uses like, it fine tunes itself or it writes to an external memory or something

那么这种持续学习，比如它自我微调，或者写入外部记忆之类的，

will just sort of like emerge spontaneously.

就会自然而然地出现。

Do you think things are there that are plausible?

你觉得有那些是合理的吗？

I just, I don't have really a prior over like, how plausible is that?

我只是，我真的没有一个先验判断，比如那有多合理？

How likely is that to happen?

那发生的可能性有多大？

I don't know that I fully resonate with that,

我不确定我完全认同这一点，

because I feel like these models, when you boot them up and they have zero tokens in the window,

因为我觉得这些模型，当你启动它们、窗口里没有任何词元时，

they're always like restarting from scratch where they were.

它们总是像从它们原来的地方从头开始。

So I don't actually know in that worldview what it looks like.

所以在那样的世界观里，我其实不知道那是什么样子。

Because, again, maybe making some analogies to humans just because I think it's roughly concrete and kind of interesting to think through.

因为，再说一次，也许做一些与人类的类比，只是因为我觉得这大致上很具体，而且思考起来挺有意思的。

I feel like when I'm awake, I'm building up a context window of stuff that's happening during the day.

我觉得当我醒着的时候，我在建立一个白天发生的事情的上下文窗口。

But I feel like when I go to sleep, something magical happens where I don't actually think that that context window stays around.

但我觉得当我睡觉时，会发生某种神奇的事情，我其实不认为那个上下文窗口会一直保留。

I think there's some process of distillation into weights of my brain.

我认为存在某种将信息蒸馏到我大脑权重中的过程。

And this happens during sleep and all this kind of stuff.

而这发生在睡眠期间以及诸如此类的时候。

We don't have an equivalent of that in large language models.

我们在大型语言模型中没有与之等价的东西。

And that's, to me, more adjacent to when you talk about continual learning and so on, as absent.

而对我来说，这更接近于你谈论持续学习等等时所说的缺失。

These models don't really have this distillation phase of taking what happened, analyzing it, obsessively thinking through it, basically doing some kind of a synthetic data generation process and distilling it back into the weights,

这些模型并没有这个蒸馏阶段：把发生的事情拿来，分析它，反复思考它，基本上就是做某种合成数据生成过程，再把它蒸馏回权重中，

and maybe having a specific neural net per person.

也许每个人都有一个特定的神经网络。

Maybe it's a LoRa. It's not a full weight neural network.

也许它是一个LoRa。它不是一个全权重的神经网络。

It's just some of the small sparse subset of the weights are changed.

只是权重中一小部分稀疏子集被改变了。

But basically, we do want to create ways of creating these individuals that have very long contacts.

但基本上，我们确实想要创造一些方法来创造这些拥有非常长联系的个体。

It's not only remaining in the context window because the context windows grow very, very long.

这不仅仅是因为上下文窗口变得非常非常长而留在上下文窗口中。

Like maybe we have some very elaborate sparse attention over it.

就像也许我们对此有一些非常精细的稀疏注意力。

But I still think that humans obviously have some process for distilling some of that knowledge into the weights.

但我仍然认为人类显然有某种过程将一些知识提炼到权重中。

We're missing it.

我们缺少它。

And I do also think that humans have some kind of a very elaborate sparse attention scheme.

而且我也确实认为人类有某种非常精细的稀疏注意力机制。

which I think we're starting to see some early hints of.

我认为我们开始看到一些早期迹象。

So DeepSeq v3.2 just came out, and I saw that they have like a sparse attention as an example.

所以DeepSeq v3.2刚刚发布，我看到他们有一个稀疏注意力的例子。

And this is one way to have very, very long context windows.

这是获得非常非常长上下文窗口的一种方式。

So I almost feel like we are redoing a lot of the cognitive tricks that evolution came up with through a very different process.

所以我几乎觉得我们正在通过一个非常不同的过程重做很多进化想出的认知技巧。

But we're, I think, can converge on a similar architecture cognitively.

但我认为我们可以在认知上收敛到类似的架构。

Interesting.

有趣。

In 10 years, do you think it'll still be something like a transformer, but with a much more modified attention and more sparse MLPs and so forth?

10年后，你认为它还会像Transformer一样，但具有更修改的注意力和更稀疏的MLP等等吗？

Well, the way I like to think about it is,

嗯，我喜欢这样想，

Okay, let's translation invariance in time, right?

好吧，让我们考虑时间平移不变性，对吧？

So 10 years ago, where were we?

那么10年前，我们在哪里？

2015, we had convolutional neural networks primarily.

2015年，我们主要使用卷积神经网络。

Residual networks just came out.

残差网络刚刚出现。

So remarkably similar, I guess, but quite a bit different still.

所以我想，非常相似，但仍然相当不同。

I mean, Transformer was not around.

我是说，Transformer 当时还不存在。

You know, all these sort of like more modern tweaks on a Transformer were not around.

你知道，所有这些对 Transformer 的更现代的改进都还不存在。

So maybe some of the things that we can bet on, I think, in 10 years by translational sort of equivariance

所以也许我们可以押注的一些事情，我认为，10年后通过平移等变性

is we're still training giant neural networks with forward, backward, pass, and update through gradient descent.

是我们仍然通过梯度下降用前向、后向、传递和更新来训练巨大的神经网络。

But maybe it looks a little bit different and it's just everything is much bigger.

但也许它看起来有点不同，只是所有东西都大得多。

Actually, recently I also went back all the way to 1989, which was kind of a fun exercise for me a few years ago

实际上，最近我还一路回溯到1989年，这对我来说是几年前一次有趣的练习

because I was reproducing Jan LeCun's 1989 convolutional network, which was the first neural network I'm aware of trained via gradient descent, like modern neural network trained gradient descent on digit recognition.

因为我当时在复现Jan LeCun 1989年的卷积网络，这是我所知的第一个通过梯度下降训练的神经网络，就像现代神经网络在数字识别上通过梯度下降训练一样。

And I was just interested in, okay, how can I modernize this?

我只是感兴趣，好吧，我怎样才能把它现代化？

How much of this is algorithms?

这其中有多少是算法？

How much of this is data?

这其中有多少是数据？

How much of this progress is compute and systems?

这其中有多少进展是计算和系统？

And I was able to very quickly like half the learning rate, just knowing my time travel by 33 years.

而我能够很快地把学习率减半，仅仅因为知道我的时间旅行了33年。

So if I time travel by algorithms to 33 years, I could adjust what Jan LeCun did in 1989, and I could basically half the learning, half the error.

所以如果我在算法上时间旅行33年，我可以调整Jan LeCun在1989年所做的，我基本上可以把学习减半，误差减半。

But to get further gains, I had to add a lot more data.

但为了获得进一步的提升，我不得不添加更多的数据。

I had to like 10x the training set.

我不得不把训练集扩大10倍。

And then I had to actually add more computational optimizations,

然后我实际上还得加入更多计算优化，

had to basically train for much longer with dropout and other regularization techniques.

基本上得用dropout和其他正则化技术训练更长时间。

And so it's almost like all these things have to improve simultaneously.

所以这几乎就像所有这些东西都必须同时改进。

So, you know, we're probably going to have a lot more data.

所以，你知道，我们可能会有多得多的数据。

We're probably going to have a lot better hardware.

我们可能会有好得多的硬件。

Probably going to have a lot better kernels and software.

可能会有好得多的内核和软件。

We're probably going to have better algorithms.

我们可能会有更好的算法。

And all of those, it's almost like no one of them is winning too much.

而所有这些，几乎就像没有哪一个因素占主导。

All of them are surprisingly equal.

它们全都出奇地均衡。

And this has kind of been the trend for a while.

而这种趋势已经持续了一段时间。

So I guess to answer maybe your question,

所以我想，也许回答你的问题，

I expect differences algorithmically to what's happening today.

我预计算法上会与今天的情况有所不同。

But I do also expect that some of the things that have stuck around for a very long time will probably still be there.

但我也确实预计，一些已经存在了很长时间的东西可能仍然会保留。

It's probably still a giant neural network trained with gradient descent.

它可能仍然是一个用梯度下降训练的大型神经网络。

That would be my guess.

这就是我的猜测。

It's surprising that all of those things together only halved half the error.

令人惊讶的是，所有这些因素加在一起只把误差减半了。

Yeah. Which is like 30 years of progress.

是的。这就像是30年的进展。

Maybe half is a lot because if you halve the error, that actually means that...

也许减半已经很多了，因为如果你把误差减半，那实际上意味着……

Half is a lot, yeah.

减半已经很多了，是的。

Yeah, okay.

是的，好的。

But I guess what was shocking to me is everything needs to improve across the board.

但我想让我震惊的是，所有方面都需要全面提升。

Yeah. Architecture, optimize the loss function, and also has improved across the board forever.

是的。架构、优化损失函数，而且所有方面一直都在改进。

So I kind of expect all those changes to be alive and well.

所以我有点预期所有这些变化都会继续存在并发展良好。

Yeah, actually, I was about to ask a very similar question about NanoChat.

是的，其实我正要问一个关于NanoChat的非常类似的问题。

Because since you just coded up recently, every single sort of step in the process of building a chatbot is like fresh in your RAM.

因为既然你最近刚编写过，构建聊天机器人的每一个步骤都像新鲜地存在你的内存里。

And I'm curious if you had similar thoughts about, like, oh, there was no one thing that was relevant to going from GPT-2 to NanoChat.

我很好奇你是否有类似的想法，比如，哦，从GPT-2到NanoChat并没有哪一件事是特别相关的。

What are sort of like surprising takeaways from the experience? Building NanoChat?

从这段经历中有什么令人惊讶的收获？构建NanoChat？

So NanoChat is a kind of a repository I released, was it yesterday or the day before? I can't remember.

所以NanoChat是我发布的一个仓库，是昨天还是前天？我记不清了。

We can see this leave liberation that went into the... Well, it's just trying to be a, it's trying to be the simplest complete repository that covers the whole pipeline end-to-end of building a ChatGPT clone.

我们可以看到这个……嗯，它只是想成为一个，它想成为最简单的完整仓库，涵盖端到端构建ChatGPT克隆的整个流程。

And so, you know, you have all of the steps, not just any individual step, which is a bunch of, I worked on all the individual steps sort of in the past and really small pieces of code that kind of show you how that's done in algorithmic sense in like simple code.

所以，你知道，你有所有的步骤，而不仅仅是某个单独的步骤，那只是一堆……我过去研究过所有单独的步骤，以及非常小的代码片段，向你展示在算法意义上如何用简单的代码完成。

But this kind of handles all the entire pipeline.

但这个基本上处理了整个流程。

I think in terms of learning, it's not so much, I don't know that I actually found something that I learned from it necessarily.

我认为就学习而言，并不是说，我不确定我是否真的从中发现了什么必然学到的东西。

I kind of already had in my mind as like how you build it.

我脑子里其实已经有了类似如何构建它的想法。

And this is just a process of mechanically building it and making it clean enough

而这只是一个机械地构建它并把它做得足够干净的过程，

and so that people can actually learn from it and that they find it useful.

这样人们才能真正从中学到东西，并且觉得它有用。

Yeah. What is the best way for somebody to learn from it?

是的。那对某人来说，从中学到东西的最好方式是什么？

Is it just like delete all the code and try to reimplement from scratch, try to add modifications to it?

是不是就像删掉所有代码，然后试着从头重新实现，试着对它做一些修改？

Yeah, I think that's a great question.

是的，我觉得这是个很好的问题。

I would probably say, so basically it's about 8,000 lines of code that takes you through the entire pipeline.

我大概会说，基本上它大约有8000行代码，带你走完整个流程。

I would probably put it on the right monitor.

我大概会把它放在右边的显示器上。

Like if you have two monitors, you put it on the right.

就像如果你有两个显示器，你就把它放在右边。

And you want to build it from scratch.

然后你想从头开始构建它。

You build it from start.

你从一开始就构建它。

You're not allowed to copy-paste.

你不允许复制粘贴。

You're allowed to reference.

你可以参考。

You're not allowed to copy-paste.

你不允许复制粘贴。

Maybe that's how I would do it.

也许那就是我会用的方法。

I also think the repository by itself, it is like a pretty large beast.

我还觉得这个仓库本身就像一头相当大的野兽。

I mean, when you write this code, you don't go from top to bottom.

我的意思是，当你写这些代码时，你不是从上到下写的。

You go from chunks and you grow the chunks.

你是从一块一块开始的，然后把这些块逐渐扩展。

And that information is absent.

而那个信息是缺失的。

Like you wouldn't know where to start.

就像你不知道该从哪里开始。

And so I think it's not just a final repository that's needed.

所以我觉得需要的不仅仅是一个最终的仓库。

It's like the building of the repository, which is a complicated chunk growing process.

而是仓库的构建过程，那是一个复杂的、不断扩展的过程。

So that part is not there yet.

所以那部分还没有。

I would love to actually like add that probably later this week or something in some way.

我很想真的把这部分加进去，大概这周晚些时候或者什么时候，以某种方式。

Like either it's probably a video or something like that.

比如，它可能是一个视频或者类似的东西。

But maybe, roughly speaking, that's what I would try to do is build the stuff yourself,

但也许，大致来说，我会尝试做的就是自己动手构建这些东西，

but don't allow yourself copy-paste.

但不要让自己直接复制粘贴。

I do think that there's two types of knowledge almost.

我确实认为几乎有两种类型的知识。

Like there's the high-level surface knowledge,

比如有那种高层次的表面知识，

but the thing is that when you actually build something from scratch,

但问题是，当你真正从零开始构建某个东西时，

you're forced to come to terms with what you don't actually understand

你被迫去面对你实际上并不理解的东西，

and you don't know that you don't understand it.

而且你并不知道自己其实并不理解它。

Interesting.

有意思。

And it always leads to a deeper understanding.

而这总是会带来更深的理解。

And it's like just the only way to build is like,

就好像，构建的唯一方式就是，

if I can't build it, I don't understand it.

如果我不能构建它，我就不理解它。

Is that a Feynman quote, I believe, or something along those lines?

我相信那是费曼的一句话，或者类似的意思吧？

I 100% I've always believed this very strongly

我百分之百，我一直非常坚信这一点，

because there's all these like micro things

因为存在所有这些微小的东西，

that are just not properly arranged and you don't really have the knowledge.

它们只是没有被正确地组织起来，而你其实并没有真正掌握这些知识。

You just think you have the knowledge.

你只是以为自己掌握了这些知识。

So don't write blog posts, don't do slides, don't do any of that.

所以不要写博客文章，不要做幻灯片，不要做那些。

Like build a code, arrange it, get it to work.

比如写代码，把它组织好，让它运行起来。

It's the only way to go.

这是唯一的办法。

Otherwise you're missing knowledge.

否则你就会缺失知识。

You tweeted out that coding models were actually of very little help to you in assembling this repository.

你发推说，编程模型在你组装这个代码库时其实帮不上什么忙。

And I'm curious why that was.

我很好奇那是为什么。

Yeah.

是的。

So the repository, I guess I built it over a period of a bit more than a month.

所以这个代码库，我想我是用了一个多月的时间构建的。

And I would say there's like three major classes of how people interact with code right now.

我想说，目前人们与代码交互的方式大致有三类。

Some people completely reject all of LLMs and they are just writing by scratch.

有些人完全拒绝所有大语言模型，他们就是从头开始写。

I think this is probably not the right thing to do anymore.

我觉得现在这样做可能已经不对了。

The intermediate part, which is where I am, is you still write a lot of things from scratch, but you use the autocomplete that's basically available now from these models.

中间那部分，也就是我所在的阶段，是你仍然从头写很多东西，但你会用现在这些模型基本都能提供的自动补全。

So when you start writing out a little piece of it, it will autocomplete from you and you can just tap through and most of the time it's correct.

所以当你开始写出一小段时，它会根据你写的内容自动补全，你只要一路点过去，大多数时候它都是对的。

Sometimes it's not and you edit it.

有时候它不对，你就改一下。

But you're still very much the architect of what you're writing.

但你仍然在很大程度上是你所写内容的架构师。

And then there's the, you know, vibe coding.

然后还有那种，你知道的，氛围编程。

You know, hi, please implement this or that, you know, enter, and then let the model do it.

你知道，嗨，请实现这个或那个，你知道，回车，然后让模型去做。

And that's the agents.

那就是智能体。

I do feel like the agents work in very specific settings, and I would use them in specific settings.

我确实觉得智能体在非常特定的场景下能起作用，而我会在特定的场景下使用它们。

But again, these are all tools available to you, and you have to, like, learn what they're good at and what they're not good at and when to use them.

但话说回来，这些都是你可以使用的工具，你必须去了解它们擅长什么、不擅长什么，以及什么时候该用它们。

So the agents are actually pretty good, for example, if you're doing boilerplate stuff.

所以智能体其实相当不错，比如说，如果你在做样板代码之类的东西。

Boilerplate code that's like just copy-paste stuff.

就是那种直接复制粘贴的样板代码。

They're very good at that.

它们非常擅长这个。

They're very good at stuff that occurs very often on the internet.

它们非常擅长处理互联网上经常出现的东西。

because there's lots of examples of it in the training sets of these models.

因为这些模型的训练集里有很多这样的例子。

So there's like features of things where the models will do very well.

所以有些特征方面，模型会做得非常好。

I would say NanoChat is not an example of this because it's a fairly unique repository.

我想说NanoChat并不是这样的例子，因为它是一个相当独特的代码库。

There's not that much code, I think, in the way that I've structured it.

我觉得，按照我组织它的方式，代码量并不算多。

And it's not boilerplate code.

而且它不是样板代码。

It's like actually like intellectually intense code almost.

它几乎像是那种真正需要智力投入的代码。

And everything has to be very precisely arranged.

而且所有东西都必须非常精确地安排。

And the models were always trying to, they kept trying to, I mean, they have so many cognitive deficits, right?

而这些模型总是试图，它们一直试图，我是说，它们有那么多认知缺陷，对吧？

So one example, they keep trying to, they keep misunderstanding the code because

举个例子，它们总是试图，它们总是误解代码，因为

they have too much memory from all the typical ways of doing things on the internet that I just wasn't adopting.

它们从互联网上那些我根本没有采用的典型做法中获得了太多记忆。

So the models, for example, I mean, I don't know if I want to get into the full details,

所以这些模型，比如说，我是说，我不知道要不要讲全部细节，

but they keep thinking I'm writing normal code and I'm not.

但它们一直以为我在写普通代码，而我不是。

Maybe one example. Maybe one example is, so the way to synchronize,

也许举个例子。也许一个例子就是，同步的方式，

so we have eight GPUs that are all doing forward backwards.

所以我们有八块GPU，全都在做前向和反向传播。

The way to synchronize gradients between them is to use a distributed data parallel container of PyTorch.

在它们之间同步梯度的方法是使用PyTorch的分布式数据并行容器。

which automatically does all the, as you're doing the backward, it will start communicating and synchronizing gradients.

它会自动完成所有那些，当你在做反向传播时，它就会开始通信并同步梯度。

I didn't use DDP because I didn't want to use it because it's not necessary.

我没有用DDP，因为我不想用它，因为没必要。

So I threw it out.

所以我把它扔掉了。

And I basically wrote my own synchronization routine that's inside the step of the optimizer.

我基本上自己写了一个同步例程，就在优化器的step里面。

And so the models were trying to get me to use the DDP container.

所以那些模型一直想让我用DDP容器。

And they were very concerned about,

它们非常担心，

okay, this gets way too technical, but I wasn't using that container because I don't need it and I have a custom implementation of something like it.

好吧，这变得太技术了，但我没用那个容器，因为我不需要它，而且我有一个类似东西的自定义实现。

And they just couldn't internalize it.

它们就是无法理解这一点。

You had your own.

你有你自己的。

Yeah, they couldn't get past that.

是的，它们就是过不了这一关。

And then they kept trying to like mess up the style.

然后它们一直试图搞乱风格。

Like they're way too over defensive.

它们太过于防御了。

They make all these try catch statements.

它们写了一大堆try catch语句。

They keep trying to make a production code base.

它们一直想做成生产级的代码库。

And I have a bunch of assumptions in my code and it's okay.

而我的代码里有一堆假设，这没关系。

And it's just like, I don't need all this extra stuff in there.

就像，我不需要里面所有这些多余的东西。

And so I just kind of feel like they're bloating the code base.

所以我就觉得它们在让代码库变得臃肿。

They're bloating the complexity.

它们在让复杂度膨胀。

They keep misunderstanding.

它们一直误解。

They're using deprecated APIs a bunch of times.

它们多次使用已弃用的API。

So it's total mess.

所以完全是一团糟。

And it's just not net useful.

而且它净收益并不大。

I can go in, I can clean it up, but it's not that useful.

我可以进去清理一下，但它没那么有用。

I also feel like it's kind of annoying to have to type out what I want in English

我还觉得，必须用英语打出我想要的东西有点烦人，

because it's just too much typing.

因为打字太多了。

Like if I just navigate to the part of the code that I want

就像如果我只是导航到代码中我想要的那部分

and I go where I know the code has to appear

然后我走到我知道代码必须出现的地方

and I start typing out the first three letters,

然后我开始打出前三个字母，

Autocomplete gets it and just gives you the code.

自动补全就懂了，直接给你代码。

And so I think this is a very high information bandwidth to specify what you want.

所以我觉得这是一种非常高的信息带宽，用来指定你想要什么。

If you point to the code where you want it and you type out the first few pieces and the model will complete it.

如果你指向你想要代码的位置，然后打出前几部分，模型就会把它补全。

So I guess what I mean is I think these models are good in certain parts of the stack.

所以我想我的意思是，我认为这些模型在技术栈的某些部分表现很好。

I actually use the models a little bit in, there are two examples where I actually use the models that I think are illustrative.

我其实稍微用了一下这些模型，有两个例子，我实际用了这些模型，我觉得挺有说明性的。

One was when I generated the report, that's actually more boilerplate-y.

一个是我生成报告的时候，那其实更像是模板化的东西。

So actually, if I coded partially some of that stuff, that was fine.

所以其实，如果我部分地编写了那些东西，那也没问题。

Because it's not like mission-critical stuff and it works fine.

因为它不是什么关键任务的东西，而且运行得挺好。

And then the other part is when I was rewriting the tokenizer in Rust,

然后另一部分是我用 Rust 重写分词器的时候，

I'm actually not as good at Rust because I'm fairly new to Rust.

我其实不太擅长 Rust，因为我对 Rust 还比较陌生。

So I was doing, there's a bit of Vibe coding going on when I was writing some of the Rust code.

所以我在做的时候，写那些 Rust 代码时有点在搞 Vibe coding。

But I had Python implementation that I fully understand, and I'm just making sure I'm making a more efficient version of it and I have tests.

但我有一个我完全理解的 Python 实现，我只是在确保我做一个更高效的版本，而且我有测试。

So I feel safer doing that stuff.

所以做那些东西我感觉更安心。

And so basically they lower or like they increase accessibility to languages or paradigms that you might not be as familiar with.

所以基本上，它们降低或者说是提高了你对可能不太熟悉的语言或范式的可及性。

So I think they're very helpful there as well.

所以我觉得它们在这方面也很有帮助。

Because there's a ton of Rust code out there.

因为外面有大量的Rust代码。

The models are actually pretty good at it.

这些模型实际上相当擅长这个。

I happen to not know that much about it.

我碰巧对它不太了解。

So the models are very useful there.

所以模型在这方面非常有用。

The reason I think this question is so interesting is because

我觉得这个问题之所以如此有趣，是因为

the main story people have about AI exploding and getting to super intelligence pretty rapidly is AI automating, AI engineering, and AI research.

人们对AI爆炸式发展并相当迅速地达到超级智能的主要叙事是AI自动化、AI工程和AI研究。

So they'll look at the fact that you can have cloud code and make entire CRUD applications from scratch and be like,

所以他们会看到这样一个事实：你可以用云代码从零开始做出完整的CRUD应用，然后说，

if you had this same capability inside of open AI and deep mind and everything,

如果你在OpenAI、DeepMind等所有地方都有同样的能力，

well, just imagine the level of like just, you know, a thousand of you or a million of you in parallel finding little architectural tweaks.

嗯，就想象一下，比如，你知道，一千个你或一百万个你并行地寻找小的架构调整。

And so it's quite interesting to hear you say that this is the thing they're sort of asymmetrically worse at.

所以听你说这是它们某种程度上不对称地更不擅长的事情，这相当有趣。

And it's quite relevant to forecasting whether the AI 2027 type explosion is likely to happen anytime soon.

而这与预测AI 2027那种爆炸是否可能很快发生相当相关。

I think that's a good way of putting it.

我觉得这个说法很好。

And I think you're getting at some of my, like, why my timelines are a bit longer.

而且我觉得你触及了我的一些，比如，为什么我的时间线要长一些。

You're right.

你说得对。

I think, yeah, they're not very good at code that hasn't never been written before.

我觉得，是的，它们不太擅长处理那些从未被写过的代码。

Maybe it's like one way to put it, which is like what we're trying to achieve when we're building these models.

也许这是一种说法，就像我们在构建这些模型时试图实现的目标。

Very naive question, but the architectural tweaks that you're adding to NanoChat,

非常天真的问题，但你添加到NanoChat中的架构调整，

they're in a paper somewhere, right? They might even be in a repo somewhere.

它们在某篇论文里，对吧？它们甚至可能在某处的一个代码库里。

So is it surprising that they aren't able to integrate that whenever you're like add rope embeddings or something, they do that in the wrong way?

所以，当你比如添加rope嵌入之类的，它们无法整合那个，它们会以错误的方式做，这令人惊讶吗？

It's tough. I think they kind of know, but they don't fully know.

这很难。我觉得它们有点知道，但并非完全知道。

And they don't know how to fully integrate it into the repo and your style and your code and your place and some of the custom things that you're doing and how it fits with all the assumptions of the repository and all this kind of stuff.

而且它们不知道如何将它完全整合到代码库、你的风格、你的代码、你的位置以及你正在做的一些自定义内容中，以及它如何与代码库的所有假设和所有这些事情相适应。

So I think they do have some knowledge, but they haven't gotten to the place where they can actually integrate it, make sense of it, and so on.

所以我认为它们确实有一些知识，但还没有达到能够真正整合它、理解它等等的地步。

I do think that a lot of the stuff, by the way, continues to improve.

顺便说一句，我确实认为很多东西在持续改进。

So I think currently probably state-of-the-art model that I go to is the GPT-5 Pro.

所以我认为目前我使用的最先进的模型可能是GPT-5 Pro。

And that's a very, very powerful model.

那是一个非常非常强大的模型。

So if I actually have 20 minutes, I will copy-paste my entire repo and I go to GPT-5 Pro, the Oracle, for like some questions.

所以如果我真的有20分钟，我会复制粘贴我的整个代码库，然后我去找GPT-5 Pro，那个神谕，问一些问题。

And often it's not too bad, and surprisingly good compared to what existed a year ago.

而且通常它不算太糟，跟一年前存在的东西相比，好得令人惊讶。

But I do think that overall the models are, they're not there.

但我确实认为，总体而言这些模型，它们还没到那个水平。

And I kind of feel like the industry, it's over, it's making too big of a jump and it's trying to pretend like this is amazing and it's not, it's slop.

我有点觉得这个行业，它完了，它跳得太大了，它试图假装这很了不起，但并不是，它就是垃圾。

And I think they're not coming to terms with it and maybe they're trying to fundraise or something like that.

我觉得他们还没接受这一点，也许他们是想融资或者什么的。

I'm not sure what's going on, but we're at this intermediate stage.

我不确定发生了什么，但我们正处于这个中间阶段。

The models are amazing. They still need a lot of work.

这些模型很了不起。它们还需要大量的改进。

For now, autocomplete is my sweet spot.

目前来说，自动补全是我的最佳选择。

But sometimes for some types of code, I will go to a nullim agent.

但有时候对于某些类型的代码，我会去用一个nullim智能体。

Yeah. Actually, here's another reason why this is really interesting.

是的。其实，这也是为什么这件事真的很有意思的另一个原因。

Through the history of programming, there's been many productivity improvements, compilers, linting, better programming languages, et cetera,

纵观编程的历史，有过很多生产力的提升，编译器、代码检查、更好的编程语言等等，

which have increased programmer productivity, but have not led to an explosion.

这些提高了程序员的生产力，但并没有带来爆发式的增长。

So that sounds very much like autocomplete tab.

所以那听起来非常像自动补全的Tab键。

And this other category is just like automation of the programmer.

而另一个类别就像是程序员的自动化。

Yeah. And it's interesting you're seeing more in the category of the historical analogies of like, you know, better compilers or something.

是的。有意思的是，你在历史类比这个类别里看到更多，比如，你知道，更好的编译器之类的。

Maybe because this gets at one other kind of thought that is like,

也许是因为这触及了另一种想法，就是，

I do feel like I have a hard time differentiating where AI begins and stops.

我确实觉得我很难区分AI从哪里开始、到哪里结束。

Because I do see AI as fundamentally an extension of computing in some pretty fundamental way.

因为我确实把AI看作是在某种相当根本的层面上对计算的延伸。

And I feel like I see a continuum of this kind of like recursive self-improvement or like of speeding up programmers all the way from the beginning.

而且我觉得我看到了一种连续体，就是这种递归式的自我改进，或者说从最开始就一直在加速程序员的工作。

Like even like I would say like code editors, syntax highlighting, syntax or like checking even of the types, like data type checking.

就像甚至我会说，比如代码编辑器、语法高亮、语法检查，甚至类型检查，比如数据类型检查。

All these kinds of tools that we've built for each other, even search engines.

所有这些我们为彼此打造的工具，甚至搜索引擎。

Like why aren't search engines part of AI?

比如为什么搜索引擎不算AI的一部分呢？

Like I don't know, like ranking is kind of AI, right?

比如我不知道，像排名排序也算是一种AI，对吧？

At some point Google was like, even early on they were thinking of themselves as an AI company doing Google search engine, which I think is totally fair.

在某个时候，谷歌就觉得，甚至在早期，他们就把自己看作是一家做谷歌搜索引擎的AI公司，我觉得这完全合理。

And so I kind of see it as a lot more of a continuum than I think other people do.

所以我觉得它更像是一个连续体，比其他人认为的程度要高得多。

and I don't, it's hard for me to draw the line.

而我不，我很难划清这条界线。

And I kind of feel like, okay, we're now getting a much better autocomplete.

我有点觉得，好吧，我们现在得到了一个更好的自动补全。

And now we're also getting some agents which are kind of like these loopy things, but they kind of go off rails sometimes.

现在我们也有了一些智能体，它们有点像这种循环的东西，但有时候会跑偏。

And what's going on is that the human is progressively doing a bit less and less of the low-level stuff.

而正在发生的是，人类逐渐越来越少地做那些底层的事情。

For example, we're not writing the assembly code because we have compilers, right?

比如说，我们不写汇编代码了，因为我们有编译器，对吧？

Like compilers will take my high-level language and see and write the assembly code.

就像编译器会接收我的高级语言，然后查看并编写汇编代码。

So we're abstracting ourselves very, very slowly.

所以我们正在非常、非常缓慢地把自己抽象化。

And there's this, what I call autonomy slider of like more and more stuff is automated of the stuff that can be automated at any point in time.

还有这个，我称之为自主性滑块，就是越来越多的事情被自动化，那些在任何时间点可以被自动化的事情。

And we're doing a bit less and less and raising ourselves in the layer abstraction over the automation.

而我们做得越来越少，并在自动化之上提升自己的抽象层次。

One of the big problems with RL is that it's incredibly information sparse.

强化学习的一大问题是它的信息极其稀疏。

Labelbox can help you with this by increasing the amount of information that your agent gets to learn from with every single episode.

Labelbox 可以通过增加你的智能体在每个回合中能够学习的信息量来帮助你解决这个问题。

For example, one of their customers wanted to train a coding agent.

例如，他们的一个客户想要训练一个编码智能体。

So Labelbox augmented an IDE with a bunch of extra data collection tools and staffed a team of expert software engineers from their aligner network to generate trajectories that were optimized for training.

所以 Labelbox 为一个 IDE 增加了许多额外的数据收集工具，并从他们的对齐器网络中配备了一支专家软件工程师团队，来生成针对训练优化的轨迹。

Now, obviously, these engineers evaluated these interactions on a pass-fail basis,

现在，显然，这些工程师以通过/失败的方式评估了这些交互，

but they also rated every single response on a bunch of different dimensions, like readability and performance.

但他们也根据许多不同的维度对每一个回应进行了评分，比如可读性和性能。

And they wrote down their thought processes for every single rating that they gave.

并且他们写下了他们给出的每一个评分的思考过程。

So you're basically showing every single step an engineer takes and every single thought that they have while they're doing their job.

所以你基本上展示了工程师采取的每一个步骤，以及他们在工作时的每一个想法。

And this is just something you could never get from usage data alone.

而这是你仅从使用数据中永远无法获得的东西。

And so LabelBox packaged up all these evaluations and included all the agent trajectories and the corrective human edits for the customer to train on.

所以 LabelBox 打包了所有这些评估，并包含了所有智能体轨迹和纠正性的人工编辑，供客户用来训练。

This is just one example.

这只是一个例子。

So go check out how LabelBox can get you high quality frontier data across domains, modalities, and training paradigms.

所以去看看 LabelBox 如何为你获取跨领域、跨模态和跨训练范式的高质量前沿数据。

Reach out at labelbox.com slash Thawarkech.

请通过 labelbox.com/Thawarkech 联系我们。

Let's talk about Aura a little bit.

我们来稍微谈谈 Aura。

You two did some very interesting things about this.

你们俩在这方面做了一些非常有趣的事情。

Conceptually, how should we think about the way that humans are able to build a rich world model just from interacting with our environment

从概念上讲，我们应该如何看待人类仅仅通过与环境的互动就能构建丰富的世界模型

and in ways that seems almost irrespective of the final reward at the end of the episode?

而且其方式似乎几乎与回合结束时的最终奖励无关？

If somebody's starting to start a business and at the end of 10 years she finds out whether the business succeeded or failed,

如果某人开始创业，十年后她才知道生意成功还是失败，

we say that she's earned a bunch of wisdom and experience.

我们会说她获得了一堆智慧和经验。

But it's not because the log probs of every single thing that happened over the last 10 years are up-weighted or down-weighted.

但这不是因为过去十年里发生的每一件事的对数概率被上调或下调了。

It's something much more deliberate and rich is happening.

而是正在发生某种更加刻意且丰富的事情。

What is the ML analogy, and how does that compare to what we're doing with LLMs right now?

机器学习中的类比是什么，它与我们现在用大语言模型做的事情相比如何？

Yeah, maybe the way I would put it is humans don't use reinforcement learning, as I've said.

是的，也许我会这样表述：正如我所说，人类不使用强化学习。

I think they do something different, which is, yeah, you experience.

我认为他们做的是不同的事情，那就是，是的，你体验。

So reinforcement learning is a lot worse than I think the average person thinks.

所以强化学习比我认为普通人想象的要糟糕得多。

Reinforcement learning is terrible.

强化学习很糟糕。

It just so happens that everything that we had before is much worse.

只是碰巧我们之前的一切都要糟糕得多。

Because previously we were just imitating people, so it has all these issues.

因为之前我们只是在模仿人类，所以它存在所有这些毛病。

So in reinforcement learning, say you're working with, you're solving a math problem.

所以在强化学习中，假设你在处理，你在解一道数学题。

This is very simple.

这非常简单。

You're given a math problem and you're trying to find the solution.

你拿到一道数学题，然后你试图找到解答。

Now in reinforcement learning, you will try lots of things in parallel first.

而在强化学习中，你会先并行尝试很多种做法。

So you're given a problem, you try hundreds of different attempts.

所以你拿到一个问题，你尝试几百种不同的解法。

And these attempts can be complex, right?

而这些尝试可能很复杂，对吧？

They can be like, oh, let me try this, let me try that, this didn't work, that didn't work, et cetera.

它们可能是这样的：哦，让我试试这个，让我试试那个，这个不行，那个也不行，等等。

And then maybe you get an answer.

然后也许你得到了一个答案。

And now you check the back of the book, and you see, okay, the correct answer is this.

现在你翻到书后面查看，你看到，好吧，正确答案是这个。

And then you can see that, okay, this one, this one, and that one got the correct answer, but these other 97 of them didn't.

然后你能看到，好吧，这一个、这一个，还有那一个得到了正确答案，但其他97个都没有。

So literally what reinforcement learning does is it goes to the ones that worked really well.

所以强化学习实际上做的就是，它去找那些效果非常好的解法。

And every single thing you did along the way, every single token gets up-weighted of like, do more of this.

而你一路上做的每一件事，每一个词元都会被加权，就像是，多做这个。

The problem with that is, I mean, people will say that your estimator has high variance,

这样做的问题在于，我的意思是，人们会说你的估计器方差很高，

but I mean, it's just noisy. It's noisy.

但我的意思是，它只是有噪声。它有噪声。

So basically, it kind of almost assumes that every single little piece of the solution

所以基本上，它差不多是假设解决方案的每一个小部分

that you made, that right-to-de-dry answer was the correct thing to do, which is not true.

你做的，那个所谓正确的答案就是该做的事，而这是不对的。

Like you may have gone down the wrong alleys until you arrived the right solution.

比如你可能走错了路，直到你找到了正确的解决方案。

Every single one of those incorrect things you did, as long as you got to the correct solution, will be up-weighted as do more of this.

你做的每一件错误的事，只要你最终得到了正确的解决方案，都会被加权，就像“多做这个”一样。

It's terrible. Yeah. It's noise.

这太糟糕了。是啊。这就是噪声。

You've done all this work only to find a single, at the end, you get a single number of like, oh, you did correct.

你做了所有这些工作，结果最后只得到一个单一的数字，比如，哦，你做对了。

And based on that, you weigh that entire trajectory as like upweight or downweight.

然后基于这个，你把整条轨迹加权，比如上调或下调。

And so the way I like to put it is you're sucking supervision through a straw

所以我喜欢这样形容：你是在用吸管吸监督信号

because you've done all this work that could be a minute to roll out and you're like sucking the bits of supervision of the final reward signal through a straw

因为你做了所有这些可能要花一分钟才能展开的工作，而你却像用吸管吸着最终奖励信号的那一点点监督

and you're like putting it, you're like basically like, yeah, you're broadcasting that across the entire trajectory and using that to upweight or downweight that trajectory.

然后你就像把它，你基本上就像，是的，你把它广播到整条轨迹上，并用它来上调或下调那条轨迹。

It's crazy. A human would never do this.

这太疯狂了。人类绝不会这样做。

Number one, a human would never do hundreds of rollouts.

第一，人类绝不会做数百次展开。

Number two, when a person sort of finds a solution,

第二，当一个人找到某种解决方案时，

they will have a pretty complicated process of review of like, okay, I think these parts that I did well, these parts I did not do that well.

他们会有相当复杂的审查过程，比如，我觉得这些部分我做得不错，这些部分我做得不太好。

I should probably do this or that. And they think through things.

我可能应该这样做或那样做。然后他们会仔细思考。

There's nothing in current LLMs that does this. There's no equivalent of it.

当前的LLM中没有任何东西能做到这一点。没有与之等价的东西。

But I do see papers popping out that are trying to do this because it's obvious to everyone in the field.

但我确实看到一些论文冒出来，试图做到这一点，因为这对领域里的每个人来说都是显而易见的。

So I kind of see as like, the first imitation learning actually, by the way, was extremely surprising and miraculous and amazing that we can fine-tune by imitation on humans.

所以我有点把它看作，其实顺便说一句，第一个模仿学习是非常令人惊讶、神奇和了不起的，我们可以通过对人类进行模仿来微调。

And that was incredible. Because in the beginning, all we had was base models.

那太不可思议了。因为一开始，我们只有基础模型。

base models are autocomplete. And it wasn't obvious to me at the time,

基础模型就是自动补全。当时我并没有意识到，

and I had to learn this, and the paper that like blew my mind was InstructGPT,

我必须学习这一点，而让我大开眼界的论文是InstructGPT，

because it pointed out that, hey, you can take the pre-trained model, which is autocomplete.

因为它指出，嘿，你可以拿预训练模型，也就是自动补全模型。

And if you just fine-tune it on text that looks like conversations,

如果你只是用看起来像对话的文本对它进行微调，

the model will very rapidly adapt to become very conversational, and it keeps all the knowledge from pre-training.

模型会非常迅速地适应，变得非常对话化，并且保留预训练中的所有知识。

And this blew my mind, because I didn't understand that it's just like stylistically can adjust so quickly and become an assistant to a user.

这让我大开眼界，因为我不明白它只是像在风格上可以如此迅速地调整，并成为用户的助手。

through just a few loops of fine-tuning on that kind of data.

只需要在那类数据上做几轮微调。

It was very miraculous to me that that worked.

这对我来说非常神奇，居然那样就能奏效。

So incredible, and that was like two years, three years of work.

太不可思议了，而那大概是两三年、三年的工作。

And now came RL.

然后强化学习登场了。

And RL allows you to do a bit better than just imitation learning, right?

而强化学习让你能比单纯的模仿学习做得更好一点，对吧？

Because you can't have these reward functions, and you can hill climb on the reward functions.

因为你可以设定这些奖励函数，然后在这些奖励函数上爬山。

And so some problems have just correct answers.

所以有些问题就是有正确答案的。

You can hill climb on that without getting expert trajectories to imitate.

你可以在那上面爬山，而不需要获取专家轨迹来模仿。

So that's amazing.

所以这太棒了。

And the model can also discover solutions that a human might never come up with.

而且模型还能发现人类可能永远想不到的解决方案。

So this is incredible.

所以这太不可思议了。

And yet, it's so stupid.

然而，它又那么蠢。

So I think we need more.

所以我觉得我们需要更多。

And so I saw a paper from Google yesterday that tried to have this reflect and review page idea in mind.

所以我昨天看到谷歌的一篇论文，试图把这种反思与回顾页面的想法纳入其中。

What was the memory bank paper or something?

那个记忆库论文还是什么来着？

I don't know.

我不知道。

I've actually seen a few papers along these lines.

我其实见过几篇沿着这个思路的论文。

So I expect there to be some kind of a major update to how we do algorithms for LLMs coming in that realm.

所以我预计在那个领域，我们为大型语言模型做算法的方式会有某种重大更新。

And then I think we need three or four or five more.

然后我觉得我们还需要三四个或五个更多。

Something like that.

差不多是这样。

But you're so good to come up with evocative phrases, sucking supervision through a straw.

但你太擅长想出这种生动的说法了，用吸管吸监督信号。

It's like so good.

这说法真是太好了。

Why hasn't, so you're saying like your problem with outcome-based reward is that you have this huge trajectory.

为什么还没有，所以你是说，你对基于结果的奖励的问题在于你有一条巨大的轨迹。

And then at the end, you're trying to learn every single possible thing about what you should do and what you should learn about the world from that one final bit.

然后到最后，你试图从那最后一点中学习所有可能的东西，关于你应该做什么，以及你应该从这个世界学到什么。

Why hasn't, given the fact that this is obvious, why hasn't process-based supervision as an alternative been a successful way to make models more capable?

为什么，既然这一点很明显，为什么基于过程的监督作为一种替代方案，没有成为让模型更强大的成功方法？

What has been preventing us from using this alternative paradigm?

是什么阻止了我们使用这种替代范式？

So process-based supervision just refers to the fact that we're not going to have a reward function only at the very end of, after you've made 10 minutes of work, I'm not going to tell you you did well or not well.

所以基于过程的监督只是指这样一个事实：我们不会只在最后才有一个奖励函数，在你做了10分钟的工作之后，我不会告诉你你做得好还是不好。

I'm going to tell you at every single step of the way how well you're doing.

我会在每一步都告诉你你做得怎么样。

And this is basically the reason we don't have that. It's not as tricky how you do that properly because you have partial solutions and you don't know how to assign credit.

而这基本上就是我们没有那个的原因。如何正确地做到这一点并没有那么棘手，因为你有部分解决方案，而你不知道如何分配功劳。

So when you get the right answer, it's just an equality match to the answer. Very simple to implement.

所以当你得到正确答案时，它只是与答案进行相等匹配。实现起来非常简单。

If you're doing basically process supervision, how do you assign in an automatical way partial credit assignment?

如果你基本上是在做过程监督，你如何以自动化的方式分配部分功劳？

It's not obvious how you do it. Lots of labs, I think, are trying to do it with these LLM judges.

如何做到这一点并不明显。我认为很多实验室都在尝试用这些LLM评判员来做这件事。

So basically, you get LLMs to try to do it.

所以基本上，你让LLM来尝试做这件事。

So you prompt an LLM, hey, look at a partial solution of a student.

所以你给LLM一个提示，嘿，看看一个学生的部分解答。

How well do you think they're doing if the answer is this?

如果答案是这样一个，你觉得他们做得怎么样？

And they try to tune the prompt.

然后他们试着调整提示。

The reason that I think this is kind of tricky is quite subtle.

我认为这件事有点棘手的原因相当微妙。

And it's the fact that anytime you use an LLM to assign a reward,

而事实是，任何时候你用LLM来分配奖励，

those LLMs are giant things with billions of parameters and they're gameable.

那些LLM是拥有数十亿参数的庞然大物，而且它们是可以被钻空子的。

And if you're reinforcement learning with respect to them, you will find adversarial examples for your LLM judges, almost guaranteed.

如果你针对它们做强化学习，你几乎肯定会为你的LLM评判器找到对抗性样本。

You can't do this for too long.

你不能这样做太久。

You do maybe 10 steps or 20 steps, maybe it will work,

你也许做10步或20步，也许它会奏效，

but you can't do 100 or 1,000 because it's not obvious.

但你不能做100步或1000步，因为这不明显。

because I understand it's not obvious, but basically the model will find little cracks.

因为我明白这不明显，但基本上模型会找到一些小裂缝。

It will find all these like spurious things in the nooks and crannies of the giant model and find a way to cheat it.

它会在这个巨大模型的角角落落里找到所有这些像是虚假的东西，并找到一种方法来欺骗它。

So one example that's prominently in my mind is,

所以一个我印象深刻的例子是，

I think this was probably public, but basically if you're using an LM judge for a reward,

我想这可能是公开的，但基本上如果你用一个语言模型评判器来作为奖励，

so you just give it a solution from a student and ask it if the student did well or not.

所以你只要给它一个学生的解答，然后问它这个学生做得好不好。

We were training with reinforcement learning against that reward function and it worked really well.

我们当时针对那个奖励函数做强化学习训练，而且效果非常好。

And then suddenly the reward became extremely large.

然后突然之间，奖励变得极其巨大。

Like it was a massive jump and it did perfect.

就像是一次巨大的跳跃，它做得完美无缺。

And you're looking at it like, wow,

然后你看着它，心想，哇，

this means the student is perfect in all these problems.

这意味着这个学生在所有这些问题上都是完美的。

It's fully solved math.

这是完全解出来的数学题。

But actually what's happening is that when you look at the completions that you're getting from the model,

但实际上，当你看看从模型那里得到的补全结果时，

they are complete nonsense.

它们完全是一派胡言。

They start out okay and then they change to da-da-da-da-da-da-da.

它们一开始还行，然后就变成了哒哒哒哒哒哒哒。

So it's just like, oh, okay, let's take two plus three and we do this and this and then da-da-da-da-da-da-da.

所以就像，哦，好吧，我们拿二加三，我们这样做，这样做，然后就哒哒哒哒哒哒哒。

And you're looking at it and it's like, this is crazy.

然后你看着它，心想，这太疯狂了。

How is it getting a reward of one or 100%?

它怎么会得到一或100%的奖励？

And you look at the LLM judge, and it turns out the the-the-the-the-the is an adversarial example.

然后你看看那个LLM评判器，结果发现那个那个那个那个那个是一个对抗性样本。

It's for the model, and it assigns 100% probability to it.

它是针对这个模型的，而且它给它分配了100%的概率。

And it's just because this is an out-of-sample example to the LLM.

而这只是因为这对LLM来说是一个样本外的例子。

It's never seen you during training, and you're in pure generalization land.

它在训练期间从未见过你，而你正处于纯粹的泛化领域。

It's never seen you during training.

它在训练期间从未见过你。

And in the pure generalization land, you can find these examples that break it.

而在纯粹的泛化领域，你可以找到这些能攻破它的例子。

You're basically training the LLM to be a prompt injection model.

你基本上是在把LLM训练成一个提示注入模型。

Not even that.

甚至都不是那样。

Prompt injection is way too fancy.

提示注入太花哨了。

You're finding adversarial examples, as they're called.

你是在寻找所谓的对抗性样本。

These are nonsensical solutions that are obviously wrong, but the model thinks are amazing.

这些是毫无意义的解法，明显是错的，但模型却觉得它们很棒。

So to the extent you think this is the bottleneck to making RL more functional,

所以，如果你认为这是让强化学习更实用的瓶颈，

then that will require making LLMs better judges if you want to do this in an automated way.

那么如果你想以自动化的方式做到这一点，就需要让大语言模型成为更好的评判者。

And then so is it just going to be like some sort of GAN-like approach where you had to train models to be more robust?

那么，这会不会就像某种类似生成对抗网络的方法，你需要训练模型变得更鲁棒？

Yeah, I think the labs are probably doing all that.

是的，我认为实验室可能都在做这些。

Like, okay, so the obvious thing is like, the, the, the should not get 100% reward.

就像，好吧，所以显而易见的是，那个，那个，那个不应该得到100%的奖励。

Okay, well, take the, the, the, put in the training set of the LLM judge and say, this is not 100%. This is 0%. You can do this.

好吧，那么，拿那个，那个，那个，放进大语言模型评判者的训练集里，然后说，这不是100%。这是0%。你可以这样做。

But every time you do this, you get a new LLM and it still has adversarial examples.

但每次你这样做，你会得到一个新的大语言模型，而它仍然有对抗样本。

There's infinity adversarial examples.

对抗样本是无穷无尽的。

And I think probably if you iterate this a few times, it'll probably be harder and harder to find adversarial examples.

而且我认为，如果你把这个过程迭代几次，可能找到对抗样本会越来越难。

But I'm not 100% sure because this thing has a trillion parameters or whatnot.

但我不百分之百确定，因为这个东西有一万亿个参数之类的。

So I bet you the LLL labs are trying.

所以我敢打赌，那些大语言模型实验室正在尝试。

I don't actually, I still think, I still think we need other ideas.

其实我并不，我仍然认为，我仍然认为我们需要其他想法。

Interesting. Do you have some shape of what the other idea?

有意思。你对那个其他想法有什么大致的概念吗？

So this idea of review a solution and come up with synthetic examples such that when you train on them, you get better and meta-learn it in some way.

所以这个想法是，审查一个解决方案，并想出合成样本，使得当你用它们训练时，你会变得更好，并以某种方式元学习它。

And I think there's some papers that I'm starting to see pop out.

我觉得有些论文开始冒出来了。

I only am at a stage of reading abstracts

我只处于读摘要的阶段，

because a lot of these papers, they're just ideas.

因为很多这些论文，它们只是想法。

Someone has to actually make it work on a frontier LLM lab scale in full generality

得有人真正让它在前沿LLM实验室规模上完全通用地运作起来，

because when you see these papers, they pop up and it's just a little bit noisy.

因为当你看到这些论文时，它们冒出来，而且有点嘈杂。

It's cool ideas,

想法很酷，

But I haven't actually seen anyone convincingly show that this is possible.

但我还没真正看到有人令人信服地证明这是可能的。

That said, the LLM labs are fairly closed.

话虽如此，LLM实验室相当封闭。

So who knows what they're doing now.

所以谁知道他们现在在做什么。

But yeah.

但是啊。

So I guess I see a very, not easy, but like I can conceptualize how you would be able to train on synthetic examples or synthetic problems that you have made for yourself.

所以我想我能看到一种非常、不容易，但就像我能构想出你怎么能基于自己制造的合成例子或合成问题来训练。

But there seems to be another thing humans do.

但似乎还有另一件人类会做的事。

Maybe sleep is this.

也许睡眠就是这个。

Maybe daydreaming is this.

也许白日梦就是这个。

Which is not necessarily come up with fake problems, but just like reflect.

这不一定是要想出假问题，而只是像反思。

Yeah.

是的。

And I'm not sure what the ML analogy for, you know, daydreaming or sleeping, but just like just reflecting, I haven't come up with a new problem.

而且我不确定对于，你知道，白日梦或睡眠，但就像只是反思，机器学习上的类比是什么，我还没想出一个新问题。

I mean, obviously the very basic analogy would just be like fine tuning on reflection bits,

我是说，显然最基本的类比就像是在反思片段上微调，

but I feel like in practice that probably wouldn't work that well.

但我觉得在实践中那可能不会那么有效。

So I don't know if you have some take on what the analogy of like this thing is.

所以我不知道你对像这件事的类比有什么看法。

Yeah, I do think that we're missing some aspects there.

是的，我确实觉得我们在那里遗漏了一些方面。

So as an example, when you're reading a book, I almost feel like currently

所以举个例子，当你在读一本书时，我几乎觉得目前

when LLMs are reading a book, what that means is we stretch out the sequence of text and the model is predicting the next token and it's getting some knowledge from that.

当大语言模型在读一本书时，这意味着我们把文本序列拉长，模型在预测下一个词元，并从中获取一些知识。

That's not really what humans do, right?

这其实不是人类所做的，对吧？

So when you're reading a book, I almost don't even feel like the book is like exposition I'm supposed to be attending to and training on.

所以当你在读一本书时，我几乎甚至不觉得这本书是我应该关注和训练的说明性内容。

The book is a set of prompts for me to do synthetic data generation or for you to get into a book club and talk about it with your friends.

这本书是一组提示，让我进行合成数据生成，或者让你加入读书俱乐部，和朋友们讨论它。

And it's by manipulating that information that you actually gain that knowledge.

而正是通过操纵那些信息，你才真正获得了那些知识。

And I think we have no equivalent of that, again, with all alums.

而且我认为，再说一次，在所有校友中，我们没有与之等价的东西。

They don't really do that, but I'd love to see during pre-training some kind of a stage that thinks through the material and tries to reconcile it with what it already knows and thinks through for like some amount of time and gets that to work.

它们其实并不那样做，但我很想在预训练期间看到某种阶段，它会思考这些材料，试图将其与它已知的内容相协调，并思考一段时间，然后让这奏效。

And so there's no equivalence of any of this. This is all research.

所以这些都没有等价物。这全是研究。

There's some subtle, very subtle that I think are very hard to understand reasons why it's not trivial.

有一些微妙的、非常微妙的原因，我认为很难理解为什么这并非微不足道。

So if I can just describe one.

所以如果我能只描述一个的话。

Why can't we just synthetically generate and train on it?

为什么我们不能直接合成生成数据并在其上训练呢？

Well, because every synthetic example, like if I just give synthetic generation of the model thinking about a book,

嗯，因为每一个合成样本，比如我只是让模型合成生成关于一本书的思考，

you look at it and you're like, this looks great.

你看一看，然后你会觉得，这看起来很棒。

Why can't I train on it?

为什么我不能用它来训练呢？

Well, you could try, but the model will actually get much worse if you continue trying.

嗯，你可以试试，但如果你继续尝试，模型实际上会变得更糟。

And that's because all of the samples you get from models are silently collapsed.

那是因为你从模型得到的所有样本都在无声地坍缩。

They're silently, this is not obvious if you look at any individual example of it,

它们无声地，如果你看任何一个单独的样本，这并不明显，

they occupy a very tiny manifold of the possible space of sort of thoughts about content.

它们只占据了关于内容的可能思维空间中一个非常微小的流形。

So the LLMs, when they come off, they're what we call collapsed.

所以大语言模型，当它们输出时，它们就是我们所说的坍缩状态。

They have a collapsed data distribution.

它们有一个坍缩的数据分布。

If you sample, one easy way to say it is go to ChatGPT and ask it, tell me a joke.

如果你采样，一个简单的说法就是去问ChatGPT，给我讲个笑话。

It only has like three jokes.

它只有大概三个笑话。

It's not giving you the whole breadth of possible jokes.

它并没有给你所有可能笑话的广度。

It's giving you like, it knows like three jokes.

它给你的就像，它只知道大概三个笑话。

They're silently collapsed.

它们在无声地坍缩。

So basically, you're not getting the richness and diversity and the entropy from these models as you would get from humans.

所以基本上，你从这些模型得到的丰富性、多样性和熵，并不像从人类那里得到的那样。

So humans are a lot more sort of noisier, but at least they're not biased.

所以人类要嘈杂得多，但至少他们没有偏差。

They're not in a statistical sense.

从统计意义上说，他们不是。

They're not silently collapsed.

他们没有无声地坍缩。

They maintain a huge amount of entropy.

他们保持着大量的熵。

So how do you get synthetic data generation to work despite the collapse and while maintaining the entropy is a research problem?

那么，如何在崩溃的情况下让合成数据生成正常工作，同时保持熵，这是一个研究问题吗？

Just to make sure I understood, the reason that the collapse is relevant to synthetic data generation is because you want to be able to come up with synthetic problems or reflections which are not already in your data distribution?

只是想确认一下我理解得对不对，崩溃之所以与合成数据生成相关，是因为你希望能够提出一些尚未存在于你的数据分布中的合成问题或反思？

I guess what I'm saying is, say we have a chapter of a book and I ask Nolan to think about it.

我想说的是，假设我们有一本书的一章，我让诺兰去思考它。

It will give you something that looks very reasonable. But if I ask it 10 times, you'll notice that all of them are the same.

它会给你一些看起来非常合理的东西。但如果我问它10次，你会发现它们全都一样。

You can't just leave scaling, quote unquote, reflection on the same amount of prompt information and then get returns from that.

你不能只是把所谓的“反思”停留在同样数量的提示信息上，然后指望从中获得回报。

Yeah, yeah, yeah. So any individual sample will look okay, but the distribution of it is quite terrible.

是的，是的，是的。所以任何单个样本看起来都还行，但它的分布却相当糟糕。

And it's quite terrible in such a way that if you continue training on too much of your own stuff, you actually collapse.

而且它糟糕到这种程度：如果你继续用太多自己的东西训练，你实际上会崩溃。

I actually think that there's no fundamental solutions to this possibly.

我实际上认为，这个问题可能没有根本性的解决方案。

And I also think humans collapse over time.

而且我也认为人类会随着时间推移而崩溃。

I think this is, again, these analogies are surprisingly good, but humans collapse during the course of their lives.

我认为这再次说明，这些类比出奇地好，但人类会在他们的一生中崩溃。

This is why children have completely, you know, they haven't overfit yet.

这就是为什么孩子们完全，你知道，他们还没有过拟合。

And they will say stuff that will shock you because it's kind of, you can see where they're coming from,

他们会说一些让你震惊的话，因为你能看出他们的出发点，

but it's just not the thing people say.

但那并不是人们会说的话。

And because they're not yet collapsed, but we're collapsed.

因为他们还没有崩塌，但我们已经崩塌了。

We end up revisiting the same thoughts.

我们最终会反复回到同样的想法上。

We end up, you know, saying more and more of the same stuff and the learning rates go down and the collapse continues to get worse.

我们最终，你知道，越来越多地说着同样的东西，学习率下降，崩塌继续恶化。

And then everything deteriorates.

然后一切都会恶化。

Have you seen a super interesting paper that dreaming is a way of preventing this kind of overfitting and collapse?

你见过一篇超级有趣的论文吗，说做梦是防止这种过拟合和崩塌的一种方式？

That the reason dreaming is evolutionary adaptive is to put you in weird situations that are like very unlike your day-to-day reality so that to prevent this kind of overfitting?

说做梦之所以具有进化适应性，是为了把你置于一些非常不像你日常现实的奇怪情境中，从而防止这种过拟合？

It's an interesting idea.

这是个有趣的想法。

I mean, I do think that when you're generating things in your head and then you're attending to it, you're kind of like training on your own samples.

我的意思是，我确实认为当你在脑子里生成东西，然后你关注它时，你有点像在用自己的样本进行训练。

You're training on your synthetic data.

你在用你的合成数据进行训练。

And if you do it for too long, you go off rails and you collapse way too much.

如果你做太久，你就会失控，然后崩塌得太厉害。

So you always have to like seek entropy in your life.

所以你总是得在生活中寻找熵。

So talking to other people is a great source of entropy and things like that.

所以和他人交谈是熵的一个很好的来源，诸如此类。

So maybe the brain has also built some internal mechanisms for increasing the amount of entropy in that process.

所以也许大脑也建立了一些内部机制，来增加那个过程中的熵量。

But yeah, maybe that's an interesting idea.

但是，是的，也许那是个有趣的想法。

This is a very ill-formed thought, so I'll just put it out and let you react to it.

这是一个非常不成熟的想法，所以我只是把它说出来，让你来回应。

The best learners that we are aware of, which are children, are extremely bad at recollecting information.

我们所知道的最好的学习者，也就是儿童，在回忆信息方面非常糟糕。

In fact, at the very earliest stages of childhood, you will forget everything.

事实上，在童年的最早阶段，你会忘记一切。

You're just an amnesiac about everything that happens before a certain year date.

对于在某一年份日期之前发生的一切，你就像个失忆者。

But you're like extremely good at picking up new languages and learning from the world.

但你在学习新语言和从世界中学习方面非常擅长。

And maybe there's some element of like being able to see the forest for the trees.

也许有某种能够看到森林而不是树木的元素。

Whereas if you compare it to the opposite end of the spectrum, you have LLM pre-training,

而如果你把它与光谱的另一端相比，你有LLM预训练，

which these models will literally able to regurgitate word for word what is the next thing in a Wikipedia page.

这些模型将能够逐字逐句地复述维基百科页面上的下一个内容。

But their ability to learn abstract concepts really quickly the way a child can is much more limited.

但它们像孩子那样快速学习抽象概念的能力要有限得多。

And then adults are somewhere in between where they don't have the flexibility of childhood learning,

然后成年人介于两者之间，他们没有童年学习的灵活性，

but they can, you know, adults can memorize facts and information in a way that is harder for kids.

但他们可以，你知道，成年人可以以孩子更难做到的方式记忆事实和信息。

And I don't know if there's something interesting about that.

我不知道这其中是否有有趣的地方。

I think there's something very interesting about that.

我认为这其中有一些非常有趣的东西。

Yeah, 100%.

是的，百分之百。

I do think that humans actually, they do kind of like have a lot more of an element compared to LLMs

我确实认为，人类实际上，相比大语言模型，他们确实有更多的一种元素

of like seeing the forest for the trees.

就是那种看到森林而不是只看到树木的能力。

And we're not actually that good at memorization, which is actually a feature.

而我们其实并不那么擅长记忆，这其实是一个特点。

Because we're not that good at memorization, we actually are kind of like forced to find the patterns like in a more general sense.

因为我们不擅长记忆，我们实际上有点像是被迫去从更普遍的意义上寻找模式。

I think LLMs in comparison are extremely good at memorization.

我认为相比之下，大语言模型非常擅长记忆。

They will recite passages from all these training sources.

它们会背诵所有这些训练来源中的段落。

You can give them completely nonsensical data.

你可以给它们完全无意义的数据。

Like you can take, you can hash some amount of text or something like that.

比如你可以拿一些文本做哈希之类的。

You get a completely random sequence.

你会得到一个完全随机的序列。

If you train on it, even just, I think, a single iteration or two, it can suddenly regurgitate the entire thing.

如果你用它训练，哪怕只是，我觉得，一两次迭代，它就能突然把整个东西原样吐出来。

It will memorize it.

它会记住它。

There's no way a person can read a single sequence of random numbers and recite it to you.

一个人不可能读一遍随机数字序列就背给你听。

And that's a feature, not a bug, almost,

而这几乎是一个特点，而不是缺陷，

because it forces you to only learn the generalizable components,

因为它迫使你只学习那些可泛化的部分，

whereas LLMs are distracted by all the memory that they have of the pre-trained documents.

而大语言模型则被它们所拥有的预训练文档的记忆分散了注意力。

And it's probably very distracting to them in a certain sense.

从某种意义上说，这对它们来说可能非常分散注意力。

So that's why when I talk about the cognitive core,

所以这就是为什么当我谈论认知核心时，

I actually want to remove the memory,

我实际上想要移除记忆，

which is what we talked about,

也就是我们之前谈到的，

I'd love to have them have less memory so that they have to look things up.

我希望让它们拥有更少的记忆，这样它们就不得不去查找信息。

And they only maintain the algorithms for like thought and the idea of an experiment and all this cognitive glue of acting.

而它们只保留思考的算法、实验的想法，以及所有这些行动中的认知粘合剂。

And this is also relevant to preventing model collapse.

这也与防止模型崩溃有关。

Let me think.

让我想想。

I'm not sure. I think it's almost like a separate axis.

我不确定。我觉得这几乎像是一个独立的维度。

It's almost like the models are way too good at memorization,

这几乎就像是模型在记忆方面太擅长了，

and somehow we should remove that.

而不知怎么的，我们应该移除这一点。

And I think people are much worse, but it's a good thing.

我认为人类要差得多，但这是件好事。

What is a solution to model collapse?

模型崩溃的解决方案是什么？

I mean, there's very naive things you could attempt.

我的意思是，你可以尝试一些非常天真的方法。

It's just like the distribution over logis should be wider or something.

就像逻辑值上的分布应该更宽之类的。

There's many naive things you could try.

有很多天真的方法你可以尝试。

What ends up being the problem with the naive approaches?

这些天真方法最终的问题是什么？

Yeah, I think that's a great question.

是的，我认为这是个很好的问题。

I mean, you can imagine having a regularization for entropy and things like that.

我的意思是，你可以想象对熵进行正则化之类的。

I guess they just don't work as well empirically

我猜它们只是在经验上效果不太好，

because right now, like the models are collapsed.

因为现在，模型已经崩溃了。

But I will say most of the tasks that we want of them don't actually demand the diversity.

但我要说，我们想要它们完成的大多数任务实际上并不需要多样性。

It's probably the answer of what's going on.

这很可能就是正在发生的事情的答案。