#490 – State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI

The following is a conversation all about the state-of-the-art in artificial intelligence,

接下来的对话全部关于人工智能领域的最新技术，

including some of the exciting technical breakthroughs and developments in AI that happened over the past year,

包括过去一年中人工智能领域发生的一些令人兴奋的技术突破和发展，

and some of the interesting things we think might happen this upcoming year.

以及我们认为今年可能发生的一些有趣的事情。

At times, it does get super technical,

有时候，内容确实会非常技术性，

but we do try to make sure that it remains accessible to folks outside the field without ever dumbing it down.

但我们确实努力确保它对领域外的人也能理解，同时绝不降低深度。

It is a great honor and pleasure to be able to do this kind of episode with two of my favorite people in the AI community, Sebastian Rashka and Nathan Lambert.

能够与人工智能社区中我最喜欢的两个人——塞巴斯蒂安·拉什卡和内森·兰伯特——一起做这样一期节目，是我的莫大荣幸和快乐。

They are both widely respected machine learning researchers and engineers who also happen to be great communicators, educators, writers, and Twitterers, ex-posters.

他们都是广受尊敬的机器学习研究者和工程师，同时也恰好是出色的沟通者、教育者、作家和推特用户，前发帖人。

Sebastian is the author of two books I highly recommend for beginners and experts alike.

塞巴斯蒂安是两本书的作者，我强烈推荐给初学者和专家。

First is build a large language model from scratch and build a reasoning model from scratch.

第一本是《从零构建大型语言模型》和《从零构建推理模型》。

I truly believe in the machine learning computer science world, the best way to learn and understand something is to build it yourself from scratch.

我真心相信，在机器学习计算机科学领域，学习和理解某件事的最佳方式就是自己从零开始构建它。

Nathan is the post-training lead at the Allen Institute for AI and author of the definitive book on reinforcement learning from human feedback.

内森是艾伦人工智能研究所的后训练负责人，也是关于从人类反馈中进行强化学习的权威书籍的作者。

Both of them have great X accounts, great substacks.

他们两人都有很棒的X账号和很棒的Substack。

Sebastian has courses on YouTube.

塞巴斯蒂安在YouTube上有课程。

Nathan has a podcast.

内森有一个播客。

And everyone should absolutely follow all of those.

大家都绝对应该关注所有这些。

And now a quick few second mention of each sponsor.

现在快速用几秒钟提一下每个赞助商。

Check them out in the description or at lexfriedman.com/sponsors.

请在描述中或访问 lexfriedman.com/sponsors 查看它们。

It is, in fact, the best way to support this podcast.

事实上，这是支持这个播客的最佳方式。

We got a bunch of great sponsors.

我们有很多很棒的赞助商。

for intelligent content management, Quo for a phone system

用于智能内容管理，Quo 用于电话系统

like call stacks, contacts for your business,

比如呼叫栈、企业联系人，

Uplift Desk, the desk I'm sitting behind, and my favorite office desk.

Uplift Desk，我正坐在后面的桌子，也是我最喜欢的办公桌。

Fin for customer service AI agents,

Fin 用于客户服务 AI 代理，

Shopify for selling stuff online,

Shopify 用于在线销售，

CodeRabbit for AI-powered code review,

CodeRabbit 用于 AI 驱动的代码审查，

Element for electrolytes,

Element 用于电解质，

and of course, our longtime friend, Perplexity, for curiosity-driven knowledge exploration.

当然，还有我们的老朋友 Perplexity，用于好奇心驱动的知识探索。

Choose wisely, my friends.

明智地选择吧，我的朋友们。

And now, on to the full ad reads.

现在，进入完整的广告朗读。

I try to make them interesting, but if you do skip,

我尽量让它们有趣，但如果你跳过，

please still check out the sponsors.

请仍然查看赞助商。

I enjoy their stuff.

我喜欢他们的东西。

Maybe you will, too.

也许你也会喜欢。

To get in touch with me, for whatever reason, go to lexfreemer.com/contact.

要联系我，无论什么原因，请访问 lexfreemer.com/contact。

If you can't tell, I'm trying to have a bit of a pep in my step.

如果你看不出来，我正试着让自己精神一点。

At the moment.

目前。

Because I had a long night, didn't get much sleep at all.

因为我熬了一夜，根本没怎么睡。

So I am running on fumes, delirious, happy, unsure of what is reality and what is a dream.

所以我靠着一股劲在撑，神志不清，开心，分不清什么是现实什么是梦境。

In fact, we could right now be living inside of a dream.

事实上，我们现在可能正生活在一个梦里。

I have been going through a lot.

我经历了很多。

I have been working insane hours, so much going on.

我一直都在疯狂地工作，有太多事情在发生。

I am so overwhelmed.

我真是太不堪重负了。

Of course, as always, truly grateful and happy to be alive.

当然，一如既往，真心感激并且高兴自己还活着。

but have not been able to publish as many episodes as you would like,

但没能发布像你希望的那样多的节目，

so there's a bunch of sponsors we have to catch up on.

所以有一堆赞助商我们需要补上。

Your support truly means the world.

你的支持真的意义重大。

Please check out all the sponsors.

请查看所有的赞助商。

If you think it might be useful to you, buy their stuff.

如果你觉得可能对你有用，就买他们的东西。

It really is the best way to support this podcast.

这真的是支持这个播客的最好方式。

All right, let's go.

好的，我们开始吧。

First up, this episode is brought to you by Box,

首先，本期节目由 Box 赞助，

a cloud-based platform for content management, file sharing,

一个基于云的内容管理、文件共享平台，

And all kinds of collaboration, all kinds of content for your businesses.

以及各种协作，各种适合你企业的内容。

Like with a lot of companies, the big question is,

就像很多公司一样，最大的问题是，

how is AI leveraged to make whatever the business does better?

如何利用 AI 让企业所做的任何事情变得更好？

A lot of companies kind of use it for the hype and the label.

很多公司有点把它用于炒作和标签。

It's kind of hilarious to watch people just say like, powered by AI.

看着人们只是说“由 AI 驱动”，有点好笑。

I don't care if you're a bakery, powered by AI.

我不在乎你是不是一家面包店，由 AI 驱动。

I don't know.

我不知道。

But outside of all of the hype,

但抛开所有的炒作，

it is one of the most incredible things that humans have ever created.

它是人类创造过的最不可思议的事物之一。

And so companies that can leverage that well are the companies that win.

所以能够很好地利用它的公司就是赢家。

And of course, Box is legendary for its file and content management,

当然，Box 因其文件和内容管理而闻名，

especially when you're talking about scale.

尤其是在谈到规模的时候。

So obviously, it's amenable for the utilization of AI to help automate some of the document processing, some of the workflow, some of the organization.

所以很明显，它适合利用AI来帮助自动化一些文档处理、一些工作流程、一些组织工作。

and they do that exceptionally well.

而且他们做得非常出色。

They have a system called, as you could imagine, Box AI that does just that.

他们有一个系统，正如你可以想象的，叫做Box AI，正是做这个的。

I love it.

我很喜欢它。

They do an excellent implementation on the interface side, on the backend side.

他们在界面端和后端都做了出色的实现。

Everything works extremely nicely.

一切都运行得非常好。

Help scale AI across your organization today and go to box.com slash AI.

今天就帮助在你的组织中扩展AI，请访问box.com/AI。

That's box.com slash AI to learn more.

那就是box.com/AI，了解更多。

This episode is also brought to you by Quo, spelled Q-U-O.

本期节目还由Quo赞助，拼写为Q-U-O。

Also happens to be a company name with just three letters that will help you win at Scrabble.

它恰好也是一个只有三个字母的公司名，能帮你在拼字游戏中获胜。

Are you allowed to use company names with Scrabble?

拼字游戏允许使用公司名称吗？

How many points is Q?

Q值多少分？

How many points is U?

U值多少分？

I'm imagining a lot.

我想象有很多分。

That was one of the big confusions to me when I was first learning the English language.

这是我最初学习英语时最大的困惑之一。

It always felt like Q should be at the end of the alphabet, maybe like QZ.

总觉得Q应该在字母表的末尾，也许像QZ那样。

It was always surprising to my limited brain capacity that Q was earlier on in the alphabet.

以我有限的大脑容量，Q在字母表中更靠前总是让我感到惊讶。

What is it?

它是什么？

OPQ?

OPQ？

I can't even actually localize letters in the alphabet.

我甚至无法在字母表中定位字母。

I'm sure that's the case for a lot of people without reading the alphabet in my head sequentially.

我相信很多人都是这样，除非我在脑子里按顺序读字母表。

All of this has to do with short-term and long-term memory access,

所有这些都与短期和长期记忆的获取有关，

the functioning, the limitation of human cognition,

人类认知的运作和局限，

and maybe cognitive systems in general,

也许还有一般的认知系统，

all of it relevant to this particular episode,

所有这些都与这一集特别相关，

and not so relevant to the awesomeness of Quo, formerly known as OpenPhone,

而与Quo（原名OpenPhone）的精彩之处关系不大，

that I should be talking about.

那才是我应该谈论的。

Of course, as is always the case,

当然，一如既往，

I think the point here and at the point everywhere in the point of life

我认为这里的重点，以及生活中处处都是重点的重点，

is to talk from the heart about whatever you want.

就是发自内心地谈论任何你想谈的事。

And that's what I try to do with everything.

这就是我对待一切事情的方式。

And to generalize that even more, to talk whenever I want and to shut the F up

再进一步概括来说，就是想说的时候就说，不想说的时候就闭嘴，

whenever I want and listen.

想听的时候就听。

And I prefer that more often than I prefer to talk.

而且比起说话，我更常喜欢这样。

Insert clever transition here because talk is somehow relevant, it is.

在这里插入一个巧妙的过渡，因为谈话不知怎么地确实相关，确实如此。

So Quo, formerly known as OpenPhone, helps over 90,000 businesses manage phone calls, texts, contacts,

所以Quo，原名OpenPhone，帮助超过9万家企业管理电话、短信、联系人，

all kinds of phone-related stuff for a business.

以及各种与企业电话相关的事务。

You have a bunch of customers, a bunch of incoming calls,

你有一大堆客户，一大堆来电，

a bunch of people on the business side that have to answer those calls,

企业这边有一大堆人必须接听这些电话，

have to manage it, what's the status of this particular request,

必须管理这些，这个特定请求的状态是什么，

voicemails, transcripts, all that kind of stuff,

语音留言、文字记录，诸如此类，

and obviously a really nice, effective utilization of AI to make that really efficient.

显然，这是对AI非常好、非常有效的利用，让这一切变得非常高效。

But really, what's really important for things like this is that the interface is good,

但真正重要的是，对于这类东西来说，界面要好，

that team collaboration is good, and Quo delivers on that.

团队协作要好，而Quo在这方面做到了。

Try Quo for free, plus get 20% off your first six months when you go to quo.com slash lex.

免费试用Quo，另外访问quo.com/lex，前六个月还可享受20%折扣。

That's Q-U-O dot com slash lex.

网址是Q-U-O点com斜杠lex。

Tell your friends about it, because it just might help them win at Scrabble.

告诉你的朋友们，因为这可能正好帮他们在拼字游戏中获胜。

Speaking of Scrabble, you usually want to play Scrabble on a table.

说到拼字游戏，你通常想在桌子上玩拼字游戏。

It's such a magical experience.

这是一种如此神奇的体验。

I just had a vision from a distant past of me sitting with a friend and playing Scrabble at a table.

我刚刚浮现出一个遥远过去的画面，我和一个朋友坐在桌边玩拼字游戏。

What is this life full of beautiful memories and that it's over too soon?

这是什么人生，充满了美好的回忆，却结束得太快？

Yeah. That melancholy feeling is beautiful, I think.

是的。我觉得那种忧郁的感觉很美。

Insert another clever transition, a la Mark Norman maybe, because the name of this next company is Uplift Desk.

插入另一个巧妙的过渡，也许像马克·诺曼那样，因为下一家公司的名字是Uplift Desk。

As I said. Okay, it's my go-to favorite office desk, and it's also the desk that I use for podcast furniture.

正如我所说。好的，这是我最喜欢、最常用的办公桌，也是我用来做播客家具的桌子。

I have, I already lost count, I have a lot of Uplift desks, standing desks in my place, everywhere.

我已经数不清了，我有很多Uplift桌子，站立式桌子，到处都是。

It's desks everywhere. I have a mattress on the floor and Uplift desks.

到处都是桌子。我有一张床垫放在地板上，还有Uplift桌子。

So I have a Linux box for robotics.

所以我有一台用于机器人技术的Linux主机。

I have a machine where I do a lot of the editing.

我有一台机器，我在上面做很多剪辑工作。

All of that is on a desk.

所有这些都放在桌子上。

I have the three tables for the podcast desk,

我有三张用于播客桌的桌子，

the very one you've seen over the past several years.

就是你过去几年一直看到的那一张。

That's all uplift desks.

那都是Uplift的桌子。

I usually don't put them in standing mode,

我通常不把它们调到站立模式，

but they are standing desks.

但它们是站立式桌子。

It allows me to do all kinds of stuff,

它让我能做各种事情，

really easy to work with,

真的很容易使用，

really nice material,

材质真的很好，

really sturdy.

真的很结实。

I just love everything about uplift desk.

我就是喜欢Uplift桌子的一切。

When they said they want to sponsor,

当他们说想要赞助时，

After I've been using them for many years,

在我用了它们很多年之后，

I lost my mind.

我简直疯了。

I love it when I've been in love with a company,

我喜欢当我爱上一家公司的时候，

in love with their product for such a long time,

爱上他们的产品这么长时间，

and I get to also sing them praises.

而且我还能赞美他们。

I mean, come on.

我是说，拜托。

What are you going to tell me next that FFMPEG wants to sponsor this podcast?

接下来你还要告诉我FFMPEG想赞助这个播客吗？

Another sort of open source project.

另一个开源项目。

It's not a company that I've been in love with.

这不是一个我一直热爱的公司。

Anyway, go to upliftdesk.com slash Lex and use code Lex to get four free accessories

总之，去 upliftdesk.com/lex 并使用代码 Lex 获得四个免费配件，

free same-day shipping,

免费当日送达，

free returns,

免费退货，

a 15-year warranty,

15年保修，

and an extra discount off your entire order.

以及整个订单的额外折扣。

That's upliftdesk.com.

那就是 upliftdesk.com。

The spelling it out really help anybody.

把它拼出来真的能帮到任何人。

I don't know,

我不知道，

but they really said pretty please.

但他们真的说了拜托了。

The one request is spell it out.

唯一的要求就是把它拼出来。

Again, what is this life?

再说一次，这是什么生活？

Incredible.

不可思议。

This episode is also brought to you by Finn, the number one AI agent for customer service.

本期节目同样由 Finn 赞助，它是客户服务领域的头号 AI 代理。

Find the niche and become number one.

找到细分市场，成为第一名。

That's the idea here.

这就是这里的理念。

Anybody building an AI company, and we talk about this, is the dream of AGI dead?

任何正在创建 AI 公司的人，我们谈论这个话题，AGI 的梦想死了吗？

I think for a lot of companies, success is in the niche,

我认为对很多公司来说，成功在于细分市场，

but there is a few, and Finn delivers on that niche.

但有一些公司，Finn 在那个细分市场做到了。

It's trusted by over 6,000 customer service leaders at top companies, including AI companies.

它受到超过 6,000 名顶级公司客户服务负责人的信任，包括 AI 公司。

When an AI company trusts your company to do its customer service, that means you're legit.

当一家 AI 公司信任你的公司来做它的客户服务时，这意味着你是靠谱的。

90-day money bag guarantee up to $1 million built to handle complex multi-step priorities like returns, exchanges, and disputes.

90 天资金保证，最高 100 万美元，专为处理复杂的多步骤优先事项而构建，如退货、换货和纠纷。

Go to fin.ai.lex to learn more about transforming your customer service and scaling your support team.

访问 fin.ai.lex 了解更多关于转变您的客户服务和扩展您的支持团队的信息。

That's fin.ai slash lex.

那就是 fin.ai/lex。

I don't know why I switched to this hyping voice.

我不知道为什么我切换到了这种夸张的声音。

Crappy announcer, crappy radio jockey, crappy ad read voice.

糟糕的播音员，糟糕的电台主持人，糟糕的广告朗读声音。

It is what it is.

事情就是这样。

Thank you for sticking with me this long.

感谢你陪我这么久。

I feel the love and I send it right back at you.

我感受到了爱，我也把爱回送给你。

This episode is also brought to you by a company whose engineers are also full of love, Shopify.

本期节目同样由一家工程师也充满爱的公司赞助，Shopify。

It just brings a smile to my face.

它只是让我脸上露出微笑。

Every time I think about Shopify, I got to see their engineering booth at NeurIPS, which is a machine learning conference.

每次我想到Shopify，我都会想起在NeurIPS看到他们的工程展台，那是一个机器学习会议。

Really brilliant people, wonderful people.

非常聪明的人，很棒的人。

Of course, the CEO, Toby, is still programming, still building stuff, still in on the details of the engineering,

当然，CEO Toby仍然在编程，仍然在构建东西，仍然深入工程的细节，

and now is talking quite a bit about utilization of LLMs for his own sort of pet projects, but also inside the company.

而现在他经常谈论LLM的利用，既用于他自己的个人项目，也用于公司内部。

It's just incredible when, from the very top, the company is in love with engineering.

当公司从最高层就热爱工程时，这真是令人难以置信。

It's a celebration of great engineering,

这是对伟大工程的庆祝，

Just like the conversation with DHH, who is the guy behind Ruby on Rails that Shopify was built on.

就像与DHH的对话一样，他是Shopify所基于的Ruby on Rails背后的人。

That conversation was a celebration of great engineering.

那次对话是对伟大工程的庆祝。

The beauty of engineering as well.

也是工程之美。

Anyway, listen to that episode to see some of the magic of Ruby on Rails and the magic of Shopify and the magic of Toby that we talk about.

总之，听听那一集，看看我们谈论的Ruby on Rails的魔力、Shopify的魔力以及Toby的魔力。

Anyway, sign up for a $1 per month trial period at shopify.com slash lux. That's all lowercase.

总之，在shopify.com/lux注册每月1美元的试用期。全部小写。

Go to shopify.com slash lex to take your business to the next level today.

访问shopify.com/lex，今天就让你的业务更上一层楼。

This episode is also brought to you by CodeRabbit, a platform that provides AI-powered code reviews directly within your terminal.

本期节目还由CodeRabbit赞助，这是一个直接在终端中提供AI驱动代码审查的平台。

We talk a lot in this episode about the timeline for the full automation of the human programmer.

我们在本期节目中大量讨论了人类程序员完全自动化的时间线。

I think we're quite far away from taking the human out of the loop.

我认为我们离把人类排除在循环之外还差得很远。

That review process, the debugging process, all of that, that's such a crucial part of programming,

那个审查过程、调试过程，所有这些，都是编程中极其关键的一部分，

especially just like we talk about in the episode.

尤其是就像我们在这一集里谈到的那样。

When we're not talking about a personal website where HTML slop is something that a web browser magically, automagically,

当我们不是在谈论一个个人网站，那里的HTML垃圾是网页浏览器神奇地、自动地

I don't know how they're possibly able to do such incredible job of rendering slop,

我不知道它们怎么可能把垃圾渲染得如此出色，

but a web browser is in fact able to render slop, including AI slop.

但网页浏览器实际上确实能够渲染垃圾，包括AI垃圾。

It just finds a way. So really the question is when you have production code, something that a lot of users are relying on,

它总能找到办法。所以真正的问题是，当你有生产代码，也就是很多用户依赖的东西时，

how do you review that code? How do you make sure you're catching errors?

你如何审查那段代码？你如何确保能捕捉到错误？

How are you making sure that you put a backstop to hallucinations and the logical errors that AI coding agents can generate.

你如何确保为幻觉以及AI编码代理可能产生的逻辑错误设置一道防线。

Anyway, CodeRabbit supports all programming languages.

总之，CodeRabbit支持所有编程语言。

Install CodeRabbit CLI today at coderabbit.ai.lex.

今天就在coderabbit.ai.lex安装CodeRabbit CLI。

That's coderabbit.ai.lex.

就是coderabbit.ai.lex。

This episode is also brought to you by Element,

本期节目还由Element赞助，

my daily zero-sugar and delicious electrolyte mix.

我每天喝的无糖又美味的电解质冲剂。

Reminds me of the fact that I need to get to editing the video of me in the jungle when

这让我想起，我得去剪辑我在丛林里的视频，当时

Paul Rosling and I, such an incredible human,

保罗·罗斯林和我，多么了不起的人，

congratulations to Paul on all of his success.

祝贺保罗取得的所有成功。

Go get his book. It's an incredible book.

去买他的书吧。那是一本了不起的书。

Again, he's an incredible person with an incredible mission.

再说一次，他是一个了不起的人，肩负着了不起的使命。

And yes, I need to edit and publish, hoping to at the very least.

是的，我需要编辑并发布，至少希望能做到这一点。

The story of our journey in the jungle,

我们在丛林中的旅程故事，

because it was a beautiful celebration of nature and the jungle and friendship and the full richness of the human experience.

因为那是对自然、丛林、友谊以及人类经验全部丰富性的一次美丽庆祝。

It was beautiful.

那很美。

The reason I mentioned that is I was, as part of that journey, severely dehydrated.

我提到这个的原因是在那段旅程中，我严重脱水。

And I remember dreaming of element of a cold drink of water with the electrolytes.

我记得我梦见元素，一杯带电解质的冷饮。

Your body craves it.

你的身体渴望它。

And it craves it because it needs it.

它渴望它，因为它需要它。

Electrolytes, sodium, potassium, magnesium.

电解质、钠、钾、镁。

When you're deprived, it's not just water, it's electrolytes.

当你缺乏时，不仅仅是水，还有电解质。

So anyway, I always remember that.

总之，我一直记得这一点。

Get a free eight-count sample pack with any purchase.

任何购买都可获得免费八支装样品包。

Try it at drinkelement.com slash lex.

请在 drinkelement.com/lex 试用。

This is the Lex Friedman Podcast.

这是莱克斯·弗里德曼播客。

To support it, please check out our sponsors in the description, where you can also find links to contact me, ask questions, get feedback, and so on.

为了支持它，请查看描述中的赞助商，在那里你还可以找到联系我、提问、获取反馈等的链接。

And now, dear friends, here's Sebastian Rashka and Nathan Lambert.

现在，亲爱的朋友们，有请塞巴斯蒂安·拉什卡和内森·兰伯特。

So I think one useful lens to look at all of this through is the DeepSeq, so-called DeepSeq moment.

所以我认为看待这一切的一个有用视角是 DeepSeq，所谓的 DeepSeq 时刻。

This happened about a year ago in January 2025

这大约发生在一年前，2025年1月

when the open-weight Chinese company DeepSeq released DeepSeq R1

当时中国开源权重公司DeepSeq发布了DeepSeq R1

that I think it's fair to say surprised everyone with near or at state-of-the-art performance

我认为可以说，它以接近甚至达到最先进的性能让所有人惊讶

with allegedly much less compute for much cheaper.

据称计算量少得多，成本也便宜得多。

and from then to today, the AI competition has gotten insane,

而从那时到今天，AI竞争已经变得疯狂，

both on the research level and the product level.

无论是在研究层面还是产品层面。

It's just been accelerating.

它一直在加速。

Let's discuss all of this today and maybe let's start with some spicy questions if we can.

我们今天来讨论这一切，也许我们可以从一些辛辣的问题开始。

Who is winning at the international level?

在国际层面上，谁在赢？

Would you say it's the set of companies in China or the set of companies in the United States?

你会说是中国的公司，还是美国的公司？

And Sebastian, Nathan, it's good to see you guys.

还有Sebastian、Nathan，很高兴见到你们。

So Sebastian, who do you think is winning?

那么Sebastian，你认为谁在赢？

So winning is a very broad term.

所以“赢”是一个非常宽泛的词。

I would say you mentioned the DeepSeek moment,

我想说，你提到了DeepSeek时刻，

and I do think DeepSeek is definitely winning the hearts of the people who work on open weight models

我确实认为DeepSeek绝对赢得了从事开源权重模型工作的人们的心

because they share these as open models.

因为他们把这些作为开放模型分享出来。

Winning, I think, has multiple timescales to it.

我认为，“赢”有多个时间尺度。

We have today, we have next year, we have in 10 years.

我们有今天，有明年，有10年后。

One thing I know for sure is that I don't think nowadays, 2026,

我确定知道的一件事是，我觉得如今，2026年，

that there will be any company who is, let's say, having access to a technology that no other company has access to.

会有任何一家公司，比如说，拥有其他公司无法获得的某项技术。

And that is mainly because researchers are frequently changing jobs, changing labs, they rotate.

这主要是因为研究人员经常换工作、换实验室，他们会轮换。

So I don't think there will be a clear winner in terms of technology access.

所以我认为在技术获取方面不会有明显的赢家。

However, I do think there will be the differentiating factor will be budget and hardware constraint.

不过，我确实认为差异化的因素将是预算和硬件限制。

So I don't think the ideas will be proprietary, but the way or the resources that are needed to implement them.

所以我认为想法不会是专有的，而是实现它们所需的方式或资源。

And so I don't see currently a take it all scenario where a winner takes it all.

所以目前我看不到一个赢家通吃的局面。

I can't see that at the moment.

目前我看不到这种情况。

Nathan, what do you think?

内森，你怎么看？

You see the labs put different energy into what they're trying to do.

你可以看到各个实验室在他们想做的事情上投入了不同的精力。

And I think to demarcate the point in time when we're recording this,

我想为了标明我们录制这个的时间点，

the hype over Anthropic Cloud Opus 4.5 model has been absolutely insane,

关于Anthropic Cloud Opus 4.5模型的炒作简直疯狂，

which is just I mean, I've used it and built stuff in the last few weeks.

我的意思是，我在过去几周里用过它并做了一些东西。

And it's it's almost gotten to the point where it feels like a bit of a meme in terms of the hype.

它几乎已经到了在炒作方面感觉有点像梗的地步。

And it's kind of funny because this is very organic.

这有点好笑，因为这是非常自然的。

And then if we go back a few months ago, we can get the release date in the notes as Gemini 3 from Google got released.

然后如果我们回到几个月前，我们可以在笔记里看到发布日期，是谷歌的 Gemini 3 发布了。

And it seemed like the marketing and just like wow factor of that release was super high.

而且那次发布的营销和那种哇塞效应似乎超级高。

But then at the end of November, Claude Opus 4.5 was released and the hype has been growing.

但到了11月底，Claude Opus 4.5 发布了，热度一直在增长。

But Gemini 3 was before this.

但 Gemini 3 是在这之前。

And it kind of feels like people don't really talk about it as much, even though when it came out, everybody was like, this is Gemini's moment to retake kind of Google's structural advantages in AI.

而且感觉人们并没有那么频繁地谈论它，尽管它刚出来时，大家都说，这是 Gemini 重新夺回谷歌在 AI 领域结构性优势的时刻。

And Gemini 3 is a fantastic model and I still use it.

而且 Gemini 3 是一个很棒的模型，我还在用它。

It's with sebastian what you're saying with all these like the idea space is very fluid but um culturally anthropic is known for betting very hard on code which is cloud code thing is working out for them right now so i think that even if the ideas flow pretty freely so much of this is bottlenecked by human effort and kind of culture of organizations where anthropic seems to at least be presenting as the least chaotic it's a bit of an advantage and if they can keep doing that for a while but

塞巴斯蒂安，你说的这些，就像整个创意空间非常流动，但嗯，在文化上 Anthropic 以在代码上押重注而闻名，也就是 Cloud Code 这件事现在对他们很有效，所以我认为即使想法流动得相当自由，这其中很多都受限于人力以及组织文化，而 Anthropic 似乎至少表现得是最不混乱的，这算是一个优势，如果他们能再保持一段时间，但

On the other side of things, there's a lot of ominous technology from China,

另一方面，中国有很多令人不安的技术，

where there's way more labs than DeepSeek.

那里的实验室远不止DeepSeek一家。

So DeepSeek kicked off a movement within China.

所以DeepSeek在中国国内掀起了一场运动。

I'd say kind of similar to how ChatGPT kicked off a movement in the U.S.,

我会说这有点像ChatGPT在美国掀起的那场运动，

where everything had a chatbot.

那时所有东西都配了个聊天机器人。

There's now tons of tech companies in China that are releasing very strong frontier open weight models,

现在中国有大量科技公司正在发布非常强大的前沿开放权重模型，

to the point where I would say that DeepSeek is kind of losing its crown

以至于我会说DeepSeek正在失去它的王冠，

as the preeminent open model maker in China, and the likes of Z.ai's with their GLM models, Minimax's models, Kimi Moonshot, especially in the last few months, have shown more brightly.

作为中国最杰出的开放模型制造商，而像Z.ai的GLM模型、Minimax的模型、Kimi月之暗面，尤其是最近几个月，表现得更加亮眼。

The new DeepSeek models are still very strong,

新的DeepSeek模型仍然非常强大，

but that's kind of a, it could look back as a big narrative point

但这有点像，回头看可能是一个重大的叙事节点，

where in 2025 DeepSeek came and then all, and it kind of provided this platform for way more Chinese companies that are releasing these fantastic models to kind of have this new type of operation.

在2025年DeepSeek出现之后，它某种程度上为更多中国公司提供了一个平台，让它们发布这些出色的模型，从而拥有这种新型的运营模式。

So these models from these Chinese companies are open weights.

所以这些中国公司的模型是开放权重的。

And depending on the trajectory of business models that these American companies are doing could be at risk.

而取决于这些美国公司所采用的商业模式的走向，它们可能会面临风险。

But currently, a lot of people are paying for AI software in the U.S. and historically in China and other parts of the world,

但目前，很多人在美国为AI软件付费，而历史上在中国和世界其他地区，

people don't pay a lot for software.

人们不会为软件付很多钱。

So some of these models like DeepSeek have the love of the people because they are open weight.

所以像DeepSeek这样的一些模型受到人们的喜爱，因为它们是开放权重的。

How long do you think the Chinese companies keep releasing open weight models?

你认为中国公司会继续发布开放权重模型多久？

I would say for a few years.

我会说几年吧。

I think that like in the US, there's not a clear business model for it.

我认为像在美国，它没有清晰的商业模式。

I have been writing about open models for a while and these Chinese companies have realized it.

我写关于开放模型的文章有一段时间了，这些中国公司已经意识到了这一点。

So I get inbound from some of them.

所以我收到他们中一些人的联系。

And they're smart and realize the same constraints, which is that a lot of US tech companies and other IT companies won't pay for an API subscription to Chinese companies for security concerns.

他们很聪明，也意识到同样的限制，那就是很多美国科技公司和其他IT公司出于安全考虑不会向中国公司支付API订阅费用。

This has been a longstanding habit in tech.

这在科技行业一直是个长期习惯。

And the people at these companies then see open weight models as an ability to influence and take part of a huge growing AI expenditure market in the US.

这些公司的人随后将开放权重模型视为一种影响和参与美国巨大增长的AI支出市场的能力。

And they're very realistic about this.

他们对此非常现实。

And it's working for them.

而且这对他们有效。

And I think that the government will see that that is building a lot of influence internationally in terms of uptake of the technology.

而且我认为政府会看到，这在技术采用方面正在国际上建立很大的影响力。

So there's going to be a lot of incentives to keep it going.

所以会有很多激励措施让它继续下去。

But building these models and doing the research is very expensive.

但构建这些模型和做研究非常昂贵。

So at some point I expect consolidation,

所以在某个时候我预计会整合，

but I don't expect that to be a story of 2026,

但我不认为那会是2026年的故事，

where there will be more open model builders throughout 2026 than there were in 2025.

到那时2026年全年的开源模型构建者会比2025年更多。

And a lot of the notable ones will be in China.

而且很多值得注意的会来自中国。

You were going to say something?

你刚才想说什么？

Yes.

是的。

You mentioned DeepSeq losing its crown.

你提到DeepSeq失去了它的王冠。

I do think to some extent, yes.

我确实认为在某种程度上是这样。

But we also have to consider, though, they are still, I would say, slightly ahead.

但我们也必须考虑到，他们仍然，我会说，稍微领先。

And the other ones, it's not that DeepSeq got worse.

而其他的，并不是DeepSeq变差了。

It's just like the other ones are using the ideas from ZipSeq.

只是其他模型在借鉴ZipSeq的想法。

For example, you mentioned Kimi, same architecture.

例如，你提到的Kimi，相同的架构。

They're training it.

他们正在训练它。

And then again, we have this leapfrogging where they might be at some point in time a bit better because they have the more recent model.

然后我们又有这种跳跃式发展，他们可能在某个时间点稍微更好，因为他们有更新的模型。

And I think this comes back to the fact that there won't be a clear winner.

我认为这又回到了一个事实：不会有明确的赢家。

It will just be like that.

就会像那样。

One person releases something, the other one comes in.

一个人发布一些东西，另一个人就来了。

And the most recent model is probably always the best model.

而最新的模型可能总是最好的模型。

Yeah.

是的。

We'll also see that Chinese companies have different incentives.

我们还会看到中国公司有不同的激励措施。

So like DeepSeq is very secretive,

所以像DeepSeq非常神秘，

where some of these startups are like the Minimaxes and Z.AIs of the world.

这些初创公司就像世界上的MiniMax和Z.AI一样。

Those two literally have filed IPO paperwork and they're trying to get Western mindshare and do a lot of outreach there.

那两家确实已经提交了IPO文件，他们正试图在西方赢得关注度，并在那里做大量推广。

So I don't know if these incentives will kind of change the model development

所以我不知道这些激励措施是否会改变模型开发，

because DeepSeq famously is built by a hedge fund, High Flyer Capital.

因为DeepSeq众所周知是由一家对冲基金High Flyer Capital建立的。

And we don't know exactly what we don't know what they use the models for or if they care about this.

我们并不确切知道我们不知道他们用这些模型做什么，或者他们是否在意这个。

They're secretive in terms of communication.

他们在沟通方面很神秘。

They're not secretive in terms of the technical reports that describe how their models work.

但在描述其模型工作原理的技术报告方面，他们并不神秘。

They're still open on that front.

他们在那一方面仍然开放。

And we should also say on the Opus 4.5 hype,

我们还应该说说关于Opus 4.5的炒作，

there's the layer of something being the darling of the X echo chamber on Twitter echo chamber

有一层是某样东西成为X上回音室或Twitter回音室的宠儿，

and the actual amount of people that are using the model.

以及实际使用该模型的人数。

I think it's probably fair to say that JGPT and Gemini are focused on the broad user base

我认为可以说JGPT和Gemini专注于广泛的用户群，

that just want to solve problems in their daily lives.

他们只想解决日常生活中的问题。

And that user base is gigantic.

而这个用户群是巨大的。

So the hype about the coding may not be represented by the actual use.

所以关于编程的炒作可能并不代表实际使用情况。

I would say also a lot of the usage patterns are, like you said, name recognition, brand and stuff,

我想说，很多使用模式，就像你说的，是名称认知、品牌之类的，

but also muscle memory almost, where, you know, like JGPD has been around for a long time.

但几乎也是肌肉记忆，你知道，就像JGPD已经存在很长时间了。

People just got used to using it and it's kind of like almost like a flywheel.

人们只是习惯了使用它，它几乎就像一个飞轮。

they recommend it to other users and that stuff.

他们把它推荐给其他用户之类的。

One interesting point is also the customization of LLMs.

另一个有趣的点是LLM的定制化。

For example, ChatGPT has a memory feature, right?

例如，ChatGPT有一个记忆功能，对吧？

And so you may have a subscription and you use it for personal stuff,

所以你可能有一个订阅，你用它来处理个人事务，

but I don't know if you want to use that same thing at work, you know, because it's a boundary between private and work.

但我不知道你是否想在工作中使用同样的东西，你知道，因为这是私人和工作之间的界限。

If you're working at a company, they might not allow that or you may not want that.

如果你在一家公司工作，他们可能不允许那样，或者你可能不想要那样。

And I think that's also an interesting point where you might have multiple subscriptions.

我认为这也是一个有趣的点，你可能拥有多个订阅。

One is just clean code. It has nothing of your personal images or hobby projects in there.

一个只是干净的代码。里面没有你的个人图像或爱好项目。

It's just like the work thing.

它就像工作用的东西。

And then the other one is your personal thing.

然后另一个是你的个人东西。

So I think that's also something where two different use cases, and it doesn't mean you only have to have one.

所以我认为这也是两个不同用例的情况，这并不意味着你只能有一个。

I think the future is also multiple ones.

我认为未来也是多个。

What model do you think won 2025?

你认为哪个模型赢得了2025年？

And what model do you think is going to win 26?

你认为哪个模型将赢得26年？

I think in the context of a consumer chatbots is a question of,

我认为在消费者聊天机器人的背景下，这是一个问题，

are you willing to bet on Gemini over ChatGPT?

你愿意押注Gemini而不是ChatGPT吗？

which I would say in my gut feels like a bit of a risky bet

我会说，我的直觉感觉这有点冒险

because open AI has been the incumbent and there's so many benefits to that in tech.

因为OpenAI一直是现任者，在科技领域这有很多好处。

I think that momentum if you look at 2025 was on gemini side

我认为如果你看2025年，势头是在Gemini这边

but they were starting from such a low point

但他们是从如此低的起点开始的

i think on rip bard and these earlier attempts of getting started

我认为在RIP Bard和这些早期的尝试中

i think huge credit for them for powering through the organizational chaos to make that happen

我认为他们非常值得称赞，能够克服组织混乱实现这一目标

but also it's hard to bet against chat to open ai

但同样很难押注ChatGPT会输给OpenAI

because they always come off cast as so chaotic

因为他们总是显得如此混乱

but they're very good at landing things

但他们非常擅长把事情做成

And I think like personally, I have very mixed reviews of GPT-5,

而且我认为就个人而言，我对GPT-5的评价非常复杂，

but it had to have saved them so much money with the headline feature being a router where most users are no longer charging their GPU costs as much.

但它一定为他们节省了很多钱，主要功能是一个路由器，大多数用户不再像以前那样承担那么多GPU成本。

So I think it's very hard to dissociate the things that I like out of models versus the things that are going to actually be a general public differentiator.

所以我认为很难将我喜欢模型的地方与真正会成为大众差异化因素的地方分开。

What do you think about 2026? Who's going to win?

你觉得2026年怎么样？谁会赢？

I'll say something, even though it's risky.

我要说一些话，尽管这有风险。

I will say that I think Gemini will continue to take progress on ChatGPT.

我想说，我认为Gemini会在ChatGPT的基础上继续取得进展。

I think Google scale when both of these are operating at such extreme scales.

我认为当这两者都以如此极端的规模运作时，谷歌的规模优势就体现出来了。

And Google has the ability to separate that research and product a bit better.

而谷歌有能力把研究和产品稍微更好地分开。

We hear so much about OpenAI being chaotic operationally and chasing the high impact thing, which is a very startup culture.

我们听到很多关于OpenAI在运营上混乱、追逐高影响力的事情，这是一种非常创业公司的文化。

And then on the software and enterprise side, I think Anthropic will have continued to success as they've again and again been set up for that.

然后在软件和企业方面，我认为Anthropic会继续取得成功，因为他们一次又一次地为此做好了准备。

And obviously Google's cloud has a lot of offerings, but I think this kind of like Gemini name brand is important for them to build.

显然谷歌云有很多产品，但我认为这种Gemini品牌对他们来说很重要，需要去建立。

And Google's cloud will continue to do well, but that's kind of a more complex thing to explain in the ecosystem

谷歌云会继续表现良好，但这是在生态系统中更复杂的事情，需要解释

because that's competing with the likes of Azure and AWS rather than on the model provider side.

因为那是与Azure和AWS等竞争，而不是在模型提供商方面。

Well, infrastructure, you think GPUs give an advantage?

那么，基础设施方面，你认为GPU能带来优势吗？

Largely because the margin on NVIDIA chips is insane.

主要是因为NVIDIA芯片的利润率非常高。

And Google can develop everything from top to bottom to fit their stack and not have to pay this margin.

而谷歌可以从上到下开发一切来适配他们的技术栈，而不必支付这个利润。

And they've had a head start in building data centers.

而且他们在建设数据中心方面有先发优势。

So all of these things that have both high lead times and very hard margins on high costs,

所以所有这些既交货周期长、又要在高成本上承受极高利润压力的东西，

Google has just kind of a historical advantage there.

谷歌在这方面恰好拥有一种历史性的优势。

And if there's going to be a new paradigm, it's most likely to come from OpenAI,

而如果会出现一种新范式，它最有可能来自 OpenAI，

where their research division again and again has kind of shown this ability to land a new research idea or a product.

因为他们的研究部门一次又一次地展现出这种能力：能够落地一个新的研究想法或产品。

I think deep research, Sora, O1 thinking models, all these definitional things have come from OpenAI,

我认为深度研究、Sora、O1 思维模型，所有这些具有定义意义的东西都来自 OpenAI，

and that's got to be one of their top traits as an organization.

而这一定是他们作为一个组织的最大特质之一。

So it's kind of hard to bet against that,

所以很难去赌他们不行，

but I think a lot of this year will be about scale and optimizing what could be described as low-hanging fruit in models.

但我认为今年很大程度上将围绕规模扩展，以及优化模型中可被称为“低垂果实”的部分。

And clearly there's a trade-off between intelligence and speed.

而且显然，智能和速度之间存在权衡。

This is what ChatGPT 5 was trying to solve behind the scenes.

这就是 ChatGPT 5 在幕后试图解决的问题。

It's like, do people actually want intelligence, the broad public, or do they want speed?

就像是在问，大众真的想要智能，还是想要速度？

I think it's a nice variety, actually, or the option to have a toggle there.

其实我觉得提供多样性，或者说有一个可以切换的选项，是件好事。

I mean, first, for my personal usage, most of the time when I look something up, I use ChatGPT to ask a quick question, get the information, I want it fast.

我的意思是，首先，就我个人使用而言，大多数时候我查东西时，会用 ChatGPT 问个快速问题、获取信息，我希望它快。

For, you know, most daily tasks, I use the quick model.

对于大多数日常任务，我会用快速模型。

Nowadays, I think the auto mode is pretty good where you don't have to specifically say thinking or, you know, non-thinking and stuff.

如今，我觉得自动模式相当不错，你不需要特意说“思考”或者“非思考”之类的。

Then again, I also sometimes want the pro mode.

不过话说回来，我有时也想要专业模式。

Very often what I do is when I have something written, I put it into a chatubity and say, hey, do a very thorough check.

很多时候我会这样做：当我写好东西后，我把它放进一个聊天工具里，说，嘿，做一个非常彻底的检查。

Are all my references correct? Are all my thoughts correct? Did I make any formatting mistakes? And are the figure numbers wrong or something like that?

我所有的引用都正确吗？我所有的想法都正确吗？我有没有犯任何格式错误？还有图表编号是不是错了之类的？

And I don't need that right away. It's something, okay, I finish my stuff, maybe have dinner, let it run, come back, and it goes through this.

而且我并不需要马上得到结果。就是那种，好吧，我做完我的事，也许去吃个晚饭，让它跑着，回来之后，它就把这些过一遍。

And I think, see, this is where I think it's important to have this option.

我觉得，你看，这就是我认为拥有这个选项很重要的地方。

I would go crazy if for each query I would have to wait 30 minutes or 10 minutes.

如果每次查询我都得等30分钟或10分钟，我会疯掉的。

That's me. I'm like sitting over here, losing my mind, that you use the router and the non-thinking model.

这就是我。我就像坐在这儿，快疯了，你居然用路由器和非思考模型。

I'm like, how do you live with that? It's like my reaction.

我就像在说，你怎么受得了？这就是我的反应。

I've been heavily on Chat to BT for a while.

我有一阵子一直在重度使用Chat to BT。

Never touched five non-thinking.

从没碰过5的非思考模式。

I find its tone and then its propensity of errors.

我发现它的语气，还有它容易出错的倾向。

It's just like it has a higher likelihood of errors.

就好像它出错的概率更高。

Some of this is from back when opening, I released 03, which was the first model to do this deep search and find many sources and integrate them for you.

其中一部分来自当初发布的时候，我发布了03，那是第一个能做这种深度搜索、找到很多来源并为你整合的模型。

So I became habituated with that.

所以我就习惯了那样。

So I will only use GPT 5.2 thinking or pro

所以我只会用 GPT 5.2 的思考模式或专业模式

when I'm finding any sort of information query for work,

当我查找任何与工作相关的信息查询时，

whether that's a paper or some code reference that I found.

无论是论文还是我找到的某些代码参考。

And it's just like, I will regularly have like five pro queries going simultaneously,

就像这样，我通常会同时运行五个专业查询，

each looking for one specific paper or feedback on an equation or something.

每个查询都在寻找一篇特定的论文，或者关于某个方程的反馈之类的。

I have a fun example where I just needed to answer as fast as possible.

我有一个有趣的例子，当时我需要尽快给出答案。

For this podcast, before I was going on the trip,

为了这个播客，在我出发旅行之前，

I have a local GPU running at home and I wanted to run a long RL experiment.

我家里有一台本地 GPU 在运行，我想跑一个长时间的强化学习实验。

And usually I also unplug things because

通常我也会拔掉一些东西，因为

you never know if you're not at home, you don't want to have things plugged in.

你永远不知道，如果你不在家，你不想让东西一直插着电。

And I accidentally unplugged the GPU.

我不小心拔掉了 GPU 的电源。

It was like my wife was already in the car and it's like oh dang

就像我妻子已经在车里了，然后就像，哦糟糕

and then basically i wanted as fast as possible a bash script that runs my different uh experiments and evaluation and

然后基本上我想要尽快写一个 bash 脚本，用来运行我不同的呃实验和评估，然后

i did something i know i learned how to use the bash uh interface or bash terminal

我做了一些我知道的事情，我学会了如何使用 bash 呃界面或 bash 终端

but in that moment i just needed like 10 seconds

但在那一刻，我只需要大约 10 秒钟

give me the command this is a hilarious situation but yes

给我命令，这情况太搞笑了，但没错

what did you use so i did the non-thinking fastest model it gave me

你用了什么，所以我用了非思考模式的最快模型，它给了我

The bash command I to chain different scripts to each other and then

那个 bash 命令，我把不同的脚本串联起来，然后

the thing is like you have the t thing where you want to route this to a log file top of my head

问题是，就像你有那个 t 命令，你想把这个路由到一个日志文件，我脑子里首先想到的

I was just like in a hurry I could have thought about it myself by the way

我当时就是很匆忙，顺便说一句，我本可以自己想想的

I don't know if there's a representative case why you wait in the car you have to run you unplug the GPU you have to generate a bash script

我不知道是否有代表性案例，为什么你要在车里等，你得跑，你拔掉 GPU，你得生成一个 bash 脚本

This sounds like a movie like in which possible I use Gemini for that

这听起来像一部电影，就像可能我用 Gemini 来做那个

So I use thinking for all the information stuff and then Gemini for fast things or stuff that I could sometimes Google

所以对于所有信息类的东西，我用 thinking，然后对于快速的事情，或者我有时可以谷歌的东西，用 Gemini

which is like it's good at explaining things and I trust that it has this kind of background of knowledge and it's simple and the Gemini app has got a lot better and it's good for that sort of things

它就像，它擅长解释事情，我相信它有这种知识背景，而且它很简单，Gemini 应用已经好多了，它适合那种事情

and then for code and any sort of philosophical discussion I use Claude Opus 4.5 also always with extended thinking extended thinking and inference time scaling

然后对于代码和任何类型的哲学讨论，我用 Claude Opus 4.5，也总是用扩展思考、扩展思考和推理时间缩放

is just a way to make the models marginally smarter and I will always edge on that side when the progress is very high

只是一种让模型稍微更聪明的方法，当进展非常快时，我总是会偏向那一边

because you don't know when that'll unlock a new use case

因为你不知道什么时候那会解锁一个新的使用场景

and then sometimes use grok for real-time information

然后有时候用Grok获取实时信息

or finding something on AI Twitter

或者在AI推特上找东西

that I knew I saw and I need to dig up and I just fixated on

我知道我见过，我需要挖出来，而我只是盯着

although when Grok 4 came out, the Grok 4, what is super heavy

尽管当Grok 4出来时，Grok 4，什么是超重型

which was like their pro variant was actually very good

那就像是他们的专业版，实际上非常好

and I was pretty impressed with it

我对它印象相当深刻

and I just kind of like muscle memory lost track of it

而我只是有点肌肉记忆，跟丢了它

with having the chat to BT app open

因为一直开着聊天到BT应用

so I use many different things

所以我用很多不同的东西

yeah I actually do use Grok 4 heavy for debugging

是的，我实际上确实用Grok 4重型来调试

for like hardcore debugging

用于像硬核调试

and the other ones can't solve it I find that it's the best

而其他的解决不了，我发现它是最好的

and I it's interesting because you say Chad GPT is the best interface for me for that same reason

而且我，有趣的是，你说Chad GPT对我来说是最好的界面，出于同样的原因

but this could be just momentum

但这可能只是惯性

Gemini is the better interface for me I think

我认为Gemini对我来说是更好的界面

because I fell in love with their best needle in the haystack

因为我爱上了他们最好的大海捞针

if I ever put something that has a lot of context

如果我曾经放入一些有很多上下文的东西

but I'm looking for very specific kinds of information

但我正在寻找非常具体的信息

make sure it tracks all of it

确保它追踪所有内容

I find at least the Gemini for me has been the best.

我发现至少Gemini对我来说一直是最好的。

So it's funny with some of these models, if they win your heart over for one particular feature on a one particular day for that particular query, that prompt,

所以有趣的是，对于其中一些模型，如果它们在某一天、针对某个特定的查询、那个提示词，用某个特定功能赢得了你的心，

you're like, this model is better.

你就会觉得，这个模型更好。

And so you'll just stick with it for a bit until it does something really dumb.

于是你就会继续用它一阵子，直到它做出一些非常愚蠢的事。

There's like a threshold effect, some smart thing, and then you fall in love with it.

就像有一种阈值效应，它做了些聪明的事，然后你就爱上它了。

And then it does some dumb thing.

然后它又做了些愚蠢的事。

And you're like, you know what? I'm going to switch and try clawed and JGBT and all that kind of stuff.

然后你会说，你知道吗？我要换一个，试试Clawed和JGBT之类的。

This is exactly like you use it until it breaks, until you have a problem, and then you change the LM.

这就像你用一样东西，直到它坏掉，直到你遇到问题，然后你就换掉那个语言模型。

And I think it's the same how we use anything, like our favorite text editor, operating systems, or the browser.

我觉得我们使用任何东西都是这样，比如我们最喜欢的文本编辑器、操作系统，或者浏览器。

I mean, there are so many browser options, Safari, Firefox, Chrome, all the, kind of relatively similar,

我的意思是，浏览器选项太多了，Safari、Firefox、Chrome，都差不多，

but then there are edge cases, maybe extensions you want to use, and then you switch.

但然后会有一些边缘情况，也许是你想用的扩展，然后你就换了。

But I don't think there is any one who types the same thing, like the website, into different browsers and compares them.

但我觉得没有人会把同样的东西，比如网站，输入到不同的浏览器里然后比较它们。

You only do that when the website doesn't render, if something breaks, I think.

我觉得你只有在网站无法渲染、出问题的时候才会那么做。

So that's a good point. I think you use it until it breaks, and then you explore other options, I think.

所以这是个好观点。我觉得你用一样东西直到它坏掉，然后你再探索其他选项，我觉得是这样。

On the long context thing, I was also a Gemini user for this.

关于长上下文这件事，我在这方面也是Gemini的用户。

But the GPT-5.2 release blog had crazy long context scores

但GPT-5.2的发布博客有疯狂的长上下文分数

where a lot of people were like, did they just figure out some algorithmic change?

很多人都在想，他们是不是只是搞出了某种算法上的改变？

It went from 30% to 70% or something in this minor model update.

在这个小版本模型更新中，它从30%左右涨到了70%左右。

So it's also very hard to keep track of all of these things.

所以也很难跟踪所有这些事情。

But now I look more favorably at GPT-5.2's long context.

但现在我更看好GPT-5.2的长上下文。

So it's just kind of like, how do I actually get to testing this never-ending battle?

所以这就像，我到底要怎么去测试这场永无止境的战斗？

It's interesting that none of us talked about the Chinese models from a user usage perspective.

有趣的是，我们没有人从用户使用角度谈论中国模型。

What does that say? Does that mean the Chinese models are not as good?

这说明什么？是不是意味着中国模型没那么好？

Or does that mean we're just very biased and U.S.-focused?

还是说我们只是非常偏颇、只关注美国？

I do think that that's currently the discrepancy between just the model and the platform.

我确实认为，这就是目前模型和平台之间的差异。

So I think the open models, they are more known for the open weights, not the platform yet.

所以我认为，开源模型更出名的是开放权重，而不是平台。

There are also a lot of companies that are willing to sell you the open model inference at a very low cost.

也有很多公司愿意以非常低的成本向你出售开源模型的推理服务。

I think like Open Router, it's easy to do the look at multi-model things.

我觉得像Open Router，很容易查看多模型的东西。

You can run DeepSeq on perplexity.

你可以在Perplexity上运行DeepSeq。

I think all of us sitting here are like, we use OpenAI GPT-5 Pro consistently.

我觉得我们坐在这里的所有人都一样，我们一直使用OpenAI GPT-5 Pro。

We're all willing to pay for the marginal intelligence gain.

我们都愿意为那一点边际智能提升付费。

And anyone that's like these models from the U.S. are better.

而任何认为这些来自美国的模型更好的人。

And in terms of the outputs, I think that the question is, will they stay better for this year and for years going?

就输出而言，我认为问题是，它们在今年以及未来几年会保持更好吗？

But it's like so long as they're better, I'm going to pay for it to use them.

但就像只要它们更好，我就会付费使用它们。

I think there's also analysis that shows that like the.

我认为也有分析表明，就像那个。

way that the chinese models are served this

中国模型被服务的方式，这

you could argue due to expert controls or not is that they use fewer gpus for replica which makes them slower and have different errors and it's like speed and intelligence if these things are in your favor as a user i think in the u.s a lot of users will go for this and i think that that is something that will spur these chinese companies to want to compete in other ways whether it's like free or substantially lower costs or it'll breed creativity in terms of offerings, which is good for the ecosystem.

你可以争辩说，由于专家控制与否，它们为副本使用更少的GPU，这使它们更慢，并且有不同的错误，这就像速度和智能，如果这些对你作为用户有利，我认为在美国很多用户会倾向于这个，我认为这会促使这些中国公司想以其他方式竞争，无论是免费还是大幅降低成本，或者会在产品方面激发创造力，这对生态系统有好处。

But I just think the simple thing is the U.S. models are currently better

但我只是认为简单的事实是，美国模型目前更好

and we use them and I try these other open models and I'm like, fun, but I don't go back to it.

我们使用它们，我尝试这些其他开源模型，我觉得，有趣，但我不会回去用它。

We didn't really mention programming. That's another use case that a lot of people deeply care about.

我们真的没有提到编程。那是另一个很多人深切关心的用例。

So I use basically half and half cursor and clog code

所以我基本上是一半一半地用 Cursor 和 Clog Code

because I find them to be fundamentally different experience and both useful.

因为我发现它们本质上是不同的体验，而且都很有用。

What do you guys, you program quite a bit.

你们呢，你们编程挺多的。

So what do you use?

那你们用什么？

What's the current vibe?

现在流行什么？

So I use the Codex plugin for VS Code.

所以我用 VS Code 的 Codex 插件。

You know, it's very convenient. It's just like a plugin.

你知道，它非常方便。就像一个插件。

And then it's a chat interface that has access to your repository.

然后它是一个可以访问你代码库的聊天界面。

I know that Cloud Code is, I think, a bit different.

我知道 Cloud Code 我觉得有点不同。

It's a bit more agentic. It touches more things. It does a whole project for you.

它更自主一些。它涉及更多东西。它为你完成整个项目。

I'm not quite there yet where I'm comfortable with that because maybe I'm a control freak,

我还没到能接受那种方式的程度，因为也许我是个控制狂，

but I still would like to see a bit what's going on.

但我还是想稍微看看发生了什么。

Codex is kind of like right now for me, like the sweet spot where it is helping me,

Codex 现在对我来说有点像最佳平衡点，它在帮助我，

but it is not taking completely over.

但它并没有完全接管。

I should mention one of the reasons I do use Cloud Code is to build the skill of programming with English.

我应该提一下，我使用 Cloud Code 的原因之一是为了培养用英语编程的技能。

I mean, the experience is fundamentally different.

我的意思是，体验是根本不同的。

You're, as opposed to micromanaging the details of the process of the generation of the code

你是，而不是去微观管理代码生成过程的细节

and looking at the diff, which you can in cursor, if that's the idea you use,

然后查看差异，如果你用的是 Cursor 的话，你可以在里面看差异，

and in changing, altering, looking and reading the code and understanding the code deeply as you progress

并且在修改、改动、查看和阅读代码的过程中，随着进展深入理解代码

versus just kind of like thinking in this design space and just guiding it at this macro level,

而不是只是在这个设计空间里思考，只是在这个宏观层面上引导它，

which I think is another way of thinking about the programming process.

我认为这是思考编程过程的另一种方式。

Also, we should say that Cloud Code, it just seems to be somehow a better utilization of Cloud Opus 4.5.

另外，我们应该说，Cloud Code 似乎以某种方式更好地利用了 Cloud Opus 4.5。

It's a good side-by-side for people to do.

这是一个很适合人们并排对比使用的工具。

So you can have cloud code open, you can have cursor open, you can have VS code open, and you can select the same models on all of them and ask questions.

所以你可以打开 Cloud Code，可以打开 Cursor，可以打开 VS Code，然后在它们上面选择相同的模型并提问。

It's very interesting.

这非常有意思。

Like the cloud code is way better in that domain.

就像 Cloud Code 在那个领域要好得多。

It's remarkable.

这很了不起。

All right. We should say that both of you are legit on multiple fronts.

好的。我们应该说，你们两位在多个方面都很厉害。

Researchers, programmers, educators, tweeterers, and on the book front, too.

研究人员、程序员、教育者、推特用户，在写书方面也是如此。

So Nathan, at some point soon, hopefully has an RLHF book coming out.

所以 Nathan 希望不久之后能出一本关于 RLHF 的书。

It's available for pre-order and there's a full digital pre-print

它可以预订，还有完整的数字预印本

just making it pretty and better organized for the physical thing,

只是为了让实体书更精美、更有条理，

which is a lot of why I do it because it's fun to create things that you think are excellent in the physical form

这也是我这么做的很大原因，因为创造你认为在实体形式上很出色的东西很有趣

when so much of our life is digital.

当我们的生活如此数字化时。

I should say, going to perplexity here, Sebastian Roshka is a machine learning researcher and author known for several influential books.

我应该提一下，来到Perplexity这里，Sebastian Roshka是一位机器学习研究员和作家，以几本有影响力的书闻名。

A couple of them that I wanted to mention, which is a book I highly recommend, Build a large language model from scratch and the new one build a reasoning model from scratch

我想提几本，其中一本我强烈推荐，《从零构建大型语言模型》，还有新书《从零构建推理模型》

so i'm really excited about that building stuff from scratch

所以我对从零构建东西感到非常兴奋

is one of the most powerful ways of learning honestly

老实说，这是最强大的学习方式之一

building an element from scratch is a lot of fun it's also a lot of to learn and like you said it's probably the best way to learn how something really works

从零构建一个元素很有趣，也能学到很多，就像你说的，这可能是了解某事物真正运作方式的最佳途径

because you can look at figures but figures can have mistakes

因为你可以看图表，但图表可能有错误

you can look at concepts explanations but you might misunderstand them.

你可以看概念解释，但你可能误解它们。

But if you see there is code and the code works, you know it's correct.

但如果你看到有代码，而且代码能运行，你就知道它是正确的。

I mean, there's no misunderstanding. It's like, it's precise. Otherwise it wouldn't work.

我的意思是，没有误解。就像，它是精确的。否则它就不会运行。

And I think that's like kind of like the beauty behind coding.

我觉得这就像是编程背后的美妙之处。

It is kind of like, it doesn't lie.

它有点像是，它不会说谎。

It's math basically.

基本上就是数学。

So even though with math, I think you can have mistakes in a book you would never notice because you're not running the math when you are reading the book.

所以即使有数学，我觉得书里可能有你永远不会注意到的错误，因为你读书的时候并没有在运行数学。

You can't verify this.

你没法验证这一点。

And with code, what's nice is you can verify it.

而有了代码，好处就是你可以验证它。

Yeah, I agree with you about the LM from scratch book,

是的，我同意你关于《从零构建语言模型》这本书的看法，

it's nice to tune out everything else, the internet and so on, and just focus on the book.

能屏蔽掉其他一切，比如互联网之类的，只专注于这本书，这很好。

But, you know, I read several like, you know, history books.

但是，你知道，我读过好几本，你知道，历史书。

It's just less lonely somehow.

不知怎的，就是没那么孤独。

It's really more fun.

真的更有趣。

Like, for example, on the programming front, I think it's genuinely more fun to program with an LLM.

比如，在编程方面，我觉得用大语言模型编程真的更有趣。

And I think it's genuinely more fun to read with an LLM.

而且我觉得用大语言模型阅读真的更有趣。

But you're right, like this distraction should be minimized.

但你说得对，这种分心应该尽量减少。

So you use the LLM to basically enrich the experience, maybe add more context.

所以你用大语言模型来丰富体验，也许增加更多背景信息。

Maybe the rate of aha moments for me in a small scale is really high with LLMs.

也许在小规模上，我获得顿悟时刻的频率真的很高，有了大语言模型。

100%. I also want to correct myself.

百分之百。我也想纠正一下自己。

I'm not suggesting not to use LLMs.

我不是建议不要使用大语言模型。

I suggest doing it in multiple passes, like one pass just offline focus mode.

我建议分多次进行，比如一次只是离线专注模式。

And then after that, I mean, I also take notes,

然后在那之后，我是说，我也会做笔记，

but I try to resist the urge to immediately look things up.

但我尽量克制住立刻去查东西的冲动。

I do a second pass. It's just like for me more structured this way.

我会做第二遍。对我来说这样只是更有条理。

And I get, I mean, sometimes things are answered in the chapter,

而且我明白，我是说，有时候有些东西在章节里就有答案，

but sometimes also it just helps to let it sink in and think about it.

但有时候，让它慢慢沉淀下来、去思考一下，也很有帮助。

Other people have different preferences. I would highly recommend using LLMs when reading books.

其他人有不同的偏好。我非常推荐在读书时使用大语言模型。

For me, it's just, it's not the first thing to do. It's like the second pass.

对我来说，它只是，它不是第一件要做的事。它像是第二遍。

My way of recommendation is to say, I do the opposite.

我推荐的方式是说，我做的正好相反。

I like to use the LLM at the beginning to lay out the full context of like, what is this world that I'm now stepping into?

我喜欢在一开始就用大语言模型来铺开完整的背景，比如，我现在踏入的这个世界是什么？

But I try to avoid clicking out of the LLM into the world of like Twitter and blogs.

但我尽量不点出大语言模型，进入像推特和博客那样的世界。

And because then you're now down this rabbit hole, you're reading somebody's opinion.

因为那样你就掉进了这个兔子洞，你在读某个人的观点。

There's a flame war about a particular topic.

有一个关于某个特定话题的激烈骂战。

And all of a sudden you're no longer, you're now in the realm of the Internet and Reddit and so on.

然后突然之间你就不再是，你现在进入了互联网和Reddit等等的领域。

But if you're purely letting the LLM give you the context of why this matters, what are the big picture ideas?

但如果你纯粹让大语言模型给你提供背景，说明为什么这很重要，大局观是什么？

But sometimes books themselves are good at doing that, but not always.

但有时候书本身就擅长做这件事，但并非总是如此。

That's why I like the ChatGPT app.

这就是为什么我喜欢ChatGPT应用。

It gives the AI a home in your computer when you can focus on it rather than just being another tab in my mess of Internet options.

它给AI在你的电脑里安了个家，让你能专注于它，而不是只把它当成我那一堆乱七八糟的互联网选项里的又一个标签页。

And I think Cloud Code in particular does a good job of making that a joy, where it seems very engaging as a product.

而且我觉得Cloud Code在这方面做得特别好，让它变得很愉快，作为一个产品它看起来非常吸引人。

designed to be an interface that your ai will then go out into the world and it's something that is very kind of intangible between it and codex is that it just feels kind of warm and engaging

它被设计成一个界面，让你的AI随后走向世界，而它和Codex之间有种很微妙的东西，就是它感觉起来有点温暖、很吸引人。

where codex can often be as good from open ai but it just kind of like feels a little bit rougher on the edges

而Codex虽然来自OpenAI，往往也一样好，但它就是感觉边缘有点粗糙。

whereas cloud code is makes it fun to build things particularly from scratch

而Cloud Code则让构建东西变得有趣，尤其是从零开始。

where you just don't like you don't have to care but you trust that it'll make something like obviously good for websites and kind of refreshing tooling and stuff like this which i use it for data analysis

在这里你完全不用操心，但你可以相信它会做出一些显然不错的东西，比如网站和那种让人耳目一新的工具之类的，我就是用它来做数据分析的。

so i my blog we scrape hugging face we keep the download numbers for every data set and model over time now

所以我的博客会抓取Hugging Face的数据，我们持续记录每个数据集和模型随时间变化的下载量。

so we have them and it's like claude was

所以我们有这些数据，就像Claude当时……

Just like, yeah, I've made use of that data.

就像，是的，我利用了那些数据。

No problem. And I was like, that would have taken me days.

没问题。然后我想，那本来会花我好几天。

And I was like, then I have enough situational awareness to be like, okay, these trends obviously make sense.

然后我想，那我有足够的情境意识去说，好吧，这些趋势显然合理。

And you can check things because that's just a kind of wonderful interface

而且你可以查看东西，因为那只是一个很棒的界面

where you can have an intermediary and not have to do the kind of awful low-level work

在那里你可以有一个中介，而不必做那种糟糕的低级工作

that you would have to do to maintain different web projects and do this stuff all right

你为了维护不同的网络项目并做这些事情而不得不做的工作，好吧

So we just talked about a bunch of the closed weight models.

所以我们刚刚谈了一堆闭源模型。

Let's talk about the open ones. Uh, so tell me about the landscape of open LM models,

我们来谈谈开源的。呃，那么告诉我开源语言模型的情况，

which are interesting ones, which stand out to you and why. We already mentioned DeepSeek.

哪些是有趣的，哪些对你来说很突出，为什么。我们已经提到了DeepSeek。

Do you want to see how many we can name off the top of our head? Yeah, without looking at notes.

你想看看我们能不假思索地说出多少个吗？是的，不看笔记。

DeepSeek, Kimi, MiniMax, Z.ai, Antling, we're just going Chinese. Um, let's throw in Mistral AI, Gemma, um, yeah, GPT-OSS.

DeepSeek、Kimi、MiniMax、Z.ai、Antling，我们只是在说中国的。嗯，让我们加上Mistral AI、Gemma，嗯，是的，GPT-OSS。

The open source model by Jet GPT, actually Nvidia Nemotron had a, Nvidia had a really cool one, Nemotron 3.

Jet GPT的开源模型，实际上Nvidia Nemotron有一个，Nvidia有一个非常酷的，Nemotron 3。

Um, there's a lot of stuff especially at the end of the year.

嗯，有很多东西，尤其是在年底。

Quen? One, maybe the one... Oh yeah, Quen was the obvious name.

Quen？一，也许是那个……哦对，Quen 是那个显而易见的名字。

I was trying to get through that you can get at least 10 Chinese and at least 10 Western.

我当时想表达的是，你至少能列出10个中国的和至少10个西方的。

I think that, I mean, opening... I released their first open model since GPT-2.

我觉得，我是说，开放……我发布了他们自 GPT-2 以来的第一个开放模型。

That was when I, when I meant to talk when I was writing about opening eyes, open model release.

那是在我，当我想谈的时候，我当时在写关于开放视野、开放模型发布的内容。

They're all like, don't forget about GPT-2, which I thought was really funny.

他们都在说，别忘了 GPT-2，我觉得这真的很好笑。

Because it's just such a different time.

因为那完全是一个不同的时代。

But GPT-OSS is actually a very strong model.

但 GPT-OSS 实际上是一个非常强大的模型。

And does some things that the other models don't do very well.

而且它能做一些其他模型做不好的事情。

And I think that, selfishly, I'll promote a bunch of like Western companies.

而且我觉得，出于私心，我会推广一堆像西方公司这样的。

So both in the U.S., in Europe, have these like fully open models.

所以美国和欧洲都有这些完全开放的模型。

So I work at Allen Institute for AI. We've been building ULMO, which releases data and code and all of this.

所以我在艾伦人工智能研究所工作。我们一直在构建 ULMO，它发布数据和代码以及所有这些。

And now we have actual competition for people that are trying to release everything.

而现在，对于那些试图发布一切的人来说，我们有了真正的竞争。

So that other people can train these models.

这样其他人就能训练这些模型。

So there's the Institute for Foundation Models, or /LM 360.

所以有基础模型研究所，或者叫 /LM 360。

Which is like had their K2 models of various types.

它就像是有各种类型的 K2 模型。

APRODIS is a Swiss research consortium.

APRODIS 是一个瑞士研究联盟。

Hugging Face has Small LM, which is very popular.

Hugging Face 有 Small LM，非常受欢迎。

where NVIDIA's Neymatron has started releasing data as well.

英伟达的Neymatron也开始发布数据了。

And then Stanford's Marin Community Project, which is kind of making it so there's a pipeline for people to open a GitHub issue and implement a new idea and then have it run in a stable language modeling stack.

然后是斯坦福的Marin社区项目，它某种程度上是在打造一条流程，让人们可以提交GitHub issue、实现一个新想法，然后让它在稳定的语言建模技术栈上运行。

So this space... That list was way smaller in 2024.

所以这个领域……那份名单在2024年要小得多。

So I think it was like just AI2.

所以我觉得当时基本上就只有AI2。

So that's a great thing for more people to get involved in to understand language models,

所以这是件好事，能让更多人参与进来、理解语言模型，

which doesn't really have a Chinese company that has an analog.

而这一点其实并没有哪家中国公司能与之对应。

While I'm talking, I'll say that the Chinese open language models tend to be much bigger,

趁我讲的时候，我要说中国的开源语言模型往往要大得多，

and that gives them this higher peak performance as MOEs,

这让它们作为MoE模型能达到更高的峰值性能，

where a lot of these things that we like a lot, whether it was Gemma and Nemetron,

而我们很喜欢的很多东西，无论是Gemma还是Nemetron，

tended to be smaller models from the U.S., which is starting to change from the U.S. and Europe.

往往都是美国的小模型，而这一点正开始从美国和欧洲发生变化。

Mistral Large 3 came out, which was a giant MOE model, very similar to DeepSeek architecture in December.

Mistral Large 3发布了，那是个巨大的MoE模型，架构和12月的DeepSeek非常相似。

And then a startup, RCAI and both Nemetron and NVIDIA have teased MOE models of way bigger than 100 billion parameters, like this 400 billion parameter range coming in this Q1 2026 timeline.

然后还有一家初创公司RCAI，而且Nemetron和英伟达都预告了远超1000亿参数的MoE模型，比如这个4000亿参数级别的模型，会在2026年第一季度这个时间线推出。

So I think this kind of balance is set to change this year in terms of

所以我认为这种平衡今年将会发生变化，就

what people are using the Chinese versus U.S. open models for,

人们使用中国与美国开源模型的用途而言，

which I'm personally going to be very excited to watch.

这一点我个人会非常期待看到。

First of all, huge props for being able to name so many of these.

首先，能说出这么多模型的名字，真是了不起。

Did you actually name Llama?

Llama 真的是你命名的吗？

No. I feel like RIP.

不是。我感觉像是安息吧。

This was not on purpose.

这不是故意的。

RIP Llama.

安息吧，Llama。

All right. Can you mention what are some interesting models that stand out

好的。你能说说有哪些有趣的模型比较突出吗？

so your mission quen 3s is obviously a standout

所以你们的 Mission Quen 3s 显然很突出，

so I would say the years almost book ended by both DeepSeek version 3 and R1

所以我会说这一年几乎被 DeepSeek 的 V3 和 R1 首尾呼应，

and then on the other hand in December DeepSeek version 3.2

然后另一方面，在十二月 DeepSeek 3.2 版本

because what I like about those is they always have an interesting architecture tweak that others don't have

因为我喜欢它们的地方在于，它们总是有一些别人没有的有趣架构调整，

but otherwise if you want to go with um you know like the familiar but really good performance

但除此之外，如果你想用那种你熟悉的、但性能非常好的模型，

quen 3 and like um Nathan said also GPT OSS and I think GPT OSS

Quen 3，还有像 Nathan 说的 GPT OSS，而且我觉得 GPT OSS

what's interesting about it is kind of like the first public or like open weight model that was really trained with tool use in mind

它有趣的地方在于，它算是第一个真正以工具使用为核心训练的开源权重模型，

which I do think is kind of a bit of a paradigm shift where the ecosystem

我确实认为这算是一种范式转变，整个生态系统

was not quite ready for it, so with tool use, I mean,

还没完全准备好，所以说到工具使用，我的意思是，

that the LLM is able to do a web search, to call a Python interpreter, and I do

大语言模型能够进行网络搜索、调用Python解释器，而我确实

think this, it's a standout, because I think it's a huge unlock,

认为这一点很突出，因为我觉得这是一个巨大的突破，

because one of the most common complaints about LLMs are, for example, hallucinations, right?

因为关于大语言模型最常见的抱怨之一就是，比如说，幻觉，对吧？

And so in my opinion, one of the best ways to solve hallucinations is to not try to always remember information or make things up.

所以在我看来，解决幻觉最好的方法之一就是不要总是试图记住信息或者编造内容。

For math, why not use a calculator app or Python?

对于数学，为什么不用计算器应用或者Python呢？

If I asked the LLM who won the soccer World Cup in 1998, instead of just trying to memorize, it could go do a search.

如果我问大语言模型1998年谁赢得了足球世界杯，与其只是试图记住，它可以去搜索一下。

I think mostly it's usually still Google search.

我觉得大多数情况下通常还是谷歌搜索。

So GPT, GPT OSS, they would do a tool call to Google, maybe find the FIFA website, find, okay, it was France.

所以GPT，GPT OSS，它们会调用工具去谷歌，也许找到国际足联网站，找到，好的，是法国。

It would get you that information reliably instead of just trying to memorize it.

它会可靠地给你那个信息，而不是只是试图记住它。

So I think it's a huge unlock, which I think right now is not fully utilized yet by the open source, open-weight ecosystem.

所以我觉得这是一个巨大的突破，而我认为目前开源、开放权重生态系统还没有充分利用它。

A lot of people don't use tool call modes because I think it's first, it's a trust thing.

很多人不使用工具调用模式，因为我觉得首先这是一个信任问题。

You don't want to run this on your computer

你不想在你的电脑上运行这个

where it has access to tools could wipe your hard drive or whatever

因为它能访问工具，可能会擦除你的硬盘之类的

so you want to maybe containerize that

所以你可能想把它容器化

but I do think, you know, that that is like a really important step for the upcoming years to have this ability

但我确实认为，你知道，那真的是未来几年拥有这种能力的重要一步

so uh a few quick things first of all

所以呃，首先几件快事

thank you for defining what you mean by tool

谢谢你定义了你所说的工具是什么意思

use I think that's a great thing to do in general for the concepts we're talking about

使用，我认为总的来说，对于我们正在谈论的概念，这是一件很棒的事情

even things as sort of well established as moes

即使是像 moes 这样已经相当成熟的东西

uh you have to say that means mixture of experts and you kind of have to build up an intuition for people,

呃，你必须说那意味着专家混合，而且你有点必须为人们建立一种直觉，

what that means, how it's actually utilized, what are the different flavors.

那意味着什么，它实际上是如何被利用的，有哪些不同的风味。

So what does it mean that there's just such explosion of open models?

那么，开放模型如此爆炸式增长意味着什么？

What's your intuition? If you're releasing an open model, you want people to use it as the first and foremost thing.

你的直觉是什么？如果你发布一个开放模型，你希望人们首先使用它。

And then after that comes things like transparency and trust.

然后在那之后才是透明度和信任之类的事情。

I think when you look at China, the biggest reason is that they want people around the world to use these models.

我认为当你看看中国，最大的原因是他们希望世界各地的人们使用这些模型。

And I think a lot of people will not, if you look outside of the U.S.,

而且我认为，如果你看看美国以外的地方，很多人不会，

a lot of people will not pay for software,

很多人不会为软件付费，

but they might have computing resources where you can put a model on it and run it.

但他们可能有计算资源，可以把模型放上去运行。

I think there can also be data that you don't want to send to the cloud.

我认为也可能有些数据你不想发送到云端。

So the number one thing is getting people to use models,

所以第一件事就是让人们使用模型，

use AI or use your AI that might not be able to do it without having access to the model.

使用AI，或者使用你的AI，而如果没有访问模型的权限，可能就无法做到。

I guess we should state explicitly.

我想我们应该明确说明。

So we've been talking about these Chinese models and open weight models.

所以我们一直在讨论这些中国模型和开放权重模型。

Oftentimes the way they're run is locally.

通常它们的运行方式是本地运行。

So it's not like you're sending your data to China or to whoever developed, to Silicon Valley, whoever developed the model.

所以并不是说你在把数据发送到中国，或者发送给开发者，发送到硅谷，发送给开发这个模型的人。

A lot of American startups make money by hosting these models from China and selling them, selling tokens.

很多美国初创公司通过托管这些来自中国的模型并出售它们、出售token来赚钱。

It's called like selling tokens, which means somebody will call the model to do some piece of work.

这叫做出售token，意思是有人会调用模型来完成某项工作。

I think the other reason is for U.S. companies, opening AI is so GPU deprived.

我认为另一个原因是对美国公司来说，开放AI非常缺乏GPU。

They're at the limits of the GPUs.

他们已经达到了GPU的极限。

Whenever they make a release, they're always talking about, like, our GPUs are hurting.

每当他们发布新版本时，他们总是说，我们的GPU很吃紧。

And I think there's, like, in one of these, like, GPT OSS release sessions,

而且我觉得，就像，在这些GPT OSS发布活动中，

Sam Altman said, like, oh, we're releasing this because we can use your GPUs.

Sam Altman说，就像，哦，我们发布这个是因为我们可以用你们的GPU。

We don't have to use our GPUs.

我们不必用我们自己的GPU。

And OpenAI can still get distribution out of this, which is another very real thing.

而OpenAI仍然可以从中获得分发，这是另一个非常真实的事情。

It doesn't cost them, though, anything.

不过，这并不花费他们任何东西。

And for the user, I think also, I mean, there are users who just use the model locally

而对于用户来说，我觉得也是，我的意思是，有些用户只是在本地使用模型，

how they would use GPD.

就像他们会用GPD那样。

but also for companies, I think it's a huge unlock to have these models

但对企业来说，我认为拥有这些模型是一个巨大的解锁，

because you can customize them, you can train them, you can add post-training, add more data, specialize them into, let's say, law, medical models, whatever you have.

因为你可以定制它们，你可以训练它们，你可以添加后训练，添加更多数据，将它们专门化为，比如说，法律、医疗模型，无论你有什么。

And the appeal, you mentioned LAMA,

而吸引力，你提到了LAMA，

the appeal of the open-weight models from China is that the open-weight models are also, the licenses are even friendlier.

来自中国的开放权重模型的吸引力在于，这些开放权重模型也是，许可证甚至更友好。

I think they are just unrestricted open-source licenses where if we use something like LAMA or JAMA, there are some strings attached.

我认为它们只是无限制的开源许可证，而如果我们使用像LAMA或JAMA这样的东西，就会有一些附加条件。

I think it's like upper limit in terms of how many users you have

我觉得这就像是你拥有多少用户的上限

and then if you exceed, I don't know, so

然后如果你超过了，我不知道，所以

and so many million users you have to report your finance situation to

而且你有那么多百万用户，你就必须向……报告你的财务状况

let's say matter or something like that

比如说Matter之类的

and I think well it is a free model

然后我觉得，嗯，它确实是个免费模型

but there are strings attached

但是是有附加条件的

and people do like things where strings are not attached

而人们确实喜欢没有附加条件的东西

so I think that's also one of the reasons besides performance

所以我觉得这也是除了性能之外的其中一个原因

why the open weight models from China are so popular

为什么来自中国的开放权重模型这么受欢迎

because you you can just use them there's no catch in that sense.

因为你可以直接使用它们，从这个意义上说没有任何陷阱。

The ecosystem has gotten better on that front,

在这方面生态系统已经变得更好了，

but mostly downstream of these new providers providing such open licenses.

但主要是这些提供此类开放许可证的新供应商的下游。

That was funny when you pulled up Perplexity and said Kimi K2 Thinking hosted in the US,

当你打开Perplexity并说Kimi K2 Thinking托管在美国时，那很有趣，

which is just like an exact, I've never seen this, but it's an exact example of what we're talking about where people are sensitive to this.

这就像是一个确切的，我从未见过这个，但它是我们正在谈论的人们对此敏感的一个确切例子。

Like Kimi K2 Thinking and Kimi K2 is a model that is very popular.

就像Kimi K2 Thinking和Kimi K2是一个非常受欢迎的模型。

People say that has very good creative writing and also in doing some software things.

人们说它在创意写作以及做一些软件方面非常出色。

It's just these little quirks that people pick up on with different models that they like.

这只是人们在不同模型上发现的、他们喜欢的一些小怪癖。

What are some interesting ideas that some of these models have explored that you can speak to, like that particularly interesting to you?

这些模型探索过哪些有趣的思路，是你可以谈谈的，比如你觉得特别有意思的？

Maybe you can go chronologically.

也许你可以按时间顺序来讲。

I mean, there was, of course, DeepSeq R1 that came out in January if we just focus on 2025.

我是说，当然有DeepSeq R1，如果只聚焦2025年的话，它是一月发布的。

However, this was based on DeepSeq version 3, which came out the year before in December 2024.

然而，这是基于DeepSeq版本3，它在前一年2024年12月发布。

There are multiple things on the architecture side.

在架构方面有多个东西。

What is fascinating is you can still, I mean, that's what I do in my from scratch coding projects.

有趣的是，你仍然可以，我是说，这就是我在从零开始编码项目里做的事。

you can still start with GPT-2 and you can add things to that model to make it into this other model.

你仍然可以从GPT-2开始，然后你可以给那个模型添加东西，把它变成另一个模型。

So it's all still kind of like the same lineage, the same, it is a very close relationship between those.

所以它仍然有点像同一个谱系，同样的，它们之间关系非常紧密。

But top of my head, DeepSeq, what was unique there is the mixture of experts.

但凭记忆，DeepSeq，独特之处在于专家混合。

I mean, they were not inventing mixture of experts.

我是说，他们并没有发明专家混合。

We can maybe talk a bit more what mixture of experts means.

我们也许可以多谈谈专家混合是什么意思。

But just to list these things first, before we dive into detail, mixture of experts,

但先列出这些东西，在我们深入细节之前，专家混合，

But then they also had multi-head latent attention, which is a tweak to the attention mechanism, where this was, I would say, 2025, the main distinguishing factor between these open weight models.

但然后他们还有多头潜在注意力，这是对注意力机制的一个调整，我会说，这是2025年这些开放权重模型之间的主要区别因素。

Different tweaks to make inference or KV cache size.

不同的调整，用来改变推理或KV缓存大小。

We can also define KV cache in a few moments,

我们稍后也可以定义一下KV缓存，

but to kind of make it more economical, to have long contacts, to shrink the KV cache size.

但为了让它更经济，能处理长上下文，来缩小KV缓存大小。

So what are tweaks that we can do?

那么我们可以做哪些调整呢？

And most of them focused on the attention mechanism.

其中大多数都集中在注意力机制上。

There is multi-hat latent attention in DeepSeq.

DeepSeq中有多头潜在注意力。

There is group query attention, which is still very popular.

还有分组查询注意力，它仍然非常流行。

It's not invented by any of those models.

它不是由那些模型中的任何一个发明的。

It goes back a few years, but that would be the other option.

它可以追溯到几年前，但那会是另一个选项。

Sliding window attention, I think, almost reuses it, if I remember correctly.

滑动窗口注意力，我想，如果我没记错的话，几乎复用了它。

So there are these different tweaks that make the models different.

所以有这些不同的调整，让模型变得不同。

Otherwise, I put them all together in an article once where I just compared them.

否则，我曾经把它们都放在一篇文章里，我只是比较了它们。

are very surprisingly similar it's just different numbers in terms of how many repetitions of the transformer block you have in the center

非常惊人地相似，只是在中心有多少次transformer块重复方面数字不同

and like just little little knobs that people tune but what what's so nice about it is it's it works no matter what you can tweak things you can

还有人们调整的那些小旋钮，但它好就好在，无论你怎么调整，它都能工作

move the normalization layers around you get some performance gains and i almost always very good in ablation studies

移动归一化层的位置，你会获得一些性能提升，而且我几乎总是在消融研究中表现很好

showing what actually what it does to the model if you move something around ablation studies doesn't

展示如果你移动某些东西，实际上对模型有什么影响，消融研究并不

make it better or worse,

让它变得更好或更糟，

but there are so many

但有很多

let's say ways you can implement a transformer and make it still work.

比如说，你可以用很多方式实现一个Transformer并让它仍然有效。

Big ideas that are still prevalent is mixture of experts,

仍然流行的大思想是专家混合，

multi-latent attention, sliding window attention, group query attention,

多潜在注意力、滑动窗口注意力、分组查询注意力，

and then at the end of the year we saw a focus on making the attention mechanism scale linearly with inference token prediction.

然后年底我们看到一个重点，就是让注意力机制随推理token预测线性扩展。

So there were quen3next for example, which added a gated deltanet.

所以有比如quen3next，它加入了一个门控deltanet。

It's kind of like inspired by state-based models,

它有点像受基于状态的模型启发，

where you have a fixed state that you keep updating,

你有一个固定状态并不断更新，

but it makes essentially this attention cheaper, or it replaces attention with a cheaper operation.

但它本质上让这种注意力更便宜，或者用更便宜的操作替代注意力。

And it maybe is it useful to step back and talk about transform architecture in general?

也许退一步谈谈Transformer架构总体上有用吗？

Yeah so maybe we should start with the GPT-2 architecture, the transformer that was derived from the Attention Is All You Need paper.

是的，所以也许我们应该从GPT-2架构开始，这个Transformer源自《Attention Is All You Need》论文。

So the Attention Is All You Need paper had a transformer architecture that had two parts, an encoder and a decoder,

所以《Attention Is All You Need》论文有一个Transformer架构，它有两部分：编码器和解码器，

and GPT went just focusing in on the decoder part.

而GPT只专注于解码器部分。

It is essentially still a neural network and it has this attention mechanism inside,

它本质上仍然是一个神经网络，并且内部有这个注意力机制，

and you predict one token at a time, you pass it through an

你一次预测一个token，你把它通过一个

embedding layer. There's the transformer block.

嵌入层。然后是 Transformer 模块。

The transformer block has attention modules and a fully connected layer, and there are some normalization layers in between.

Transformer 模块包含注意力模块和一个全连接层，中间还有一些归一化层。

But it's essentially neural network layers with this attention mechanism.

但它本质上就是带有这种注意力机制的神经网络层。

So coming from GPT-2, when we move on to GPT-OSS,

所以从 GPT-2 到 GPT-OSS，

there is, for example, the mixture of experts layer.

比如说，有混合专家层。

It's not invented by GPT-OSS. It's a few years old.

它不是 GPT-OSS 发明的。它已经有好几年历史了。

But it is essentially a tweak to make the model larger without consuming more compute in each forward pass.

但它本质上是一种调整，让模型变得更大，而不会在每次前向传播中消耗更多算力。

So there is this fully connected layer.

所以有一个这样的全连接层。

And if listeners are familiar with multilayer perceptrons, you can think of a mini multilayer perceptron, a fully connected neural network layer inside the transformer.

如果听众熟悉多层感知机，你可以把它想象成 Transformer 内部的一个小型多层感知机，也就是一个全连接神经网络层。

And it's very expensive because it's fully connected.

而且它非常昂贵，因为它是全连接的。

If you have a thousand inputs, a thousand outputs, it's like one million connections.

如果你有一千个输入、一千个输出，那就相当于一百万个连接。

And it's a very expensive part in this transformer.

它是这个 Transformer 中非常昂贵的一部分。

And the idea is to kind of expand that into multiple feedforward networks.

而思路就是把它扩展成多个前馈网络。

So instead of having one, let's say you have 256, but it would make it way more expensive

所以与其只有一个，假设你有 256 个，但那会让它昂贵得多，

because now you have 256, but you don't use all of them at the same time.

因为现在你有 256 个，但你不会同时使用全部。

So you now have a router that says, okay, based on this input token, it would be useful to use this fully connected network.

所以你现在有一个路由器，它会说，好的，基于这个输入词元，使用这个全连接网络会很有用。

And in that context, it's called an expert.

而在那种语境下，它被称为专家。

So a mixture of experts means you have multiple experts.

所以混合专家意味着你有多个专家。

And depending on what your input is, let's say it's more math heavy, it would use different experts compared to, let's say, translating input text from English to Spanish.

而根据你的输入是什么，比如说它更偏重数学，它会使用不同的专家，相比于，比如说，将输入文本从英语翻译成西班牙语。

It would maybe consult different experts.

它可能会咨询不同的专家。

It's not quite clear. I mean, it's clear cut to say, OK, this is only an expert for math and for Spanish is a bit more fuzzy.

这不太清楚。我的意思是，明确地说，好吧，这只是一个数学专家，而对于西班牙语则有点模糊。

But the idea is essentially that you pack more knowledge into the network, but not all the knowledge is used all the time.

但本质上，这个想法是你将更多知识打包进网络，但并非所有知识都会一直被使用。

That would be very wasteful.

那会非常浪费。

So you're kind of like during the token generation, you're more selective.

所以你在词元生成过程中，有点像是更有选择性地使用。

There's a router that selects which tokens should go to which expert.

有一个路由器会选择哪些词元应该去哪个专家。

It's more complexity. It's harder to train.

这更复杂。也更难训练。

There's a lot of, you know, that can go wrong, like collapse and everything.

有很多，你知道，可能出错的地方，比如崩溃之类的。

So I think that's why Olmo 3 still uses dense.

所以我认为这就是为什么Olmo 3仍然使用稠密模型。

I mean, you have, I think, Olmo models with mixture of experts, but dense models where dense means also it's jargon.

我的意思是，你有，我认为，Olmo模型有混合专家，但也有稠密模型，这里的稠密也是术语。

There's a distinction between dense and sparse.

稠密和稀疏之间有一个区别。

So a mixture of experts is considered sparse because we have a lot of experts, but only few of them are active.

所以专家混合被认为是稀疏的，因为我们有很多专家，但只有少数是活跃的。

So that's called sparse.

所以这被称为稀疏。

And then dense would be the opposite where you only have like one fully connected module and it's always, you know, utilized.

然后密集则是相反的，你只有一个全连接模块，而且它总是被使用。

So maybe this is a good place to also talk about KVCache, but actually before that, even zooming out, like fundamentally, how many new ideas have been implemented from GPT-2 to today?

所以也许这是一个好地方来谈谈KVCache，但实际上在那之前，甚至从更宏观的角度来看，从根本上说，从GPT-2到今天实施了多少新想法？

Like how different really are these architectures?

比如这些架构到底有多大不同？

picture like the mixture of experts um the attention mechanism in gpt oss that would be the group query attention mechanism so it's a slight tweak for multi-head attention to group query attention so that we have two

比如专家混合，嗯，GPT OSS中的注意力机制，那将是分组查询注意力机制，所以这是对多头注意力的一点小调整，变成分组查询注意力，这样我们有两个

i think they replaced a layer norm by rms norm but it's just like a different normalization layer and not a big change it's just like a tweak um the non-linear activation function people familiar in with deep new networks i mean it's the same as changing sigmoid with relu it's it's not changing the network fundamentally it's just like a tweak like a little little tweak um and that's about it i would say it's not really fundamentally that different it's still the same same architecture so you can convert one from one uh you can

我认为他们用RMS norm替换了层归一化，但这只是一个不同的归一化层，不是什么大变化，只是一个小调整。嗯，非线性激活函数，熟悉深度神经网络的人，我的意思是，这就像把sigmoid换成relu一样，它并没有从根本上改变网络，只是一个小调整，一点点小调整。嗯，差不多就是这样。我会说它并没有根本上的不同，仍然是相同的架构，所以你可以从一个转换到另一个，嗯，你可以

go from one into the other by just adding these, these changes, basically.

基本上，只要加上这些改动，就能从一个变成另一个。

It's fundamentally is still the same architecture.

它从根本上说还是同样的架构。

Yep. So for example, you mentioned my book earlier.

对。比如说，你之前提到过我的书。

That's a GPD2 model in the book because it's simple and it's very small, um, so 124, 120 million parameters approximately.

书里那个是GPD2模型，因为它简单而且非常小，嗯，大约1.24亿、1.2亿个参数。

But in the bonus materials, I do have almost three from-scratch Gemma 3, from-scratch, and other types of from-scratch models.

但在附加材料里，我确实有差不多三个从零开始训练的Gemma 3，从零开始，还有其他类型的从零开始训练的模型。

And I always started with my GPD2 model and just, you know, tweaked a few added different components.

而我一直都是从我的GPD2模型开始，然后就是，你知道，调整一下，加几个不同的组件。

And you get from one to the other, it's like, it's kind of like a lineage in a sense.

然后你就从一个得到了另一个，这有点像，某种意义上像是一条谱系。

Yeah, can you build up an intuition for people?

对，你能帮大家建立一种直觉吗？

Because sort of when you zoom out, you look at it, there's so much rapid advancement in the AI world.

因为当你拉远来看，你会发现，AI领域的进步是如此迅速。

At the same time, fundamentally, the architectures have not changed.

但与此同时，从根本上说，架构并没有改变。

So where is all the turbulence, the turmoil of the advancement happening?

那么，进步带来的所有这些动荡、混乱都发生在哪里呢？

Where, where's the gains to be had?

哪里，哪里能获得收益呢？

So there are the different stages where you develop the network or train the network.

所以有不同的阶段，你在这些阶段开发网络或训练网络。

You have the pre-training now, um, back then they, it was just pre-training with GPT2.

现在你有预训练，嗯，那时候他们，就只是用GPT2做预训练。

Now you have pre-training, mid-training, and post-training. Um, so I think right...

现在你有预训练、中期训练和后训练。嗯，所以我觉得，对……

Now we are in the post-training focus stage.

现在我们处于后训练重点阶段。

I mean, pre-training still gives you advantages if you scale it up to better, higher quality data,

我的意思是，预训练仍然会给你带来优势，如果你把它扩展到更好、更高质量的数据，

but then we have capability unlocks that were not there with GPT-2.

但随后我们有了GPT-2所没有的能力解锁。

For example, ChatGPT, it is basically a GPT-3 model,

例如，ChatGPT，它基本上是一个GPT-3模型，

and GPT-3 is the same as GPT-2 in terms of architecture.

而GPT-3在架构上与GPT-2相同。

What was new was adding the supervised fine-tuning and the reinforcement learning with human feedback.

新的是加入了监督微调和基于人类反馈的强化学习。

So it's more on the algorithmic side rather than the architecture

所以这更多是在算法方面，而不是架构方面

I would say that the systems also change a lot

我想说系统也变化很大

I think if you listen to Nvidia's announcements

我认为如果你听英伟达的公告

they talk about these things like you now do FP8, you can now do FP4

他们谈论这些事情，比如你现在可以做FP8，你现在可以做FP4

and what is happening is these labs are figuring out how to utilize more compute to put it into one model

正在发生的是，这些实验室正在弄清楚如何利用更多计算资源将其投入一个模型

which lets them train faster and that lets them put more data in and then you can find better configurations faster by doing this

这让他们训练得更快，也让他们能放入更多数据，然后通过这样做你可以更快地找到更好的配置

so you can look at like the essentially the tokens per second per GPU is a metric that you look at when you're doing large-scale training

所以你可以看，基本上每GPU每秒的token数是你进行大规模训练时关注的一个指标

and you could get you can go from like 10k to 13k by turning on FP8 training which

你可以从大约1万提升到1.3万，通过开启FP8训练，这

means they're using less memory per parameter in the model

意味着它们在模型中每个参数使用的内存更少

and by saving less information you do less communication you can train faster

而且通过保存更少的信息，你进行的通信更少，你可以训练得更快

so all these like system things underpin way faster experimentation on data and algorithms

所以所有这些系统层面的东西支撑着在数据和算法上快得多的实验

that is kind of like it's this it's this kind of loop that keeps going

这有点像，就是这种不断循环的过程

where it's kind of hard to describe when you look at the architecture

当你看着架构的时候，这有点难以描述

and they're exactly the same but the code base used to train these models is going to be vastly different

它们完全一样，但用来训练这些模型的代码库会大不相同

and you could probably like I don't the GPUs are different

而且你可能，我不知道，GPU 是不同的

but you probably train GPT-OSS 20B way faster in wall clock time than GPT-2 was trained at the time

但你可能训练 GPT-OSS 20B 的实际时间比当时训练 GPT-2 要快得多

yeah like you said they had for example in the mixture of experts

是的，就像你说的，他们在混合专家模型中有比如

this NVFP4 optimization for example where you get more throughput

比如这个 NVFP4 优化，你可以获得更高的吞吐量

but I do think this is for the speed this is true

但我确实认为这是为了速度，这是真的

but uh it doesn't give the model new capabilities in a sense

但呃，从某种意义上说，它并没有给模型带来新的能力

it's just how much can we make the computation coarser without suffering in terms of model performance degradation.

它只是我们能在不让模型性能下降的情况下把计算做得多粗糙。

But I do think, I mean, there are alternatives popping up to the transformer.

但我确实认为，我的意思是，正在出现一些替代 Transformer 的方案。

There's text diffusion models, completely different paradigm.

有文本扩散模型，完全不同的范式。

And there's also, I mean, text diffusion models might use transformer architectures,

而且还有，我是说，文本扩散模型可能会使用Transformer架构，

but it's not an auto-regressive transformer.

但它不是自回归Transformer。

And also Mamba models, it's a state-space model, but they do have trade-offs.

还有Mamba模型，它是一种状态空间模型，但它们确实有取舍。

And what's right is there's nothing that has replaced the autoregressive transformer as state-of-the-art model.

而正确的是，没有什么能取代自回归Transformer作为最先进的模型。

So like for state-of-the-art, you would still do that, go with that thing.

所以对于最先进的，你仍然会那样做，选择那个东西。

But there are no alternatives for the cheaper and like alternatives that are kind of making compromises.

但对于更便宜的替代方案，以及那些做出妥协的替代方案，并没有其他选择。

But it's not just one architecture anymore.

但现在已经不只有一种架构了。

There are little ones coming up.

有一些小的正在出现。

But if we talk about the state-of-the-art, it's pretty much still the transformer architecture autoregressive derived from GPT-2 essentially.

但如果我们谈论最先进的，基本上仍然是从GPT-2衍生出的自回归Transformer架构。

I guess the big question here is we talked quite a bit here on the architecture behind the pre-training.

我想这里的大问题是，我们在这里已经谈了很多关于预训练背后的架构。

Are the scaling laws holding strong across pre-training, post-training, inference, context size, data, synthetic data?

缩放定律在预训练、后训练、推理、上下文大小、数据、合成数据方面是否依然成立？

I like to start with the technical definition of scaling law, which kind of informs all of this.

我喜欢从缩放定律的技术定义开始，这在一定程度上为所有这些提供了信息。

The scaling law is a power law relationship between you can think of the x-axis.

缩放定律是一种幂律关系，你可以把它看作x轴。

So kind of what you are scaling is a combination of compute and data, which are kind of similar.

所以你所缩放的是计算和数据的组合，它们有点相似。

And then the y-axis is like the held out prediction accuracy over our next tokens.

然后y轴就像是对我们下一个token的留出预测准确率。

We talk about models being auto-regressive.

我们谈论模型是自回归的。

It's like if you keep a set of text that the model has not seen, how accurate will it get when you will train?

就像如果你保留一组模型没见过的文本，当你训练时它会有多准确？

And the idea of scaling laws came when people figured out that that was a very predictable relationship.

而缩放定律的概念出现于人们发现那是一种非常可预测的关系时。

And I think that that technical term is continuing.

我认为那个技术术语是持续性的。

And then the question is like, what do users get out of it?

然后问题就像是，用户能从中得到什么？

And then there are more types of scaling where OpenAI's 01 was famous for introducing inference time scaling.

然后还有更多类型的缩放，其中 OpenAI 的 01 因引入推理时间缩放而闻名。

And I think less famously for also showing that you can scale reinforcement learning training and get kind of this log X axis and then a linear increase in performance on Y axis.

我认为不太出名的是，它还展示了你可以扩展强化学习训练，并得到这种对数 X 轴，然后在 Y 轴上性能线性增长。

So there's kind of these three axes now where the traditional scaling laws are talked about for pre-training,

所以现在有这三种轴，其中传统的缩放定律是针对预训练的，

which is how big your model is and how big your data set is.

即你的模型有多大以及你的数据集有多大。

And then scaling reinforcement learning, which is like, how long can you do this trial and error learning that we will talk about?

然后是扩展强化学习，这就像，你能进行这种我们将要讨论的试错学习多久？

We'll define more of this.

我们会进一步定义这一点。

And then this inference time compute, which is just letting the model generate more tokens on a specific problem.

然后是推理时间计算，这只是让模型在特定问题上生成更多 token。

So I'm kind of bullish where they're all really still working.

所以我有点看涨，它们都真的还在起作用。

But the low hanging fruit has mostly been taken, especially in the last year

但低垂的果实大多已经被摘走了，尤其是在过去一年里

on reinforcement learning with verifiable rewards, which is this RLVR.

在带可验证奖励的强化学习上，也就是这个 RLVR。

And then inference time scaling, which is just why these models feel so different to use

然后是推理时间扩展，这正是为什么这些模型用起来感觉如此不同

where previously you would get that first token immediately.

以前你会立刻得到第一个词元。

And now they'll go off for seconds, minutes, or even hours generating these hidden thoughts before giving you the first word of your answer.

而现在它们会花上几秒、几分钟甚至几小时来生成这些隐藏的思考，然后才给出你答案的第一个词。

And that's all about this inference time scaling, which is such a wonderful kind of step function in terms of how the models change abilities.

而这一切都关乎这个推理时间扩展，它在模型能力变化方面就像一种奇妙的阶跃函数。

They kind of enabled this tool use stuff and enabled this much better software engineering that we were talking about.

它们某种程度上实现了这种工具使用，也实现了我们之前谈到的这种好得多的软件工程。

And this is when we say enabled almost entirely downstream of the fact that this reinforcement learning with verifiable rewards training just kind of let the models pick up these skills very easily.

这就是为什么我们说，几乎完全是因为这种带可验证奖励的强化学习训练，让模型非常容易地掌握了这些技能。

So let the models learn.

所以让模型去学习吧。

So if you look at the reasoning process when the models are generating a lot of tokens,

所以如果你观察模型生成大量词元时的推理过程，

what it will be often doing is it tries a tool.

它经常会做的是尝试一个工具。

It looks at what it gets back.

它看看返回了什么。

It tries another API.

它再试另一个 API。

It sees what it gets back and if it solves the problem.

它看看返回了什么，以及是否解决了问题。

So the models, when you're training them, very quickly learn to do this.

所以这些模型，在你训练它们的时候，很快就学会做这件事。

And then at the end of the day, that gives this kind of general foundation where the model can use CLI commands very nicely in your repo and handle Git for you and move things around and organize things or search to find more information.

然后最终，这提供了一种通用的基础，让模型可以在你的代码仓库中很好地使用命令行命令，替你处理 Git，移动东西、整理东西，或者搜索以找到更多信息。

Which if we're sitting in these chairs a year ago, it's something that we didn't really think of the models being doing.

如果一年前我们坐在这几把椅子上，这是我们当时真的没想到模型会做的事情。

So this is just kind of something that has happened this year and has totally transformed how we think of using AI, which I think is very magical.

所以这只是今年发生的一件事，它彻底改变了我们对使用 AI 的看法，我觉得这非常神奇。

such an interesting evolution and just so unlock so much value

如此有趣的演变，并且释放了如此多的价值

but it's like it's not clear what the next avenue will be in terms of unlocking stuff like this

但就像，不清楚下一个途径会是什么，在解锁这类东西方面

i think that there's there's we'll get to continual learning later but there's a lot of buzz around certain areas of ai but no one knows when the next step function will really come so

我认为，我们稍后会谈到持续学习，但 AI 的某些领域有很多热议，但没人知道下一个阶跃函数真正何时会到来，所以

you you've actually said quite a lot of things there and said profound things quickly it would be nice to unpack them a little bit

你实际上在那里说了很多，而且很快说出了深刻的东西，能稍微展开一下就好了

you said you're bullish basically on every version of scaling.

你说你基本上看好每一种扩展方式。

So we just even start at the beginning, pre-training,

所以我们甚至从一开始，预训练，

Are we kind of implying that the low-hanging fruit on pre-training scaling has been picked?

我们是不是在暗示预训练扩展的低垂果实已经被摘完了？

Has pre-training hit a plateau, or is even pre-training still you're bullish on?

预训练是已经进入平台期了，还是你依然看好预训练？

Pre-training has gotten extremely expensive.

预训练已经变得极其昂贵。

I think to scale up pre-training, it's also implying that you're going to serve a very large model to the users.

我认为要扩大预训练规模，这也意味着你要向用户提供一个非常大的模型。

So I think that it's been loosely established.

所以我认为这已经大致确立了。

the likes of GPT-4 and similar models were around this order of trillion parameters at the biggest size.

像GPT-4和类似的模型，在最大规模时大约是这个万亿参数的量级。

There's a lot of rumors that they've actually gotten smaller as training has gotten more efficient.

有很多传言说，随着训练变得更高效，它们实际上变得更小了。

You want to make the model smaller because then your costs of serving go down proportionally.

你想让模型更小，因为这样你的服务成本就会成比例下降。

These models, the cost of training them is really low relative to the cost of serving them to hundreds of millions of users.

这些模型，训练它们的成本相对于向数亿用户提供服务的成本来说真的很低。

I think DeepSeek had this famous number of about $5 million for pre-training at cloud market rates,

我认为DeepSeek有一个著名的数字，按云市场价格计算，预训练大约花了500万美元，

I think Olmo 3, section 2.4 in the paper, we just detailed how long we had the GPU clusters sitting around for training,

我想Olmo 3，论文中的2.4节，我们详细说明了GPU集群闲置等待训练的时间有多长，

which includes engineering issues, multiple seeds, and it was like about $2 million to rent the cluster to deal with all the problems and headaches of training a model.

其中包括工程问题、多个随机种子，租用集群来处理训练模型的所有问题和麻烦大约花了200万美元。

So these models are pretty, like a lot of people could get $1 to $10 million to train a model,

所以这些模型相当……比如很多人可以花100万到1000万美元来训练一个模型，

but the recurring costs of serving millions of users is really billions of dollars of compute.

但服务数百万用户的经常性成本实际上是数十亿美元的计算资源。

I think that you can look at close, like a thousand GPU rental,

我觉得你可以看看，比如一千块GPU的租赁，

you can pay a hundred grand a day for, and these companies could have millions of GPUs.

你每天可能要付十万美元，而这些公司可能拥有数百万块GPU。

You can look at how much these things cost to sit around.

你可以看看这些东西闲置着要花多少钱。

So that's kind of a big thing.

所以这算是个大问题。

And then it's like, if scaling is actually giving you a better model, like, is it going to be financially worth it?

然后就像，如果扩大规模确实能给你一个更好的模型，那么，这在经济上值得吗？

And I think it'll kind of slowly, we'll push it out as AI solves more compelling tasks.

我觉得它会慢慢地，随着AI解决更多有吸引力的任务，我们会把它推出去。

So like the likes of Cloud Opus 4.5, making cloud code just work for things

所以像Cloud Opus 4.5这样的，让Cloud代码能直接用于各种事情

i think i launched this project called like the atom project

我想我启动了这个叫Atom项目的项目

which is like american truly open models in july

它就像是七月的美国真正开放模型

and that was like a true vibe coded website

那是一个真正凭感觉编码的网站

and like i have a job um make plots and stuff

而且我有一份工作，嗯，做图表之类的

and then i came back to refresh it in the last few weeks

然后我在过去几周回来更新了它

and it's like cloud opus 4.5 versus whatever model at the time was like just crushed all the issues

结果就像Cloud Opus 4.5对比当时的其他模型，简直碾压了所有问题

that it had from building in june and

它在六月构建时遇到的那些问题，而且

July and like it might be a bigger model.

七月，而且它可能是一个更大的模型。

There's a lot of things that go into this, but that's like, there's still progress coming.

这里面涉及很多因素，但就像，仍然有进展在出现。

So what you're speaking to is the nuance of the y-axis of the scaling laws,

所以你说的是缩放定律纵轴上的细微差别，

that the way it's experienced versus on a benchmark, the actual intelligence might be different.

即它被体验的方式与在基准测试上的表现相比，实际智能可能会有所不同。

But still, your intuition about pre-training, if you scale the size of compute, will the models get better?

但仍然，你对预训练的直觉是，如果你扩大计算规模，模型会变得更好吗？

Not whether it's financially viable, but just from the law aspect of it, do you think the models will get smarter?

不是问它在财务上是否可行，而只是从定律方面来看，你认为模型会变得更聪明吗？

Yeah. And I think that there's and this sometimes comes off as like almost like disillusioned from people, leadership, AI companies saying this,

是的。而且我认为，有时候这听起来几乎像是来自人们、领导层、AI公司的一种幻灭感，他们这么说，

but they're like, it's held for 13 orders of magnitude of computers or something like why would it ever end?

但他们说，这已经保持了13个数量级的计算机规模之类的，那它为什么会结束呢？

So I think fundamentally it is pretty unlikely to stop.

所以我认为从根本上说，它不太可能停止。

It's just like eventually we're not even going to be able to test the bigger scales because of all the problems that come with more compute.

就像最终我们甚至无法测试更大的规模，因为更多计算带来的所有问题。

I think that there's a lot of talk on how 2026 is a year when very large Blackwell compute clusters, like gigawatt scale facilities, hyperscalers are coming online.

我认为有很多讨论关于2026年将是大型Blackwell计算集群，比如吉瓦级设施、超大规模企业上线的一年。

And these were all contracts for power and data centers that were signed and sought out in like 22 and 2023.

这些都是2022年和2023年左右签署和寻求的电力与数据中心合同。

So before or right after ChatGPT.

所以是在ChatGPT之前或之后不久。

So it took this two to three year lead time to build these bigger clusters to train the models.

所以建造这些更大的集群来训练模型需要两到三年的前置时间。

While there's obviously immense interest in building even more data centers than that.

虽然显然有极大的兴趣建造比那更多的数据中心。

So that is like kind of the crux that people are saying is like these new clusters are coming.

所以这就像是人们所说的关键点，这些新集群即将到来。

The labs are gonna have more compute for training.

实验室将拥有更多用于训练的算力。

They're going to utilize this, but it's not a given.

他们会利用这一点，但这并不是必然的。

And it's like, I've seen so much progress that I expect it.

而且就像，我看到了如此大的进步，所以我期待它。

And I expect a little bit bigger models.

我期待稍微大一点的模型。

And I expect, I would say it's more like we'll see a $2,000 subscription this year.

我预计，我会说更像是我们今年会看到2000美元的订阅。

We see $200 subscriptions. It's like that can 10x again.

我们看到200美元的订阅。就像那可以再增长10倍。

And these are the kinds of things that could come and they're all downstream of this like bit bigger model that offers just a little bit more cutting edge.

而这些是可能出现的事情，它们都是这个稍大一点的模型的下游产物，这个模型提供稍微更前沿一点的能力。

So, you know, it's reported that XAI is going to hit that one gigawatt scale early 26 and full two gigawatt by year end.

所以，你知道，据报道XAI将在26年初达到1吉瓦规模，并在年底达到完整的2吉瓦。

How do you think they'll utilize that in the context of scaling laws?

你认为他们会在缩放定律的背景下如何利用这一点？

Is a lot of that inference, is a lot of that training?

其中很多是推理，还是很多是训练？

It ends up being all of the above.

最终是以上所有。

So I think that all of your decisions when you're training a model come back to pre-training.

所以我认为，当你训练一个模型时，你所有的决策都会回到预训练上。

So if you're going to scale RL on a model, you still need to decide on your architecture that enables this.

所以如果你要在模型上扩展强化学习，你仍然需要决定能够实现这一点的架构。

We were talking about other architectures using different types of attention.

我们之前讨论过使用不同类型注意力的其他架构。

We're also talking about mixture of experts models.

我们也在讨论专家混合模型。

This sparse nature of MOE models makes it much more efficient to do generation,

MOE模型的这种稀疏特性使得生成更加高效，

which becomes a big part of post-training.

这成为后训练的重要组成部分。

And it's like, you need to have your architecture ready so that you can actually scale up this compute.

就像，你需要准备好你的架构，这样你才能真正扩展这个计算。

I still think most of the compute is going in at pre-training.

我仍然认为大部分计算都投入在预训练中。

Because you can still make a model better, you still want to go and revisit this.

因为你仍然可以让模型变得更好，你仍然想要去重新审视这一点。

You still want the best base model that you can.

你仍然想要你能得到的最好的基础模型。

And in a few years, that'll saturate and the RL compute will just go longer.

而在几年后，那会饱和，而强化学习的计算只会持续更久。

Is there people who disagree with you?

有人不同意你的观点吗？

that say basically pre-training is dead.

他们说基本上预训练已死。

It's all about scaling inference, scaling post-training, scaling context, continual learning, scaling data, synthetic data.

一切都关乎扩展推理、扩展后训练、扩展上下文、持续学习、扩展数据、合成数据。

People vibe that way and describe it in that way,

人们有那种感觉，并那样描述，

but I think it's not the practice that is happening.

但我认为实际发生的实践并非如此。

It's just the general vibe of people saying this thing is dead.

这只是人们说这个东西已死的普遍氛围。

The excitement is elsewhere.

兴奋点在其他地方。

So the low-hanging fruit in RL is elsewhere.

所以强化学习中的低垂果实在其他地方。

For example, we released our model in November for every company has deadlines.

例如，我们在11月发布了我们的模型，因为每家公司都有截止日期。

Our deadline was like November 20th.

我们的截止日期大约是11月20日。

And for that, our RL run was five days, which compared to 2024 is a very long time

为此，我们的强化学习运行了五天，与2024年相比，这已经是很长的时间了

to just be doing post-training at a model of like 30 billion parameters.

仅仅是在一个约300亿参数的模型上进行后训练。

It's not a big model.

这不是一个大模型。

And then in December, we had another release, which was just we let the RL run go for another three and a half weeks and the model got notably better.

然后在12月，我们又有一次发布，就是让强化学习运行再持续了三周半，模型明显变得更好了。

So we release it.

所以我们发布了它。

And that's a big amount of time to just allocate to something that is going to be your peak for the year.

而这是把大量时间仅仅分配给一个将成为你年度巅峰的东西。

So it's like there's these types of discussions that happen when they're training a model where they just like can't they can't leave it forever

所以就像在训练模型时会出现这类讨论，他们就是不能永远放着不管

you have to keep you have to keep pulling in the improvements you have from your researchers

你必须不断从研究人员那里获取改进

so that's like you redo pre-training you'll do this post-training for a month but then you need to give it to your users you need to do safety testing

所以就像你重做预训练，你会做这个后训练一个月，但然后你需要把它交给用户，你需要做安全测试

so it was kind of just like i think there's a lot in place that reinforces this cycle of just keep updating the models there's things to improve you get a

所以这有点像，我认为有很多因素在强化这个循环，就是不断更新模型，总有东西可以改进，你会得到一个

New compute cluster that lets you do something maybe more stable or faster.

新的计算集群，让你能做一些可能更稳定或更快的事情。

It's like you hear a lot about Blackwell having rollout issues where at AI2,

就像你听到很多关于Blackwell推出问题的消息，在AI2，

most of the models were pre-training around like one to two thousand GPUs.

大多数模型都是在约一到两千个GPU上进行预训练的。

But when you're pre-training on 10,000 or 100,000 GPUs, you hit very different failures.

但当你在1万或10万个GPU上进行预训练时，你会遇到非常不同的故障。

So GPUs are known to break in weird ways, and doing 100,000 GPU run is like you're pretty much guaranteed to always have at least one GPU that is down,

所以GPU以奇怪的方式出故障是出了名的，而进行10万GPU的运行，就像你几乎肯定总是至少有一个GPU是宕机的，

and you need to have your training code handle that redundancy, which is just a very different problem.

而你需要让你的训练代码处理这种冗余，这只是一个非常不同的问题。

Whereas like what we're doing, like I'm playing with post-training on DGX Spark,

而像我们正在做的，比如我在DGX Spark上玩后训练，

or you have your book, it's like, or people learning ML, it's like what they're battling to train these biggest models is just like mass distributed scale.

或者你有你的书，就像，或者人们学习机器学习，就像他们努力训练这些最大模型所面临的，就像大规模分布式扩展。

And it's a very different, but that's somewhat different than like are these like, that's a systems problem.

这是一个非常不同的，但那有点不同于像这些，那是一个系统问题。

In order to enable the scaling laws, especially a pre-training, you need all these GPUs at once.

为了实现扩展定律，尤其是预训练，你需要同时拥有所有这些GPU。

When we shift to reinforcement learning, it actually lends itself to heterogeneous compute,

当我们转向强化学习时，它实际上适合异构计算，

because you have many copies of the model and to do a primer for a language

因为你有许多模型副本，并且为一个语言做入门

model reinforcement learning, what you're doing is you have two sets of GPUs.

模型强化学习，你要做的是拥有两组GPU。

One is you can call it the actor, and one you call the learner.

一组你可以称之为actor，另一组你称之为learner。

The learner is where your actual reinforcement learning updates are going to do these.

learner是实际强化学习更新将要进行的地方。

are traditionally policy gradient algorithms, proximal policy optimization, PPO, and group relative policy optimization, GRPO, are the two popular classes.

传统上是策略梯度算法，近端策略优化（PPO）和组相对策略优化（GRPO）是两种流行的类别。

And on the other side, you're going to have actors which are generating completions.

另一方面，你会有actor，它们正在生成补全。

And these completions are the things that you're going to grade.

这些补全就是你要评分的东西。

So reinforcement learning is all about optimizing reward.

所以强化学习就是关于优化奖励。

And in practice, what you can do is that you can have a lot of different actors in different parts of the world doing different types of problems.

在实践中，你可以做的是，你可以在世界不同地方有很多不同的actor，处理不同类型的问题。

And then you send it back to this highly networked compute cluster to do this actual learning.

然后你把它发送回这个高度网络化的计算集群来进行实际的学习。

where you take the gradients and you need to have a tightly meshed network where you can do different types of parallelism and spread out your model for efficient training.

在这里你获取梯度，你需要有一个紧密连接的网络，可以进行不同类型的并行计算，并分散你的模型以实现高效训练。

So there's just like a lot of every different type of training and serving has these considerations you need to scale.

所以就像每种不同类型的训练和服务都有这些你需要扩展的考虑因素。

Like we talked about pre-training, we talked about RL and then inference time scaling is like,

就像我们谈过的预训练，我们谈过强化学习，然后推理时间扩展就像是，

how do you serve a model that's thinking for an hour to 100 million users?

你如何为一个思考一小时的模型服务1亿用户？

I don't really know about that, but I know that's a hard problem.

我不太了解那个，但我知道那是个难题。

And in order to give people this intelligence, there's all the systems problems and we need more compute and you need more stable compute to do it.

而为了给人们这种智能，有所有的系统问题，我们需要更多计算，你需要更稳定的计算来做到。

you're bullish on all of these kinds of scaling is what i'm hearing on the inference on the reasoning even on the pre-training

你看好所有这些类型的扩展，这是我听到的，关于推理、关于推理能力，甚至关于预训练

yeah so that's a big can of worms here but so that

是的，所以那是个大麻烦，但所以那

basically two the knobs are the training and the inference scaling

基本上两个旋钮是训练和推理扩展

where you can get gains and so in a world where we had let's say infinite compute resources

在那里你可以获得收益，所以在一个我们拥有比如说无限计算资源的世界里

you want to do all of them about like so you have training you have inference scaling

你想做所有这些，就像所以你有训练，你有推理扩展

and training is like a hierarchy it's pre-training mid-training post-training changing the model size, more training data, training a bigger model.

而训练就像一个层级，它是预训练、中期训练、后训练，改变模型大小、更多训练数据、训练一个更大的模型。

gives you more knowledge in the model than the model

给模型带来的知识比模型

um let's say has a better, it's like a better base model back in the day, or we still we call it foundation model, and it unlocks

嗯，比如说有更好的，就像以前更好的基础模型，或者我们仍然称之为基础模型，而它解锁了

so you but you don't let's say have the model be able to solve your most complex task tasks during pre-training or after pre-training

所以，但假设你并没有让模型能够在预训练期间或预训练之后解决你最复杂的任务

you still have these other unlock phases where you have mid-training or long context for example post-training with lrvr that unlocks capabilities that the model has in terms of just knowledge in the pre-training

你仍然有这些其他的解锁阶段，比如你有中期训练或长上下文，例如使用 lrvr 的后训练，这解锁了模型在预训练中仅就知识而言所具备的能力

and I think sure if you so do more pre-training you get a better base model that you can unlock later

而且我认为，如果你做更多的预训练，你会得到一个更好的基础模型，之后你可以解锁它

but like Nathan said it just becomes too expensive so we don't have infinite compute so you have to decide do I want to spend that compute more on making the model larger

但就像 Nathan 说的，它变得太昂贵了，所以我们没有无限的计算资源，因此你必须决定：我是想把那些计算资源更多地花在让模型更大上

but you know it's like a trade-off, it's it's like in an ideal world you want to do all of them and I think in that sense scaling is still pretty much alive

但你知道，这就像一种权衡，就像在理想世界中你想做所有这些，而我认为从这个意义上说，扩展仍然相当活跃

you would still get a better model but like we saw with gpd 4.5 it's just not worth it I mean it's like because you can let's

你仍然会得到一个更好的模型，但就像我们在 gpd 4.5 上看到的，它就是不值得，我的意思是，这就像因为你可以，让我们

Say you can unlock more performance with other techniques at that current moment,

比如说，你可以通过其他技术在当前时刻解锁更多性能，

especially, um, if you look at inference scaling,

尤其是，嗯，如果你看看推理扩展，

that's one of the biggest gains this year with 01, um,

这是今年最大的收益之一，用01，嗯，

where it took a smaller model further than pre-training a larger model like GBD 4.5.

它让一个较小的模型比预训练一个更大的模型（如GBD 4.5）走得更远。

So it's like I wouldn't say pre-training scaling is that it's just like there are other more attractive ways to scale right now at the moment,

所以这就像我不会说预训练扩展就是那样，只是现在有其他更有吸引力的扩展方式，

but at some point, you know, you will still want to make some progress on the pre-training.

但在某个时候，你知道，你仍然希望在预训练上取得一些进展。

The thing is also to consider, um, where you, where do you want to spend your money?

还要考虑的是，嗯，你，你想把钱花在哪里？

If you spend it more on the pre-training, it's like a fixed cost: you train the model and then it has this capability forever, you can always use it and so forth.

如果你在预训练上花更多钱，这就像固定成本：你训练模型，然后它就永远拥有这个能力，你可以一直使用它等等。

With inference scaling, you don't spend money during training, you spend money later per query, and then it's also like the math: how long is my model going to be on the market?

对于推理扩展，你不在训练期间花钱，而是之后按查询花钱，然后这还涉及数学：我的模型将在市场上存在多久？

If I replace it in half a year, maybe it's not worth spending five million, ten million, hundred million dollars on the training.

如果我半年内替换它，也许不值得在训练上花费五百万、一千万、一亿美元。

It longer, maybe it's just I will just do more inference scaling and get

它更长，也许我只是会做更多的推理扩展并获得

The performance from there, it may cost me two million in terms of user queries.

从那里的性能来看，就用户查询而言，它可能会花费我两百万。

It becomes a question of how many users you have and then doing the math.

这就变成了一个你有多少用户然后做数学计算的问题。

Um, and I think that's also where it's interesting.

嗯，我认为这也是有趣的地方。

Where GGPD is in a position, I think they have a lot of users.

GGPD 所处的位置，我认为他们有很多用户。

Where they need to go a bit cheaper.

他们需要变得更便宜一些。

Where they have that GPD5 model that is a bit smaller.

他们有那个稍微小一点的 GPD5 模型。

Other companies that have as if your customers have other trade-offs.

其他公司，如果你的客户有其他权衡的话。

For example, there was also the math Olympiad or some of these math problems.

例如，还有数学奥林匹克竞赛或一些这样的数学问题。

Where JGBT or OpenAI, they had a proprietary model.

JGBT 或 OpenAI，他们有一个专有模型。

And I'm pretty sure it's just like a model that has been maybe fine-tuned a little bit more,

而且我很确定它只是一个可能被稍微更多微调过的模型，

but most of it was during inference scaling to achieve this peak performance in certain tasks.

但大部分是在推理扩展期间实现的，以在某些任务中达到这种峰值性能。

Where you don't need that all the time.

你并不需要一直那样。

But yeah, long story short, I do think all of these pre-training, mid-training, post-training, infant scaling, they are all still things you want to do.

但是的，长话短说，我确实认为所有这些预训练、中期训练、后训练、婴儿扩展，它们仍然都是你想做的事情。

It's just finding at the moment in this year, it's finding the right ratio that gives you the best bang for the buck, basically.

只是今年此刻，基本上就是找到能给你带来最佳性价比的正确比例。

I think this might be a good place to define pre-training, mid-training, and post-training.

我认为这可能是一个定义预训练、中期训练和后训练的好地方。

So pre-training is the classic training, one next token prediction at a time.

所以预训练就是经典的训练方式，一次预测下一个词元。

You have a big corpus of data.

你有一个庞大的数据语料库。

And Nathan also has very interesting insights there, because of OMO3, it's a big portion of the paper focuses on the right data mix.

而Nathan在这方面也有非常有趣的见解，因为OMO3，论文很大一部分都在关注正确的数据配比。

So pre-training is essentially just, you know, cross entropy loss training on next token prediction on a vast corpus of internet data, books, papers, and so forth.

所以预训练本质上就是，你知道，在庞大的互联网数据、书籍、论文等语料库上进行交叉熵损失训练，预测下一个词元。

It has changed a little bit over the years in the sense people used to throw in everything they can.

这些年来它有一点变化，就是过去人们会把所有能拿到的东西都扔进去。

Now, it's not just raw data.

现在，它不只是原始数据。

It's also synthetic data where people rephrase certain things.

它还包括合成数据，也就是人们对某些内容进行改写。

So synthetic data doesn't necessarily mean purely AI made up data.

所以合成数据并不一定意味着纯粹由AI编造的数据。

It's also taking something from an article, Wikipedia article, and then rephrasing it as a Q&A question or summarizing it, rewording it and making better data that way.

它也包括从一篇文章、维基百科文章中提取内容，然后把它改写成问答形式，或者进行总结、改写，从而制作出更好的数据。

Because I think of it also like with humans, if someone, let's say, reads the book compared to a messy, I don't know, no offense, but like Reddit post or something like that.

因为我也会把它类比到人类身上，如果一个人，比如说，读一本书，相比于一个杂乱的，我不知道，无意冒犯，但就像Reddit帖子之类的。

I do think you learn, no offense, but I think.

我确实认为你能学到东西，无意冒犯，但我认为。

There's going to be a post about this.

会有一篇关于这个的帖子。

Some Reddit data is very coveted and excellent for training.

有些Reddit数据非常抢手，非常适合训练。

You just have to filter it.

你只需要对它进行过滤。

I think that's the idea.

我觉得这就是那个思路。

I think it's like if someone took that and rephrases that in a, let's say, more concise and structured way,

我觉得就像如果有人把它拿来，用更简洁、更有结构的方式重新表述，

I think it's higher quality data that gets the LLM maybe the same.

我觉得那是更高质量的数据，让大语言模型也许得到同样的结果。

You get the same LLM out of it at the end, but it gets there faster.

你最终从中得到同样的大语言模型，但它到达得更快。

It trains faster because let's say if the grammar and the punctuation is correct,

它训练得更快，因为比如说如果语法和标点都是正确的，

it already learns the correct way versus getting information from a messy way and then learning later how to correct that and stuff like that.

它已经学到了正确的方式，而不是从混乱的方式中获取信息，然后再学习如何纠正它之类的。

So I think that is how pre-training evolved and how still why scaling still works is that it's not about just the amount of data, it's also the tricks to make that data better for you in a sense.

所以我认为这就是预训练演变的方式，以及为什么扩展仍然有效，因为它不只是关于数据量，还在于让那些数据在某种意义上对你更有用的技巧。

And mid-training is, I mean, it used to be called pre-training.

而中期训练，我的意思是，它过去被称为预训练。

I think it's called mid-training because it was awkward to have pre-training and post-training, but nothing in the middle, right?

我认为它被称为中期训练，是因为有预训练和后训练，但中间什么都没有，这很别扭，对吧？

It sounds a bit weird. You have pre-training and post-training, but what's the actual training?

听起来有点奇怪。你有预训练和后训练，但实际的训练是什么？

So the mid-training is usually similar to pre-training, but you know, it's a bit more, I would say, specialized in pre-training.

所以中期训练通常和预训练类似，但你知道，它更，我会说，更专门化于预训练。

It's the same algorithm, but what you do is you focus, for example, on long context.

算法是一样的，但你要做的是专注于，比如说，长上下文。

Like one example, you have long context documents.

举个例子，你有长上下文文档。

The reason you don't do that during just pure pre-training is because you don't have that many long context documents.

你在纯预训练期间不这么做的原因是，你没有那么多长上下文文档。

So you have a specific phase.

所以你有一个特定的阶段。

And one problem of LLMs is also still, it's a neural network.

而LLMs的一个问题也仍然是，它是一个神经网络。

has the problem of catastrophic forgetting so you teach it something it forgets other things and you want to it's not 100 forgetting but you know it's like no free lunch you can't it's also the same with humans

有灾难性遗忘的问题，所以你教它一些东西，它会忘记其他东西，你想要——不是100%遗忘，但你知道，就像没有免费的午餐，你不能——人类也是一样

if you ask me some math i learned 10 years ago i don't know i would have to look at it again

如果你问我一些我10年前学的数学，我不知道，我得再看一遍

nathan was actually saying that he's consuming so much content that there's a catastrophic forgetting issue

内森实际上在说，他消费了太多内容，以至于出现了灾难性遗忘的问题

yeah i'm like trying to learn so much about ai i was like i was learning about pre-training parallelism i'm like i lost something and i don't know what it was

是的，我就像试图学习很多关于AI的东西，我当时正在学习预训练并行性，我就像我丢失了一些东西，我不知道那是什么

i don't anthropomorphize llms but it's i think the same kind of in that sense how humans learn i mean

我不会将LLMs拟人化，但我认为在那种意义上，这就像人类学习一样，我是说

the quantity is not always better because yeah you it's like being selective and i in the mid training is being selective in terms of quality content at the end

数量并不总是更好，因为是的，你——这就像要有选择性，而我在中期训练中就是在质量内容方面要有选择性，在最后

so the last thing the lm has seen is the quality stuff and then post training is all

所以LM最后看到的东西是高质量的内容，然后后训练全是

the uh fine-tuning, supervised fine-tuning, DPO, um, reinforcement learning with verifiable rewards, with human feedback, and so forth.

呃，微调、监督微调、DPO、嗯，带可验证奖励的强化学习、带人类反馈的强化学习等等。

So the refinement stages, and it's also interesting, it's like the cost thing, right? I mean,

所以这些精炼阶段，而且这也很有意思，就像成本问题，对吧？我是说，

it's like pre-training, you spend a lot of money on that, right?

就像预训练，你在那上面花很多钱，对吧？

Now RL, a bit less. RL, you don't really...

现在强化学习，稍微少一点。强化学习，你其实不……

I would say teach it knowledge. It's more like unlocking the knowledge.

我会说是教它知识。这更像是解锁知识。

It's more like a skill learning, like how to solve problems with the knowledge that it has from pre-training.

这更像是一种技能学习，比如如何用它在预训练中获得的知识来解决问题。

There are actually three papers this year or last year, 2025, on RL for pre-training.

实际上今年或去年，2025年，有三篇关于用强化学习做预训练的论文。

But I, I mean, I don't think anyone does that in production, toy examples for now.

但，我是说，我觉得没有人在生产环境里那么做，目前只是玩具示例。

Examples, right? But to generalize, RL post-training is more like the skill unlock,

示例，对吧？但要泛化的话，强化学习后训练更像是解锁技能，

where pre-training is like soaking up the knowledge, essentially.

而预训练本质上就像是吸收知识。

A few things that could be helpful for people, a lot of people get like...

有几件事可能对人们有帮助，很多人会像……

You have, think of synthetic data as being bad for training the models, you mentioned, like

你有，把合成数据看作对训练模型有害，你提到过，就像

the DeepSeek get almost OCR, which is optical character recognition paper.

DeepSeek 那篇几乎就是 OCR，也就是光学字符识别的论文。

A lot of labs did. AI2 had one, had multiple. And the reason that each of these labs have these is because

很多实验室都做过。AI2 有一个，有多个。而这些实验室之所以都有这些，是因为

There's vast amounts of PDFs and other digital documents on the web

网上有大量的PDF和其他数字文档

that are in formats that aren't encoded with text easily, so you use these almost OCR, these or deep-seek OCR

它们的格式不容易用文本编码，所以你要用这些几乎算是OCR的，这些或者DeepSeek OCR

and we called our almost OCR to extract

我们把自己的这个几乎算是OCR的东西叫做提取

what can be trillions of tokens of candidate data for pre-training

可以提取出数万亿个token的候选预训练数据

and pre-training data set size is on the order of trillions, is measured in trillions of tokens

而预训练数据集的大小是数万亿这个量级，是以数万亿个token来衡量的

Smaller models from researchers can be something like five to ten trillion, um, quen is documented

研究人员做的小模型可能是五到十万亿，嗯，Qwen是有记录的

going up to like 50 trillion, and there's rumors that these closed labs can go to like 100 trillion tokens

能到大概50万亿，还有传言说这些闭源实验室能做到大概100万亿个token

and just getting this potential data to put in, I think they, they have a very big funnel

而只是获取这些潜在数据放进去，我觉得他们，他们有一个非常大的漏斗

and then the data you actually train the model on is a small percentage of this, like the, this character recognition data

然后你实际用来训练模型的数据只是其中的一小部分，就像这个字符识别数据

would be described as synthetic data for pre-training in a lab, and then

在实验室里会被描述为用于预训练的合成数据，然后

there's also the things like ChatGPT now

还有像现在ChatGPT这样的东西

gives wonderful answers and you can train on those best answers and that's synthetic data

能给出很棒的答案，你可以用那些最好的答案来训练，那就是合成数据

it's very different than like early ChatGPT, lots of hallucinations data

这和早期的ChatGPT很不一样，那时候有很多幻觉数据

when people became grounded synthetic data.

当人们变得有依据时的合成数据。

One interesting question is, if I recall correctly, Olmo 3 was trained with less data than specifically some other open-weight models, maybe even Olmo 2,

一个有趣的问题是，如果我没记错的话，Olmo 3 的训练数据比某些其他开放权重模型要少，甚至可能比 Olmo 2 还少，

but you still got better performance and that might be one of the examples how the data helped.

但你还是得到了更好的表现，这也许就是数据如何发挥作用的一个例子。

It's mostly down to data quality.

这主要取决于数据质量。

I think if we had more compute, we would train for longer.

我想如果我们有更多算力，我们会训练更长时间。

I think we ultimately see that as, like, just like something we would want to do

我想我们最终会把这看作，就像，我们想做的一件事，

and especially with big models, you need to have more compute

尤其是对于大模型，你需要有更多算力，

because we talk about having more parameters and we talk about knowledge and essentially there's a ratio where big models can absorb more from data

因为我们谈论拥有更多参数，我们谈论知识，本质上存在一个比例，大模型可以从数据中吸收更多，

and then you're going to you get more benefit out of this

然后你就能从中获得更多收益，

it's it's like one of these any logarithmic graph in your mind is like a small model

这就像你脑海中的那种对数图，小模型

will level off sooner if you're measuring trends of tokens and bigger and bigger models

如果你衡量的是 token 的趋势，会更早趋于平缓，而越来越大的模型

need more but mostly is we aren't training that big of models right now ai2

需要更多，但主要是我们现在并没有训练那么大的模型，AI2

and getting the highest quality data we can is the natural starting point is there

而尽可能获取最高质量的数据是自然的起点，有没有

something to be said uh about the topic of data quality is there some long-hanging fruit there still where the quality could be improved

关于数据质量这个话题还有什么可说的，是否还有一些容易摘取的果实，质量还能提高

it's like turning the crank

这就像转动曲柄一样

So I think historically in the open, there's been like a canonical

所以我认为从历史上看，在开放领域，一直有一个像是经典的

best pre-training data set that has moved around between who has the most recent one or the best recent effort.

最好的预训练数据集，它在谁拥有最新的或最好的近期成果之间流转。

Like AI2's Dolmo was very early with the first Ulmo and Hugging Face had FineWeb.

比如AI2的Dolmo很早就推出了第一个Ulmo，而Hugging Face有FineWeb。

And there's a DCLM project, which has been kind of like a, which is, it stands for data comp language model.

还有一个DCLM项目，它有点像是，也就是，它代表数据竞赛语言模型。

There's been data comp for other machine learning projects and they have had a very strong data set.

其他机器学习项目也有数据竞赛，它们拥有非常强大的数据集。

and a lot of it is the internet is becoming fairly closed off

而且其中很多是互联网正变得相当封闭

so we have common crawl which i think is hundreds of trillions of tokens and you filter it and it looks like being

所以我们有Common Crawl，我认为它有数百万亿个token，你过滤它，它看起来像是

a lot of scientific work where you're training classifiers and making decisions based on how do you prune down this this data set into the highest quality stuff and the stuff that suits your tasks

很多科学工作，你训练分类器并基于如何将这个数据集修剪成最高质量的内容以及适合你任务的内容来做决定

so previously language models were tested a lot more on like knowledge and just kind of conversational things but now they're expected to do math and code.

所以以前语言模型更多是在知识和对话类内容上测试，但现在它们被期望做数学和代码。

So to train a reasoning model, you need to remix your whole data set.

所以训练一个推理模型，你需要重新混合你的整个数据集。

And there's a lot of actually wonderful scientific methods here where you can take your gigantic data set, you sample a lot of really tiny things from different sources.

这里有很多实际上很棒的科学方法，你可以取用你庞大的数据集，从不同来源采样很多非常小的东西。

So you say you have GitHub, Stack Exchange, Reddit, Wikipedia,

所以你说你有 GitHub、Stack Exchange、Reddit、维基百科，

you can sample small things from them and train small models on each of these mixes

你可以从中抽取小样本，并在每个混合数据上训练小模型，

and measure their performance on your evaluations.

然后在你的评估集上衡量它们的表现。

And you can just do like basic linear regression and it's like, here's your optimal data set.

你可以直接做基本的线性回归，然后它就会说，这是你的最优数据集。

But if your evaluations change, your data set changes a lot.

但如果你的评估变了，你的数据集就会变化很大。

So a lot of Olmode 3 was new sources for reasoning to be better at math and code.

所以 Olmode 3 的很多内容都是新的推理来源，以便在数学和代码方面表现更好。

And then you do this mixing procedure and it gives you the answer.

然后你执行这个混合过程，它就会给你答案。

And I think that's a lot of that's happened at labs this year.

我认为今年实验室里发生了很多这样的事。

There's new hot things, whether it's coding environments or web navigation.

有新的热门方向，无论是编码环境还是网页导航。

You just need to bring in new data.

你只需要引入新数据。

You need to change your whole pre-training so your post-training can work better and stuff like this.

你需要改变整个预训练，这样你的后训练才能更好地工作，诸如此类。

And that's like the constant re-evolution and the redetermining of what they care about for their models.

这就像是不断的重新演化，以及重新确定他们关心模型的哪些方面。

Are there fun anecdotes of what sources of data are particularly high quality that we wouldn't expect?

有没有一些有趣的轶事，关于哪些数据来源质量特别高，而我们原本不会想到？

You mentioned Reddit sometimes can be a source.

你提到 Reddit 有时可以成为一个来源。

Reddit was very useful.

Reddit 非常有用。

I think that PDFs is definitely one.

我认为 PDF 绝对是一个。

Especially archive.

尤其是 archive。

Yeah, so like AI2 has run Semantic Scholar for a long time, which is a, like you can say, is a competitor to Google Scholar with a lot more features.

是的，所以像 AI2 已经运行 Semantic Scholar 很长时间了，它可以说是一个 Google Scholar 的竞争对手，而且功能多得多。

And to do this, AI2 has found and scraped a lot of PDFs

为了做到这一点，AI2找到并抓取了很多PDF文件

for openly accessible papers that might not be like behind the closed paid garden of a certain publisher.

这些是开放获取的论文，可能不在某个出版商封闭的付费花园后面。

So like truly open scientific PDFs.

所以就像是真正开放的科学PDF。

And if you like you sit on all of these and you process it and you can get value out of it.

如果你坐在所有这些数据上，处理它们，你就能从中获得价值。

And I think that like a lot of that style of work has been done by the frontier labs much earlier.

我认为很多这种风格的工作早就被前沿实验室做过了。

And it's just like you need to have a pretty skilled researcher that understands how things change models and they bring it in and they clean it.

这就像你需要一个相当熟练的研究员，了解模型如何变化，他们把它带进来并清理它。

And that's a lot of labor that like I think of a lot of frontier labs when they scale researchers a lot more goes into data.

这是大量的劳动，就像我认为很多前沿实验室在扩大研究人员规模时，更多的精力投入到数据中。

You have people like if you want to if you join a frontier lab, you want to have impact.

你有这样的人，如果你想加入一个前沿实验室，你想要产生影响。

The best way to do it is just make find new data that's better.

最好的方法就是找到更好的新数据。

And then like the fancy, glamorous algorithmic things like figuring out how to make a one is like the sexiest thought of a scientist.

然后像那些花哨、迷人的算法之类的东西，比如弄清楚如何做一个，是科学家最性感的想法。

It's like, oh, I figured out to scale RL.

就像，哦，我弄明白了如何扩展强化学习。

And there's a group that did that.

有一个团队做到了这一点。

But I think most of the contributions is like I'm going to make the data better or I'm going to make the infrastructure better

但我认为大部分的贡献就像是，我要把数据做得更好，或者我要把基础设施做得更好，

so that everybody in my team can run experiments five percent faster.

这样我团队里的每个人都能把实验跑快百分之五。

At the same time, I think it's also one of the closest guarded secrets what your training data is for legal reasons.

与此同时，我认为它也是最严守的秘密之一，出于法律原因，你的训练数据是什么。

And so there's also, I think, a lot of work that goes into hiding what your training data was, essentially.

所以我认为，也有很多工作本质上是为了隐藏你的训练数据是什么。

Like trying the model to not give away the sources because of legal reasons.

就像试图让模型不要泄露来源，因为法律原因。

The other thing to be complete is that some people are trying to train on only licensed data,

另一件需要补充完整的事情是，有些人正试图只用获得许可的数据来训练，

where Common Crawl is a scrape of like the whole internet.

而Common Crawl基本上是对整个互联网的抓取。

So if I host multiple websites, I'm happy to have them train language models, but I'm not explicitly licensing what governs it.

所以如果我托管多个网站，我愿意让它们训练语言模型，但我并没有明确地授权什么来约束它。

And therefore, the common crawl is largely unlicensed, which means that your consent really hasn't been provided for how to use the data.

因此，Common Crawl基本上是没有许可的，这意味着关于如何使用这些数据，其实并没有得到你的同意。

There's another idea where you can train language models only on data that has been licensed explicitly.

还有另一种想法，就是你可以只用明确获得许可的数据来训练语言模型。

So that kind of governing contract is provided.

这样就提供了一种约束性的合同。

And I'm not sure if Aperitice is the copyright thing or the license thing.

我不确定Aperitice是版权方面的事还是许可方面的事。