(圖說：我在大學時期也是充滿不解地想著，為什麼都是人類要去適應各種資訊系統，而不是軟體自己來適應我們人類的工作流程。當然二十幾年前有各種限制條件，但我比較受不了的是墨守成規，覺得事情就是這樣了，沒得改變了，而自己放棄改變的機會，而持續衝撞各種框框條條。雖然長大了稍微知道如何閃避障礙物，沒有進外商，但至少學著不會內傷，而保持著衝撞，也許我也是某種墨守成規，只是是我自己的成規。攝於 2016 出差，跟著在地人一起用午餐後的義式濃縮咖啡，他們說他們冰咖啡都這樣喝，我就說我也要嘗試看看。圖片來源：Ernest。)
✳️ OpenAI 首席科學家重新定義「持續學習」
當大家都在關注各種 AI 應用與工具的時候，有些關於學習與底層本質的探索，其實更值得花些時間在上面，透過這些探索也許可以更認識人類自己，也可能帶來人類與 AI 的各種共存可能性。就如 Ernest 大學時參與的 AIESEC 社團其中一個核心價值觀 Living Diversity（體現多元），在跨文化、跨背景的環境中一起工作、生活，並從不同的觀點中一起學習。也許會有世界和平的那一天吧（？
Unsupervised Learning 這一集訪問 OpenAI 首席科學家 Jakub Pachocki，聊了 Continual Learning（持續學習）、RL（強化學習）、思維鏈監控、算力分配跟介面演進。訪談也順便更新了 OpenAI 的時間表：研究實習生等級的系統目標放在 2026 年 9 月，全自動化 AI 研究員的目標是 2028 年 3 月，Codex 現在已經接手 OpenAI 內部多數的實際寫程式工作。以下抽出四段覺得可能值得記下來的觀察。
⌬ 持續學習不該被忽略，而被重新定義
- 不少人覺得「持續學習」是 AI 研究的盲點，Jakub 說其實不是這樣。
- 他說這不是被忽略的題目，而是 OpenAI 正在建構的核心。
- 他還拉回到當年的 GPT-3 論文：讓模型「在情境裡學會學習」一直是驅動 GPT 系列繼續擴展的興奮點。
- RL 在特定任務上資料效率極高，但還有一種更根本、資料效率更高的方式，就是 In-context learning（ICL; 上下文學習），而他預期這個方向會越做越好。
⌬ 思維鏈要「刻意不監督」才有對齊價值
- Jakub 解釋為什麼產品端會把思維鏈藏起來。
- 不是商業封閉，是為了保留那份「沒被修飾過」的誠實。
- 他說如果把思維鏈直接攤在使用者面前，最後勢必要拿去訓練，模型就會學會把不討喜的想法藏起來。
- 他的說法是，不要讓訓練訊號反過來跟我們作戰。
- 他也坦言這不是對齊的萬靈丹，但起碼是現在還可以派上用場的觀察手段，讓我們有機會看見模型在動機與泛化上的演變。
⌬ 長遠研究逐漸與短期產品策略脫鉤
- 被問到 OpenAI 怎麼在這麼多方向上分配算力，Jakub 的回答比較長遠：
- 他們會刻意把很大一塊算力明確預算給最具擴展性的通用智慧研究方法，即使那不是當下最有效率的配置，還是留給最核心的那條產品線。
- 不能什麼都想做，必須要刻意放棄。 .
- 他也提到 OpenAI 研究組織的長遠焦點，已經逐漸跟 ChatGPT、Codex 這類短期產品策略脫鉤。
（我們陪客戶梳理工作流程的時候，也滿常跟客戶聊到：不是每個低垂的果實都要摘，很多時候摘了反而耽擱了路程。但心中有所定見，才能堅持得住。）
⌬ Harness 的未來：AI 來我們這裡，不是我們去 AI 那裡
- 被問到各行各業要不要自己做一套 harness，
- Jakub 的看法是：harness 本身不該是限制，未來會有更通用的 harness 可以跨產業使用。
- 他想像的終局是 AI 直接在 Slack 這類使用者現有的介面工作，預設應該是「AI 來你這裡」，不是使用者被迫去 AI 的地方。
讀完整場訪談，也許 OpenAI 已經注意到原本的產品線過於分散，而調整成將很大一塊算力長期留給他們最有把握的那條產品線。然後回過頭來看我們目前陪著因為各種原因無法投資 RL 的客戶們，一起前進的過程中，時不時會提起的則是，務必現在馬上開始著手進行把企業工作流程盤點、情境上下文、資料採集、以及跨職能的權限準備好，讓接下來各大廠未來幾代 LLM 模型或是 agent 一旦登場，打家手上的公司或組織單位才有機會滾起來、動起來。AI 來我們這裡，我們自己也要準備好可以互接的通道，沒有準備的大家雖然還是有通用版本可以使用，但那就只是站在 baseline 上，然後望塵莫及。或說，只要你不往前看，其實你也沒有落後。老話一句，舒服就好。
📷 圖說 👉 我在大學時期也是充滿不解地想著，為什麼都是人類要去適應各種資訊系統，而不是軟體自己來適應我們人類的工作流程。當然二十幾年前有各種限制條件，但我比較受不了的是墨守成規，覺得事情就是這樣了，沒得改變了，而自己放棄改變的機會，而持續衝撞各種框框條條。雖然長大了稍微知道如何閃避障礙物，沒有進外商，但至少學著不會內傷，而保持著衝撞，也許我也是某種墨守成規，只是是我自己的成規。攝於 2016 出差，跟著在地人一起用午餐後的義式濃縮咖啡，他們說他們冰咖啡都這樣喝，我就說我也要嘗試看看。
不論衝撞還是墨守，走過都會留下痕跡，就像每天路過默默按個「👍」或「❤️」也都會留下痕跡，就試著保持前行的節奏，至少會瘦小腹（？
#OpenAI #JakubPachocki #ContinualLearning #ChainOfThought #AIAlignment #BringBack2016
✳️ 延伸閱讀
- 企業 AI 部署為什麼卡住？Mistral AI 的做法值得拆解
- 從 Vibe Coding 到 Agentic Coding：把話講清楚才是新戰場
- Coding Is Just Typing，然後呢？從 Explicit Programming 到 Implicit Programming
- 打造 Claude Code 的人，自己也被它改變了
- 與 AI 共存：Block 給組織重組的三條原則
✳️ 知識圖譜
(更多關於知識圖譜…)
✳️ 逐字稿與筆記
開場與節目總覽
- I definitely agree that continual learning is really the thing.
我非常同意 continual learning（持續學習）真的就是重點所在。 - It’s really the thing that we’re building.
它真的就是我們正在建構的東西。 - But I don’t really think this is like a problem that’s ignored and and off the path of what we’re doing currently.
但我真的不認為這是一個被忽略的、偏離我們目前在走的路線的題目。 - I think it is what we’re working toward.
我覺得這就是我們正在朝向的目標。 - » What are like the other research areas within alignment that you’re paying attention to or that you think are promising?
» alignment（對齊）領域裡，還有哪些研究方向是你特別關注或覺得有前景的？ - » A lot of the like longerterm challenge with alignment is about generalization.
» 對齊的長期挑戰，很多都落在 generalization（泛化）這件事上。 - What are the values that the model falls back on?
模型在退回預設時，依靠的價值觀是什麼？ - » What are the things that you need to figure out to be able to really make models work well in some of these other spaces?
» 要讓模型在這些其他領域也運作得好，你必須想清楚的是什麼？ - » I come back to this.
» 我還是會回到這一點。 - » Akopi is the chief scientist of OpenAI.
» Jakub Pachocki 是 OpenAI 的首席科學家。 - I think literally one of the most important people on the planet.
我覺得他幾乎是地球上最重要的人物之一。 - And today on Unsupervised Learning, I got to ask him literally everything that I’ve been thinking about and I know a bunch of people in the ecosystem have too.
今天在 Unsupervised Learning 節目裡，我幾乎把我一直在想、也知道生態圈很多人一直在想的問題全部問了他。 - We talked a lot about model progress, what’s required to make longrunning agents work, as well as the really interesting work Open AI has done in the AI for science world and the progress he sees in that over the next years.
我們聊了很多關於模型進展、要讓長時運作的 agent 真的能用起來需要什麼條件、還有 OpenAI 在 AI for science（AI 用於科學）領域所做的非常有趣的工作，以及他對未來幾年進展的預期。 - We talked a lot about how companies should be thinking about model building in this moment, when they should be doing reinforcement learning, how they should be thinking about the evolution of harnesses and the impact that will have.
我們聊了很多關於各家公司此刻該如何看待模型建構、何時該去做強化學習、該如何看待 harness（介接外殼）的演進，以及它會帶來的影響。 - We hit on a lot of his really interesting research, including the work he’s done around alignment, the work that OpenAI broadly has done around math competitions.
我們也聊到他許多非常有趣的研究，包括他在對齊議題上的工作，以及 OpenAI 整體在數學競賽上的成果。 - And we also talked about this focusing moment in OpenAI and what it means for the research organization and how he runs his team.
我們也聊到 OpenAI 此時的聚焦時刻，對研究組織意味著什麼，以及他如何帶領團隊。 - literally just such an awesome opportunity to talk to someone who is driving so much of the change that has revolutionized this space in the world.
能跟一位正在推動這麼多變革、徹底改變這個領域的人對談，真的是非常難得的機會。 - I hope folks enjoy this wide-ranging conversation as much as I did.
希望大家跟我一樣享受這場主題涵蓋非常廣的對談。
模型進展與時程
- I feel like you are the perfect person to talk to about all the questions everyone has in the ecosystem.
我覺得你是最適合來聊生態圈裡大家共同疑問的人選。 - Uh what’s you know happening with model progress.
模型進展上到底發生了什麼事情。 - A lot of companies are thinking about how they should be building things based on what’s happening with the models.
很多公司都在想，應該根據模型目前的進展來建構什麼東西。 - A lot of people at a societal level are thinking about the impact AI is going to have on science and broader society.
社會層級上也有很多人在想，AI 會對科學以及更廣的社會帶來什麼樣的影響。 - Uh and you’ve been at the forefront of the space for pretty much every generation of uh of improvement uh these past years and so really excited to have you on the podcast.
過去這幾年幾乎每一世代的進步，你都站在最前線，所以這次能邀請你上節目真的很令人興奮。 - » Happy to be here.
» 很開心來到節目。 - » I think I’ll start with one of the mo the juiciest things you said which is you know four months ago I think you and the open team talked about aiming for a system with research level intern capabilities by September of this year.
» 我就從你之前講過最耐人尋味的一段開始吧，四個月前你跟 OpenAI 團隊提到，目標是在今年九月前打造出具備「研究實習生等級」能力的系統。 - So coming up uh I think that’s uh what 6 months from now.
也就是差不多再六個月之後的事。 - and then a more fully automated AI researcher by March 2028.
然後在 2028 年 3 月前打造出更完整的全自動化 AI 研究員。 - And so I guess you know checking in four months later, how are you feeling about those timelines?
現在四個月過去了，你對這些時程的感覺如何？ - » Yeah, I think you know over I think over over the last months I think like the change that really happened is we’ve seen this explosive growth of coding tools.
» 嗯，我覺得過去這幾個月最實際的變化，就是我們看到寫程式工具出現了爆炸性成長。 - » Yeah.
» 對。 - » Um » it’s an understatement.
» 這樣講還算保守了。 - Yeah, we’ve definitely like really kind of gone um to a place uh in OpenAI where we use Codex for the um for the majority of um you know actual coding.
對，OpenAI 內部已經走到一個地步，我們實際在寫程式的工作，大多數都由 Codex 來處理。 - Um and so I think I think for most people like the kind of the act of programming has has has changed quite a bit.
所以對多數人來說，寫程式這件事本身已經有很大的改變。 - Um so I definitely see this as a signal that like you know something here is on track.
我把這當作是某件事情正在上軌道的訊號。 - The other kind of like very interesting update over the last few months to me has been the progress on the math research capabilities.
對我來說，過去幾個月另一項非常有意思的更新，是模型在數學研究能力上的進展。 - Uh also the results we’ve kind of seen in physics in other fields.
還有我們在物理以及其他領域看到的結果。 - I think I think this kind of level of capability this level of like ability to provide insight when combined with ability to access infrastructure ability to use maybe uh more computed test time that’s something that cod is using currently uh and very strong improvement in general intelligence which I also expect over over the next couple of months.
我覺得這種等級的能力、提供洞見的能力，搭配上存取基礎設施的能力、搭配 Codex 目前在使用的更多 test-time compute（推論期算力），再加上接下來幾個月我預期通用智慧會有非常強勁的提升。 - Yeah, it’s something we’re still very much planning for and very focused on.
是的，這仍是我們非常認真在規劃、也非常聚焦的事情。
定義研究能力
- » And how do you like know when you’ve you’ve gotten there?
» 那你怎麼知道自己已經抵達那個目標？ - like what’s like a a workflow you might look to to say hey okay I think we’ve got these you know research intern level capabilities » the the way I would distinguish you know a research intern from from full automated researcher uh is um the kind of span of time that that we would have it work um mostly autonomously or the kind of like specificity of the task that has to be given so I don’t expect uh you know we’ll have systems where you kind of just tell them oh like you know go improve
也就是你會看哪一種工作流程，來判斷說「好，我覺得我們已經具備研究實習生等級的能力了」？» 我區分研究實習生跟全自動化研究員的方式，是它能大致自主運作的時間跨度，以及任務必須講得多具體。所以我不預期今年會有那種你只跟它說「去改善 - your model capability go solve align ignment uh and you know and they will do it not this year you know I think we might get there at some point uh but I think for like more specific technical ideas like I I have this particular idea how to improve the models how to like you know run this evaluation differently I think I think we have the pieces that we mostly just need to put together Carpathy released you know a pretty viral version of of uh using some of these models to you know improve
你的模型能力」或「去解決 alignment」它就會自己去做的系統。我覺得某個時間點我們或許會走到那裡，但今年還不會。對於比較具體的技術想法，像是「我有這個改進模型的特定點子」或「用不同方法跑某個評估」，我覺得我們已經有大部分的零件，主要是把它們組合起來。Karpathy 發布了一個相當爆紅的版本，示範用這些模型去改善 - some of his uh you know obviously way less complex models than what you guys are building here but did that feel like generally in this uh you know in the spirit of of uh some of what these tools might look like.
他那些顯然比你們正在打造的模型簡單許多的模型，但這個大致上是不是在某種程度呼應這類工具未來可能的樣貌？ - » Yeah, I think it’s in the spirit.
» 對，我覺得就是那種精神。 - Yeah, I mean I I I expect it to look like a pretty continual evolution uh from kind of where Codex is now.
我預期從目前 Codex 這個狀態，會是一路持續演進。 - I think towards a bit more autonomy uh running for a longer time.
往更自主、能連續運作更長時間的方向走。
寫程式與數學對研究的影響
- Um but yeah, I I think I think we’ll see a lot of this sort of application.
我覺得我們會看到很多這類應用。 - I think in general we’ll see we’ll see like more autonomous and higher compute use of these models for different things.
整體上我們會看到這些模型在不同任務上變得更自主、消耗更多算力。 - you mentioned kind of like the math and physics side and obviously you’ve had these really impressive breakthroughs uh in math on you know uh some interesting like different kinds of competition uh you know problems maybe you know I think for our listeners it like intuitively makes sense how progress in coding directly translates to something like you know helping with AI research how does like math and physics progress like also tie into this » the the biggest role that like u you know
你提到數學跟物理這一側，你們在數學上明顯做出了很亮眼的突破，包括不同類型的競賽題目。對聽眾來說，寫程式上的進展直接能幫助 AI 研究這件事很直覺，但數學跟物理的進展又是怎麼跟這個綁在一起的？» 這些數學 benchmark（基準）對我們扮演最大的角色， - focusing on these math benchmarks has played for us as as a general yeah like benchmark and and and and a northstar for like how to improve this technology.
一直是作為一個通用的 benchmark、以及改善這項技術的北極星指標。 - Like math is very measurable, right?
因為數學很好衡量，對吧？ - It’s much easier to tell whether you’ve actually solved the math problem than whether you’ve even like produced a good uh you know piece of software and also it can get very hard right so you can have things where like it’s very definite whether you’ve solved them but it can be like arbitrarily pretty much hard to to actually solve them.
判斷你到底有沒有解出一道數學題，比判斷你有沒有寫出一份好軟體容易得多，而且它可以變得非常難，你可以有那種「有沒有解出來非常明確、但要解出來可以任意困難」的題目。 - You know, I would say like up until not too long ago like um you know, my perspective has been like well okay like we you know our models are not you know maybe able to solve like simple math problems.
就在不久之前，我對這件事的看法還是：好，我們的模型可能連簡單的數學題都解不出來。 - Okay, our models are able to solve simple enough problems but are not able to solve like IMO level problem.
或是：好，我們的模型可以解夠簡單的題目，但解不出 IMO（國際數學奧林匹亞）等級的題目。 - So clearly there is just like a gap in just like this uh you know intelligence of these models that like that that is very measurable very you know very easy to run at.
所以這些模型的智慧上顯然就有一道很好衡量、很好瞄準的落差。 - It’s very clear what we need to do and you know and this has be kind of our northstar for like reasoning models and so forth.
我們要做什麼非常清楚，這也就成為我們 reasoning model（推理模型）等等的北極星。 - Now of course um that is changing quite a bit right and we are um you know we have kind of reached these milestones that we’ve been working towards of like yeah IMO goals level solving IMO problem six and you know and making forests into research level mathematics um and you know from this point I think I think there still is uh you know there definitely still is utility like continuing to measure progress on this I think there’s also like you know there’s definitely like transfer that that
當然現在這個情況已經有相當大的變化，我們已經達到一直在往前推進的里程碑，包括 IMO 金牌等級、解出 IMO 第六題、以及在研究等級的數學上有進展。從這裡開始，我覺得繼續衡量這方面的進展當然還是有用的，而且我覺得 - you can get from like getting better at mathematical reasoning to getting better at AI research.
在數學推理上變強，會明顯地轉移到 AI 研究上變強。 - You know, a lot of our uh best researchers uh are uh you know mathematicians we’re training or from other kind of theoretical fields.
我們很多最頂尖的研究員其實是受過數學訓練的數學家，或是來自其他理論領域。 - But definitely we are uh you know we are very much uh changing how we think about you know these nerf stars and we are very focused on how the models the next models that we’re producing are actually useful in the real world you know useful you know especially for a research but also for other kind of economically valuable activities and for other uh fields of science uh and especially maybe more applied sciences.
但我們確實在很大幅度地調整我們對北極星的想法，現在非常聚焦在：我們做出來的下一代模型在真實世界是否實用，特別是對研究、對其他具有經濟價值的活動、以及其他科學領域，尤其是比較應用端的科學。
AI 在科學上的應用與經濟影響
- And the reason for this shift is because we believe the models are now capable enough, not as smart as people and always, but capable enough to actually materially change the economy, change how things are done.
這個轉向的原因是我們相信模型已經夠強，不是在每件事上都比人聰明，但強到足以實質地改變經濟、改變事情的運作方式。 - And so, uh, yeah, we feel a lot of urgency about that.
所以我們對這件事感到很大的急迫感。 - » In the early days, uh, picking a domain like math that is so, uh, hard to solve, but then easily to verify whether you did it, like it’s kind of the the perfect place to get started.
» 早期選擇像數學這種「解起來很難、但驗證有沒有解對卻很容易」的領域，幾乎就是最完美的起點。 - And I think code obviously shares a lot of attributes to that.
我覺得寫程式顯然也具備許多相同特質。 - You know, uh possible to check uh and verify and great for reinforcement learning.
可以檢查、可以驗證、非常適合強化學習。 - I think one question that a lot of people are are thinking about is okay, we’ve seen reinforcement learning work incredibly well in these domains where you can verify it rather easily.
我覺得很多人在想的一個問題是：在這些容易驗證的領域，我們看到強化學習效果出奇地好。 - A lot of, you know, valuable tasks in the world, medicine, law, finance, you know, there’s some level of of the ability to do that, but it’s certainly not to the same extent that math and code are.
世上很多有價值的任務，像是醫療、法律、金融，驗證的難易度雖然有某種程度，但絕對不到數學跟寫程式這種程度。 - And so I think a lot of people are trying to figure out, you know, are we going to see similar improvements?
所以很多人想知道，我們會不會看到類似的進步？ - You know, obviously code and math the the rates of improvement have been so astronomical and shocking.
畢竟寫程式跟數學的進步速度已經到了很誇張、很震撼的程度。 - » Yeah, I definitely expect so.
» 我絕對預期會。 - Um I think an interesting duality that we think about a lot is um you know for this more general task for these tasks are kind of harder to evaluate.
我們常在想一個有趣的對偶性：對這類比較一般、比較難評估的任務， - They share a lot lot of common uh commonalities with um just longer horizon tasks, right?
它們跟長週期任務其實有非常多共通點，對吧？ - Because if you think about even like a very well specified math or coding problem again like if it’s it’s something that you need to work on for like a year then uh you know even it’s very clear what the criteria of success are in the long term like what to do on your first day of working on it is a pretty open-ended problem.
因為就算是一個規格非常明確的數學或寫程式問題，如果你要花一年才能完成，就算成功的長期標準非常清楚，你上工第一天要做什麼其實也是個相當開放的問題。 - Yeah.
對。 - And so I I kind of believe this these difficulties coincide and they’re very clearly the next the next frontier uh for for how these systems develop.
所以我相信這些困難是同一件事的不同面向，而且它們很明顯就是這些系統接下來要推進的前沿。 - And I think we’ve definitely seen very encouraging signs both on just like our ability to scale RL on these more general domains.
我覺得在把 RL 擴展到這些更一般領域的能力上，我們已經看到非常令人鼓舞的訊號。
評估非標準任務上的進展
- And I I think also like we can we can scale um efforts that that that that that’s a lot of promise.
我也覺得我們可以擴展這些嘗試，這裡面有非常多潛力。 - » In these other domains, it feels like one of the hardest things to know is just what was success in a task, right?
» 在這些其他領域，最難判斷的就是：一項任務的成功究竟長什麼樣子，對吧？ - And you can imagine you know there’s going to be you know whatever the problems you are that are facing code of math that are short-term tasks and then longerterm tasks feels it will be amplified in the space that is you know outside of those right where a short-term uh legal task or medical task may be harder to run thousands of iterations on right and figure out you know was that done correctly and then those longer term tasks like even harder I’m curious like how you even conceptualize
你可以想像，現在在寫程式或數學上遇到的那些短期任務問題，到了長期任務時會被放大，而在寫程式跟數學以外的領域更會被放大。像是一個短期的法律任務或醫療任務，可能就很難跑上千輪迭代，也很難判斷結果對不對，長期任務就更難。我很好奇你怎麼在概念上看待 - that research challenge like what are the things that need to be that you need to figure out to be able to to really make models work well in some of these other spaces.
這個研究挑戰？要讓模型在這些其他領域也能用得好，你們必須想清楚什麼事？ - » Yeah, I think I think I I come back to this reality of like how do we make the models work for a very long time and how do we teach them to evaluate kind of partial progress.
» 嗯，我又會回到這個現實：我們怎麼讓模型能運作很長時間，以及我們怎麼教它評估自己的部分進度。 - » Yeah.
» 對。 - I mean I think if if you look at like even outside of RL like like where that sort of progress on longer horizons is coming from right like I mean as the models kind of become more consistent from just like pure supervision in pre-training um they uh they gain some idea of like you know oh what what does like a good partial artifact here look like and so I think I think even if we weren’t like scaling RL very meaningfully we would see an alongation of these horizons over time yeah it’s
我的意思是，就算在 RL 之外，長週期上的這種進展是從哪裡來的？當模型因為預訓練純粹的監督學習變得更一致，它們就會對「這裡一個好的部分產物應該長什麼樣」有一定的概念。所以我覺得就算我們沒有非常大幅度地擴展 RL，也會隨時間看到任務長度的拉長。當然 - definitely um you know a research challenge to like to figure out how to like leverage this new ideas from RL and so forth to to apply this to general domains.
怎麼把 RL 的新想法應用到通用領域上，絕對是個研究挑戰。 - But I’m quite optimistic about that.
但我對這件事相當樂觀。
模型自主性與泛化
- » Yeah.
» 對。 - And it’s interesting.
這很有意思。 - It sounds like part of your mental model is like the models themselves being able to check progress with some at some sort of cadence that is, you know, reliable enough from the outside at least.
聽起來你心裡的模型是：模型本身能以某種節奏自我檢查進度，而且至少從外部看起來是夠可靠的。 - It’s not totally clear if we’ve seen like generalization in RL yet.
我們到底有沒有看到 RL 有泛化性，其實還不是很明朗。 - feels like we yeah clearly you seem to have some techniques that really optimize models around whatever we choose to focus on but it’s like almost feels like an older school version of of ML of like one one thing at a time is that like you know I guess would you agree with that characterization and like you know how do you kind of see this this current climate » well we are buying a lot of compute right because we we don’t I mean we still believe a bit less and we believe you know more than
感覺你們顯然有一些技術，能針對我們選定要聚焦的目標把模型優化得非常好，但感覺有點像比較舊派的 ML，一次處理一件事。你會同意這個說法嗎？你怎麼看目前的氛圍？» 嗯，我們確實在買進大量算力，因為我們並不……我是說我們相信的還是稍微少一點，但我們某種程度上比以往更 - ever to some degree yeah we’ve seen you know, new techniques and I think new ways to scale, but like that that is kind of the the lens through which we’ve been viewing things.
相信這件事。我們確實看到新的技術、新的擴展方式，但這就是我們觀察事情的鏡片。 - Yeah, I think there is a certain amount of complexity that we needs to grapple with and kind of everyone needs to grapple with because, you know, we’re no longer really like purely building like um um you know, brain the sky that’s completely isolated from the real world, right?
我覺得有一定程度的複雜度是我們必須面對、大家都必須面對的，因為我們不再只是在打造一顆跟真實世界完全隔離的空中大腦，對吧？ - Like if you actually you know if you want this model to do like medical research if you want it to cure cancer at some point it needs to like learn about the real world is a meaningful way you know maybe conduct some experiment and learn from its results and for that you you need to figure out how to actually connect it right and that is going to involve something that is yeah that that goes in the direction you described but I I don’t think that goes counter to actually scaling the the like
如果你真的要讓這個模型做醫學研究、要它終有一天能治好癌症，它必須以某種有意義的方式學習真實世界、也許真的執行實驗、從結果中學習。要做到這件事，你得想清楚怎麼把它接上現實世界，這的確會牽涉到你描述的那個方向，但我不認為這跟繼續擴展 - finding and scaling the simple algorithms that that we’ve been developing.
那些我們一直在找、一直在擴展的簡單演算法有什麼矛盾。 - I feel like I talk to a lot of companies and like I one of the main questions everyone seems to be asking these days is like should we be doing you know our own reinforcement learning like take an open source model and like we have some data on a task that people do.
我跟很多公司聊過，大家最近問得最多的問題之一就是：我們該不該自己做強化學習？拿一個開源模型，我們手上有一些某個人類在做的任務資料。 - um we have evals cuz we know our domain pretty well like is this something that makes sense for us to do or like should we just wait for the models to continue to get better at at some of these things.
我們也有評估，因為我們很懂自己的領域。這件事我們自己做有沒有意義？還是應該直接等模型繼續在這些事情上變強？ - you know what advice would you guess would you give for like the many builders that listen to the podcast as they think through you know uh the extent to which they invest on the on the reinforcement learning side reinforcement learning definitely can be a very data efficient way to like really improve the model as some sort of task right there is a much more data efficient way of learning that we know right which is like learning in context right and this is maybe the most fundamental way
對於聽這集的很多建構者在思考要不要投入強化學習，你會給他們什麼建議？強化學習在特定任務上確實可以是資料效率非常高的做法。但我們知道還有一種資料效率更高的學習方式，就是 in-context learning（上下文學習）。這或許是最根本的一種 - that people you know teach these models you just prompt them with like examples with with with instructions for what you want I expect that learning is going to get much better over time.
我們教模型的方式：用範例、用你希望的指令給它提示。我預期這種學習會隨時間變得越來越好。 - And so I think it definitely really matters that the models can adapt to your context.
所以模型能不能適應你的 context（情境），真的非常重要。 - They can adapt adapt to kind of the the kind of tasks you care about.
它能適應你在乎的那種任務。 - So I think that will be very important.
所以我覺得這會是很重要的。 - I’m not sure if like you know replicating the kind of current a pipeline is going to be like the right way to go about it.
我不確定複製目前那種 RL pipeline（管線），會不會是正確的方法。 - But yeah, it’s definitely a problem that that we’re thinking about.
但這確實是我們一直在思考的問題。
Harness 與介面演進
- » Yeah.
» 對。 - So it’s almost like yeah you still have to do the work like you still should you know figure out what the eval are that matter gather the data the examples but like it may just turn out in the future you’re far better off just feeding that into this context than trying to like do anything on on you know your own model.
也就是說，你還是得做功課，還是得想清楚哪些評估重要、去蒐集資料與範例，但未來可能最好的做法就是直接把這些丟進 context，而不是在自己的模型上做什麼動作。 - Yeah, I think I think that’s quite plausible.
對，我覺得這很有可能。 - And I think that like you know obviously people have seen the success of of tools like Codex which I know you know you’ve obviously been a key part of and um and wondered like you know hey do we need to build like our own kind of you know should we build our own harnesses or our own ways of of using these things or you know uh for for our own domains whether it’s like you know uh legal or finance or or healthcare or do we kind of just like take the harnesses that the large models do um and
大家看到像 Codex 這類工具的成功，當然會想：我們該不該在自己的領域（法律、金融、醫療等）做自己的 harness、自己的使用方式？還是直接拿大型模型的 harness - and kind of use them within you know with with the context that we have.
加上我們自己的 context 來用就好？ - uh any any thoughts around like that » like the implementation of the harness shouldn’t really be a limitation for a very long time.
對這件事你有什麼想法？» harness 的實作本身，不應該是一個長久的限制。 - I think we’ll be able to get like much more general harnesses that people can use for uh for all sorts of other domains.
我覺得我們會做出通用得多的 harness，讓大家在各種不同領域都能用。 - I mean I think codex is pretty good actually if you try using it for things beyond coding.
其實你拿 Codex 來做寫程式以外的事情，它已經相當不錯了。 - » That’s so interesting.
» 這很有意思。 - Like a much more general harness being something that’s almost like uh adaptive to or like just works across whatever the you know specific set of tools you have in your domain or specific set of things you want to expose to the model.
一個更通用的 harness，幾乎就是能適應、或直接跨越你領域裡特定的工具集、或你想暴露給模型的特定東西。 - » Yeah.
» 對。 - I mean I I think and you know I think it’s also worth thinking about like you know why like you know what what what is kind of the kind of ultimate interface that we want to interact to the model with.
我覺得也值得思考一下，我們最終想要用什麼樣的介面來跟模型互動？ - So, so the model gives some the models gives some UI hard forensicness, right?
現在的模型具有一定程度的 UI 能力，對吧？ - They can build their own UIs.
它們可以自己建立 UI。 - They can kind of do things that uh you know people would find very timeconuming.
它們可以做一些人類會覺得很花時間的事情。 - Um but I yeah I definitely think there is also just like a lot of space to kind of enable the models to access like the current interfaces that we use for for people right.
但我也非常認為，讓模型存取我們現在給人用的既有介面，有很大的空間。 - So I think like we want to have um um you know AIs on Slack for example or that that are kind of plugged into our our context and uh and yeah and are able to to learn from it and a able to kind of yeah to realize this existing things right so definitely like there is some meet in the middle here but definitely I believe like longterm like uh you know like by default the AI should kind of meet you where where you are uh and if Not that would be because it kind of it has new abilities, not
比方說，我們想要有 AI 在 Slack 上，接進我們的 context、能從中學習、能去呼叫既有的東西。這裡有個折衷，但我長期相信，預設上 AI 應該來到你所在的地方；如果不是這樣，那會是因為它有了新能力，而不是 - because it has limitations.
因為它有限制。
定義有意義的研究發現
- » Yeah, it’s an interesting point that basically today it feels like these harnesses are so bespoke to certain environments, but like over time as you add more and more skills and tools and models can navigate uh across those effectively, it’s like there just be a general like you know the way humans have uh that that makes a tremendous amount of sense.
» 這個點很有趣：今天這些 harness 都還是為特定環境客製的，但隨著你加入越來越多技能跟工具、模型又能有效在其中穿梭，它就會變成像人類那樣的通用介面，這很合理。 - I guess I’m curious like you know you uh obviously I’m I’m sure like every day you see kind of crazy stuff on the research side at this point like what are the milestones that are like still meaningful to you as you think about like it would be pretty crazy if I you know uh did a run one day and saw like X or Y like what are the things you’re paying most attention to?
我很好奇，你每天在研究端肯定都看到一堆誇張的東西，現在還有哪些里程碑對你來說是有意義的？「如果哪天跑一次看到 X 或 Y，那就真的誇張了」這種事，你現在最關注的是什麼？ - » Yeah.
» 對。 - Um I mean at this point it really is about um research right like is it about it is about can the model discover new things can it execute on like a longer horizon um research problem.
現階段真的就是研究本身，也就是：模型能不能發現新東西？它能不能在一個更長週期的研究問題上執行下去？ - » It’s almost like looking for some sort of insight that you’re like oh someone on my team had come up with that that would I’ve been pretty intrigued by Yeah, we we’ve actually had like some minor uh um but I think I think quite impactful ideas uh come from uh even like GPT 5.2 Pro uh that that we’re using entirely.
» 幾乎是在找某種洞見，你會想：這個如果是我團隊裡某個人想出來的，我會很感興趣。對，我們其實已經看到一些小的、但我覺得影響不小的點子，是來自我們內部在用的 GPT 5.2 Pro 這類模型。 - But you know, I think it’s still very very small compared to where I expect it to be.
但相較於我預期它能達到的程度，現在還是非常非常小。 - » Yeah, I mean it seems like almost inevitably like these models are going to get better.
» 對，這些模型幾乎必然會變得更強。 - They will be used in research.
它們會被用在研究上。 - They’ll be used in science more generally.
更廣泛地也會被用在科學上。 - You’re like one of the first people interacting directly with these models as like research partners almost at this stage.
你幾乎是最早一批把這些模型當研究夥伴直接互動的人。 - anything like you’ve learned around the right way to do that or do you think about like what a research organization you know as these models continue to get better might look like?
關於怎麼做才對，你有什麼學到的嗎？或者你怎麼想像，隨著模型繼續變強，研究組織會長什麼樣子？ - Yeah, I I I think we’re definitely kind of at um at a transition point where kind of the shortterm immediate quality of the model uh is about to be a quite determining factor for the pace of our research progress because the models are going to drive a lot of that.
我覺得我們正站在一個轉折點，模型當下的短期品質即將成為決定我們研究進展速度的關鍵因素，因為模型本身會推動很大一部分的工作。 - And so that definitely requires um you know rewiring some intuitions about how to um run a research organization.
所以這需要我們重新調整一些經營研究組織的直覺。 - Uh you know normally you kind of try to not be too focused on like immediate quality.
通常你會盡量不要太聚焦在當下的品質。 - you try to be much more focused on like the longer term.
你會更聚焦在長期。 - I think we have like a lot of very exciting uh stuff queued up that we are kind of working towards but I feel a lot of urgency to kind of yes to actually » u execute on it and to actually use this advances in model intelligence to um accelerate research on the AI and especially AI alignment.
我覺得我們排了很多很令人興奮的事情在往前推，但我有很大的急迫感要把這些真的執行出來，並且利用模型智慧的進展來加速 AI 研究，尤其是 AI 對齊。 - Yeah, it’s such a fascinating point because I’ve heard you talk before about running a research organization and I feel like in the past it was like giving people the space to, you know, pursue a lot of things that weren’t like directly, you know, hey, this is for a month or two months of progress, but it’s like what are the ideas that are really going to drive things forward,
對，這個點很有意思。我以前聽你講過怎麼帶研究組織，感覺以前你會給大家空間去追求很多不是「這是給一兩個月進展用」的題目，而是「哪些想法才真的會推動事情往前」的題目。 - but it makes total sense that we’re in a time now where uh you’re like, look, everything we do will be so much better if we just focus on this in the in the short term and make it better.
但現在這個時機下，你會說：我們做的每件事，只要我們短期內聚焦把這個做好，全部都會好上一大截，這非常合理。 - It must be like fascinating to navigate uh that and like these maybe further off research ideas at the same time and like running an organization.
一邊處理這個、一邊同時顧著比較遠期的研究想法、又要經營組織，想必很有意思。 - » Yeah.
» 對。 - Yeah.
對。 - It’s definitely Yeah, it’s definitely something we we spend a lot of time on with Mark nowadays.
這真的是我跟 Mark 現在花很多時間在處理的事。 - Yeah.
對。
算力的策略性分配
- » Right now you have um you know a a ton of compute as a company, but you obviously you have great scaling laws on the pre-training side, you have great scaling on the RL side, you have probably lots of experiments going on that have nothing to do with either of those vectors, but are like interesting new ways.
» 你們公司現在擁有超大量算力，預訓練上有很好的 scaling law，RL 上也有很好的擴展性，可能還有很多跟這兩條路都無關、但是有趣的新實驗在跑。 - How do you even think about like allocating compute across all of this stuff?
你們到底怎麼在所有這些方向之間分配算力？ - » Yeah, it can get very complicated, right?
» 這會變得非常複雜，對吧？ - Because there’s so many things that we need to do.
因為我們要做的事情真的很多。 - One thing we’ve been one kind of discipline we’ve started keeping is we um we try to make sure we just like explicitly budget like a large chunk of our compute to the most scalable methods to the things that we believe are the most responsible for driving general model intelligence.
我們開始維持的一條紀律是：明確地把很大一塊算力預算分配給最具擴展性的方法，也就是我們相信最能推動通用模型智慧的那些做法。 - And you know even if it’s not the most efficient allocation of comput at all times because you know if you’re allocating so much compute to like one experiment or like one set of experiments you know there’s so many things you can accelerate a little bit of that compute elsewhere.
即使這不見得時時都是最高效的分配，畢竟你把這麼多算力押在一個實驗或一組實驗上，同樣的算力拿去別的地方可以順勢加速一堆東西。 - Uh but you know but I think it’s easy to kind of like with all the all all the all the interesting and important things that we’re doing I think it’ll be very easy to kind of partner all of it and like not not really end up doing the things that we believe are most important.
但我覺得我們要做的有趣又重要的事情這麼多，很容易一個一個都分一塊算力，然後最後反而沒有真正把我們相信最重要的事情做出來。 - You definitely want to like understand the kind of empirical evidence.
你一定要理解手上的實證證據。 - You definitely want to make sure your evaluations are in order and the kind of experimental rigor is there.
你一定要確保評估是嚴謹的、實驗的嚴謹度到位。 - And then you also want to apply some regularization based on like okay do we understand this method?
然後你還要再加上一點正則化思考：我們真的理解這個方法嗎？ - Do we actually expect it will scale?
我們真的預期它可以擴展嗎？ - Do we expect this is something you can actually build on in the future?
我們預期這個東西未來可以在上面繼續堆疊嗎？ - Is this kind of a one-off?
這是不是一次性的東西？ - Right.
對。 - And I think and based on that uh determine the priority.
我們就依據這些來決定優先順序。 - » Yeah, it’s so interesting.
» 對，這很有意思。 - probably find all the yeah ways that you like know you could improve things but they feel maybe like uh off off a little bit to the side of where you think the overall arc of progress is and so you end up leaving some of these like lowhanging fruits to some extent because really the most important thing is finding the future direction and then the scaling within that and uh devoting compute toward that obviously the the place where we talked about codeex a lot and and the success of coding
你大概會找到一堆可以改善的方式，但它們感覺跟你認為整體進展的大方向差了一點點，所以你某種程度上就把這些低垂的果實留在那裡。因為真正最重要的是找到未來的方向，然後在那個方向上擴展、把算力押過去。當然，剛剛我們聊很多 Codex 跟寫程式的成功就是很好的例子。
產品聚焦與研究優先順序
- and it feels like you know last year was like the year of just incredible hill climbing on on coding I’m curious you know obviously Codex has been a super successful product in many ways like anthropic was kind of first to this market you know claude code you know was it was a dominant product there what do you kind of like you know reflecting on that I guess like what do you make of the success anthropics had in this space » yeah I think I think it’s a matter of you know really focusing
感覺去年就是寫程式這一塊驚人爬坡的一年。Codex 在很多面向上顯然是非常成功的產品，但 Anthropic 算是這個市場的先行者，Claude Code 曾經是這個領域的主導產品。你回頭看，對於 Anthropic 在這個空間的成功你怎麼想？» 我覺得這就是一個 - your product direction or on where where you believe the kind of the the next application of the technology is right and um you know if you look at the kind of priorization we’ve had on the on our product right I mean we have been right like working on on cutting products but they have kind of been like a secondary thing right compared to like our main priorities and the interesting thing is that is not very reflective of like the priorities of the research organization within open AI uh I
專注問題：你要把產品方向聚焦在你認為下一個技術應用會落地的地方。如果你看我們在產品上的優先順序，我們確實有在做寫程式產品，但相較於我們的主要優先順序，它們一直是次要的。而有趣的是，這並不能反映 OpenAI 研究組織內部的優先順序。 - think you know given that like we’ve kind of had this you know explosive success of charg you know charging as it was you know I I think charging quite a bit and it’s going to evolve quite a bit but as it was in 23 right is this particular you know product that’s maybe not, you know, I think it’s definitely quite aligned with our vision of like where AI is going, but but like it’s not really like the like representative of like everything that that that that it enables.
我覺得因為我們看到 ChatGPT 的爆炸性成功（ChatGPT 已經變很多、之後還會繼續演進），但就 2023 年當時的形態而言，它跟我們對 AI 走向的願景相當一致，但並不真的代表它所能帶動、所能開啟的一切。 - And so the majority of like our work in research has been focused on like that that future thing.
所以我們研究工作的絕大部分，都是聚焦在那個未來的東西。 - And I think increasingly it has decoupled from our our our kind of like short-term product strategies, right?
而且我覺得它跟我們的短期產品策略已經越來越脫鉤了。 - Yeah.
對。 - I’m very kind of um confident about um the things we’ve been building and the things we we we are building on on on the research on the model intelligence side.
我對我們在研究端、模型智慧端正在打造的東西非常有信心。 - You know, a lot of our our rep refriation and increased focus on the on the product side is about actually kind of getting to deploy them and the belief that actually they are uh the thing that really matters now.
我們在產品端重新聚焦、加大投入，其實是為了把這些東西真的部署出去，並相信它們現在就是真正重要的東西。 - » Yeah.
» 對。 - And now it feels like you know the uh clearly the whole company priority you know is so locked in and focused on this and you’ve seen just incredible improvement in codecs in recent months for all the developers that listen to the podcast like if again it’s almost like hard to comprehend like what the world looks like as these models keep hole climbing on longer and longer tasks like what do you think will look different in their lives or like how will they be using codecs in you know three
現在看起來整間公司的優先順序顯然都鎖定在這上面，Codex 最近幾個月也看到驚人的進步。對所有聽這集的開發者來說，當模型在越來越長的任務上持續爬坡，世界會長怎樣幾乎很難想像。你覺得他們的生活會有什麼不一樣？或在三個月、 - six months.
六個月後會怎麼用 Codex？ - I realize 3 months and six months are very different timelines in this world, but take whichever uh whatever in between point you’d like.
我知道在這個世界裡三個月跟六個月是完全不同的時間尺度，你選任何一個中間點來回答都行。
開發者工具的未來
- » I would expect um just a a gradual increase in just the level of autonomy uh you feel comfortable uh foring the model just the the fagness of description that can work with you know the level of supervision it needs.
» 我預期的是：你願意把工作交給模型的自主程度會逐步提升，它能處理的指令模糊度會變大、它需要的監督程度會下降。 - I think we’re not very far for models that can work autonomously for a couple days.
我覺得能自主運作好幾天的模型，我們已經不遠了。 - Um maybe use quite a bit more computer than they’re using now and produce much higher quality artifacts on their own.
它可能會消耗比現在多很多的算力，並自行產出品質高得多的產物。 - Do you have a gut instinct on like what like you know there’s always been this question of like will the world you know do you need that software engineering skill set to supervise these models running for a few days or like hey does it turn out at some point of like being able to run for a while you know anybody can can use coding agents and supervise them to to some sort of output.
你有什麼直覺嗎？一直有一個問題是：要監督這些連續跑幾天的模型，世界是不是還需要軟體工程的技能？還是到了某個時點，當它能持續跑一段時間之後，任何人都能用 coding agent、並監督它產出結果？ - I mean I think definitely for like a lot of outputs you already don’t need much experience right I think I think still the distinction I would draw between like you know an intern here and like really an autonomous researcher software engineer would be that like if you want to build something bigger like you know you probably still want to apply supervision you still kind of want to have like an overarching thing you want to recognize like what what what building blocks fit in and what which
我覺得針對很多產出，你已經不太需要經驗。但我會區分實習生跟真正自主的研究員工程師：如果你想打造更大的東西，你大概還是會想要加上監督、還是會想要有一個整體的思考、辨認哪些組件適合放進去、哪些 - don’t but yeah I definitely expect that like that desired skill set uh to shift quite a bit over Yeah, » towards towards this like more general uh vision setting.
不適合。但我絕對預期被需要的技能會有很大的移轉，» 往比較通用的「設定願景」那個方向。 - » You know, I guess on on the on the research side, I feel like there’s been uh you know, maybe maybe like a month ago, I feel like all anyone could talk about was continual learning and there’s just you know, it was in the Zeitgeist.
» 我覺得在研究端，大概一個月前，所有人都在聊 continual learning，整個時代氛圍都是這個。
持續學習與 RL 浪潮
- There’s all these neolabs starting to go focus on continual learning.
冒出一堆新實驗室開始聚焦在持續學習上。 - Some folks left OpenAI to go focus on that.
有些人也離開 OpenAI 去做這件事。 - Um I’m curious like you know I think it part maybe part behind that is a belief that like you know uh RL alone you know either won’t get us there or will get us to like some level of very inefficient scaling and it’s kind of different than the way you know humans learn.
我很好奇，背後一部分的想法可能是：只靠 RL 要嘛到不了那裡，要嘛會帶我們到一個非常低效率的擴展層級，跟人類學習的方式差很多。 - I think even I’ve heard you say before like that you know RL is still very different today than the way that humans learn.
我甚至聽你以前說過，現在的 RL 跟人類學習的方式還是非常不一樣。 - What’s your take on on like that you know that whole movement?
你對這整個浪潮的看法是什麼？ - Yeah,
嗯， - I I am a little bit confused by it because you know in my mind like the whole kind of like excitement that like we’ve had I mean even even if you look at the titles of like the GPT uh you know three paper right like it is that like oh you know this class of models is actually capable of continue learning right it’s capable of like learning uh um learning to learn in context right that has been really you know the driving force behind the kind of excitement to like scale these GPD models
我對這個浪潮有點困惑。因為在我腦袋裡，我們一直以來的興奮感，甚至你看 GPT-3 論文的標題就知道，就是這一類模型其實具備持續學習的能力，具備 learning to learn in context（在情境中學會學習）的能力，這一直是我們之所以興奮去擴展這一系列 - further.
GPT 模型的驅動力。 - That has been like the premise for why we really need to teach them with RL like learn in context more efficiently.
這也是我們之所以需要用 RL 去教模型「更有效率地在情境中學習」的前提。 - And so I definitely agree that continual learning is really the thing, right?
所以我完全同意 continual learning 就是重點所在。 - Like it’s really the thing that we’re building, but I I don’t really think this is like a problem that’s like, oh, you know, it’s kind of ignored and off the path of what we’re doing currently.
這真的就是我們正在打造的東西，但我真的不認為這是個「被忽略、偏離目前路線」的問題。 - I think it is what we’re working towards.
我覺得它就是我們正在朝著走的方向。 - » Yeah.
» 對。 - Like in your mind, this is like the single best path to get there is to continue to kind of scale uh the pre-training in RL.
在你心裡，抵達那個目標最好的唯一路徑，就是繼續擴展預訓練跟 RL。
讓模型處理長期任務
- I think that is kind of how we’ve made the most progress on this problem so far and you know I think there are I think that there definitely are like more ideas more steps um I think also a lot of improvements that will just come from scale » yeah and I guess like you know we have a lot of folks listening that maybe have you know have been able to do a lot of simpler things with these models and then they try to do like some of these more complex you know I don’t know call it 100 step or
我覺得這就是我們目前在這個問題上取得最多進展的方式。當然一定會有更多想法、更多步驟，我也覺得很多進步會單純來自規模的擴展。» 對，很多聽眾可能用這些模型做簡單的事情沒問題，但要去做那種比較複雜、假設叫它 100 步或 - longer term tasks and they’re like oh you know the the models don’t work for this yet and I think it’s harder you on the inside constantly feel this improvement but for them it feels like hey this is like night and day away from you know being able to do this much longer thing.
更長期的任務時，他們會說：模型還搞不定這個。你在內部會一直感覺到模型在進步，但對他們來說，感覺「要能做到這種更長的事情」跟現況簡直是天壤之別。 - How do you kind of articulate to them I guess the set of things that need to be true for these like much longer steps to happen.
你會怎麼向他們說明，要發生這種更長步驟的任務，需要哪些條件必須成立？ - Is it around kind of checking in more often as you were talking about before or I feel like there’s just this belief uh among the research community of like oh all of these tasks will be solved in the next year or two and then in the wild a lot of people maybe not totally groing that like improvement line that we’ve been seeing.
是像你剛剛講的更常自我檢查嗎？研究社群普遍相信這些任務在未來一兩年就會被解決，但外面很多人其實沒有完全感受到我們一直在看到的進步曲線。 - » Yeah.
» 對。 - I mean I think a lot of that prediction comes from just looking at like historical improvement lines, right?
我覺得很多預測來自看歷史上的進步曲線。 - And but I think increasingly we can we can roughly see the the the the shape here.
但我覺得我們越來越能看清楚這個形狀。 - I do think a lot of this is about just the models becoming intelligent enough to recognize like whether you know they’re making progress.
我覺得很大一塊就是模型變得夠聰明，能自己辨認是否有在取得進展。 - Um I think some of this is like yeah this very kind of pragmatic work of like are the models actually you know can they actually access you know all the context all the files all the infrastructure they need to do the work you want them to do which yeah I remember like in the past when we were discussing you know the kind of the the road map uh that we’re taking with RL you know I definitely view like okay we just need to teach the model to kind of reason with its own tokens as kind of the
還有一塊是相當務實的工作：模型到底能不能存取它需要的所有 context、所有檔案、所有基礎設施來完成你要它做的事？以前我們在討論 RL 路線圖時，我一直覺得優先要做的就是教模型用自己的 token 做推理， - priority and then of course we’ll need it to use tools like the environment, you know, at some point we definitely need to teach it to see, right?
然後當然它需要用工具、跟環境互動。某個時間點我們絕對要教它「看」。 - At some point, we need to teach it to use a physical body, right?
某個時間點，我們要教它使用實體身體。 - Like, but like uh yeah, I mean, I think we’re definitely like well into the stage where, you know, really needs to like interact with the environment and it really needs to see uh and you know, someday soon we’ll we’ll really cover about robots, but yeah.
我覺得我們明顯已經走到那個階段：它必須跟環境互動、必須能看，不久的某天我們也會真的要去處理機器人這件事。 - » Yeah.
» 對。 - I mean, it does feel like a lot of the times when I hear people complain about, oh, a model can’t do X or Y, it’s like literally just because you haven’t fed, you know, or connected it to systems or fed enough context into it.
的確，每當我聽到有人抱怨「模型做不了 X 或 Y」，很多時候根本只是因為你沒把它接上系統、沒餵給它足夠的 context。 - Actually, I do wonder if like context was universally applicable and able to flow into these things.
我其實在想，如果 context 可以普遍適用、並且能在這些系統之間流動， - Like I feel like a lot of these problems would actually just be solved with today’s models.
我覺得很多問題其實今天的模型就能解決。
AI 在數學與科學發現上的應用
- You know, I want to talk about some of the AI for science stuff um that you guys have been working on.
我想聊聊你們在做的 AI for science 這一塊。 - And one thing in particular, you know, I feel like the coding stuff is something that everyone feels very viscerally um you know, in every company they’re using these tools and getting tons of productivity.
特別是有一件事，寫程式這件事大家都非常有感，每家公司都在用這些工具並獲得大量生產力。 - You know, on the math side, not all of us competed in in in IMO competitions and uh necessarily have as much of like an intuitive feel for some of these breakthroughs.
但在數學這一側，不是每個人都參加過 IMO，所以對這些突破不見得有直覺感受。 - And so one of them I know that was really interesting that you guys did is you use some compelling work around like first proof, right?
你們做的一件很有趣的事是在 FrontierMath 的證明題上的工作，對吧？ - And I think these are like very different problems than kind of traditional competition math.
這些題目跟傳統競賽數學很不一樣。 - I wonder if you could just speak a little bit to that because I think it’s just a space that our listeners might be less familiar with and kind of less familiar with understanding the implications of models being able to do pretty cool work here.
你可以稍微聊一下嗎？這是聽眾可能比較不熟悉的領域，也比較不熟悉模型能在這裡做到酷事情的意涵。 - Yeah, I mean you know I think yeah I I was very excited with the first proof challenge and you know again like I I kind of you particular one is kind of a benchmark right it’s like a couple you know respected mathematicians theoretical computer scientists releasing problems that like they believe are like representative of their day-to-day work but haven’t been published anywhere so that we can really have our models take a crack.
對，我對 FrontierMath 這個挑戰很興奮。它是一個 benchmark，由幾位受敬重的數學家、理論電腦科學家釋出他們認為能代表日常研究工作、而且尚未公開發表的題目，讓我們的模型真的去挑戰。 - We were so excited about this challenge, but you know, it was kind of dropped um without any any any advanced warning um with like a week-l long deadline to actually execute.
我們對這個挑戰很興奮，但它就這樣沒有事先預告地丟出來，而且只給大約一週的執行時間。 - Um we had a we had a very exciting model training uh at the time.
當時我們剛好有一個非常令人興奮的模型訓練在跑。 - And so uh um uh um one of the people in charge of training James Lee kind of started prompting the uh that model just um by hand and and and and uh and yeah and actually kind of seeing oh okay it’s actually solving these problems was really a fascinating things to see.
所以當時負責訓練的 James Lee 之一就開始手動對那個模型下 prompt，然後親眼看到它真的在解這些題目，這個經驗非常迷人。 - uh you know one of these powers actually is from a domain that I I I I did my PhD in and yeah seeing the model kind of come up with these ideas which I would you know quite proud to come up with like in a in a week or or two uh seeing it come up with them in like an hour or so that was very uh yeah it’s a very weird feeling right like like yeah I think like in the past the when I felt like that was like when watching our data bot like play just like very interesting data games infinitely
其中一題來自我博士時期的領域，看到模型提出那些我自己花一兩週想出來會很自豪的想法、它大約在一小時左右就想出來，那感覺非常奇妙。上次有類似感覺，是看著我們的 Dota bot 一直玩出非常有趣的 Dota 局。 - right and it feels like just there’s some sort of magic happening because like you know interesting things should not be like » indefinite.
會覺得某種魔幻在發生，因為有趣的東西不應該是無止境的」。 - » Yeah.
» 對。 - And so seeing that happened for math right for something that I believe like you know is actually like quite representative of of of our our or you know a precursor to a lot of the work that we’re doing and a lot of the work that like really matters in the world.
看到這件事發生在數學上、發生在我認為能代表我們很多工作、甚至是世界上真正重要工作的前驅問題上， - Um yeah definitely really increase my feeling of urgency.
這絕對讓我更有急迫感。 - One thing that’s fascinating too is the idea that you’re you’re training these models and it’s like you know you pro you throw these problems in and it’s like nobody knows whether you know how good will they be at solving them and and I think just like it must just be fascinating to see uh something that you know so well and and a space that you spend so much time in and and realizing hey probably the previous generation of models wouldn’t have been able to do that and you wouldn’t even
另一個迷人的地方是，你在訓練這些模型，把題目丟進去，沒人知道它們解題會有多好。看到你非常熟悉、花了很多時間鑽研的領域，發現上一代模型大概做不到， - thought necessarily that this was like the the benchmark to do but it’s like just generally showing the the general purpose capabilities and and improvements of the models.
而且你甚至不見得會覺得這該是要去跑的 benchmark，但它整體上呈現出模型的通用能力跟進步。 - I mean it it is at a stage where like you know we needed to like seek out experts in the in the particular domains to be to be able to tell us whether these particular proofs are correct or not but you know it’s still much easier to like tell whether you’ve you’ve actually made progress than you know than for something like uh even coding right like because sure like competitive programming you can evaluate but most programming is not competitive programming and it’s you know it’s about like
現在這階段我們還是得去找特定領域的專家來判斷這些證明對不對，但判斷有沒有取得進展這件事，仍比寫程式容易得多。競賽程式當然可以評估，但大部分程式設計不是競賽程式，而是關於 - are the abstractions right are handling all the all the cases and yeah » yeah I guess like you know I feel like there was this maybe common critic system a year ago and I don’t know if it’s as strided now that like okay these models are like pattern matchers but like you really want AI for science like we’re not going to get new ideas or like you know entirely novel things out of out of pattern matching feels like we continue to like chip away at that narrative are we getting closer to kind
抽象是否正確、是否處理了所有案例。» 一年前有個常見的批評說法，雖然現在沒那麼普遍了：這些模型只是模式比對器，如果你真的想要 AI for science，靠模式比對是拿不到全新想法、全新東西的。感覺我們一直在削弱這個敘事，我們是不是越來越接近 - of fundamentally disproving that » I believe so yeah I mean I think kind of on schedule we’re starting to see like minor advancements right like not huge things right like a small idea here or there I mean maybe maybe some like bigger papers in collaboration with with scientists, right?
從根本上推翻它？» 我相信是。我們大致按時看到小的進展，像是這裡或那裡的小點子，或許搭配跟科學家合作的更大篇論文。 - But, you know, was Alpha Zero a pattern match, Alpha Go a pattern matcher?
話說回來，AlphaZero 是模式比對嗎？AlphaGo 是模式比對器嗎？ - You know, our our datab match like they did kind of come up with new strategies for the respective games.
我們的 Dota bot 也確實為各自的遊戲發展出新策略。 - » Um, » it’s funny that there’s counter examples to it all the way back to, you know, 2016, 2017.
» 好玩的是，反例早在 2016、2017 年就已經存在。 - » Right.
» 對。 - Right.
對。 - And and, you know, and you can say like, well, I guess you can always fall to flaws in that which I think is interesting like AlphaGo can be beaten with some strategy.
你可以說它有它的瑕疵，像 AlphaGo 可以被某些策略擊敗。 - our data bots could have been been bitten with some with some strategy.
我們的 Dota bot 也可以被某些策略反制。 - I think I think there will be a lot of definitiones for a while of of like these models, right?
我覺得這些模型還會有一段時間被大家從各種角度定義。 - But but I think also like they they are able to discover new things because they have a lot of these capabilities and like the way you know yeah I mean it’s you know taken a couple years to like get go from like this like very tiny game environments to like this much more um general scientific research.
但我也覺得它們確實能發現新東西，因為它們具備了這些能力。從那些非常小的遊戲環境走到這種通用得多的科學研究，花了好幾年。 - it required kind of going through um you know like a decent approximation of like all human knowledge in the meantime and you know learning all the human languages and so forth but but um but I think the basic principle is is is very similar.
過程中需要經過對「所有人類知識的相當程度近似」的吸收、學會所有人類語言等等，但我覺得基本原理非常相似。 - » Yeah.
» 對。
AI 與實體世界的互動
- You know, it’s funny.
你知道，很好玩。 - I think like when you guys had these first proof results, um I remember like the organizers said, you know, they were commenting on these AI solutions and they were like this feels like, you know, 19th century mathematics of like brute force, you know, computationheavy approaches rather than these like elegant modern techniques.
當你們拿到 FrontierMath 的結果時，主辦方在點評這些 AI 解法時說，這感覺像是十九世紀的數學，是那種蠻力、仰賴大量計算的做法，不是優雅的現代手法。 - Um which I’m not sure is a feature or bug of of you know, obviously the the way these models work, but like you know, hearing that I mean does that like does that concern you, excite you?
我不確定這到底是這些模型運作方式的特色還是 bug，但你聽到這個會擔心還是興奮？ - » It doesn’t concern me.
» 我不擔心。 - I mean I think it’s expected that like I I’m sure I I thought for at least one of the problems like actually actually our produced pretty pretty nice pro that was quite a bit shorter than like the intended one you know but I think in general you would expect like yeah this models kind of you know they can produce so much more reasoning in a short time than like a person can right just like in terms of just raw number of like tokens or thoughts I don’t expect that to be like kind of a
我覺得這是可以預期的。其實至少有一題我們產出的證明相當漂亮，而且比原本預期的版本短不少。但一般來說，你會預期這些模型在短時間內能產生的推理量遠大於人類，就單純從 token 或想法數量來看。我不預期這會是長期 - long-term feature » it feels like there’s so much momentum behind AI for science right now and you mentioned obviously like you know at some point you do have to connect these these models to the physical world and you guys released some cool stuff with GKO and like some of these other things you’ve been experimenting with.
的特徵。» 感覺 AI for science 目前氣勢很強，你也提到某個時點你得把這些模型接到實體世界，你們跟 GKO 的合作以及其他一些你們在嘗試的東西就是這樣。 - I’m sure you’ve thought a lot about like AI for a bunch of different areas of science.
你一定想過很多 AI 應用在不同科學領域的事。 - You know, as you’ve kind of dug into some of this stuff, have you dealt with any intuition for as you think about like 3 years from now, the spaces where of science where you’re like, “Oh, that there’s going to be crazy progress there versus the ones that might prove like a little more resistant to immediate change.”
當你深入這些東西之後，對於三年後的樣貌有什麼直覺？哪些科學領域你會覺得：「哦，那邊會有誇張的進展」，又有哪些可能會對立即的改變比較抗拒？ - You know, a tempting answer would be that like oh, you know, it’s really about like um you know, do you uh you know, what are the things that kind of require some some you know, manual work like where the models are not like not not quite plugged in the ecosystem or you know like the that the the different laboratories will also kind of evolve pretty quickly to adopt to like these new technologies » within those STEM fields.
一個很誘人的答案會是：這要看哪些事情還需要相當多的手工作業、模型還沒真正接進生態圈的地方。但不同實驗室也會演進得相當快來採用這些新技術。» 在這些 STEM 領域裡。 - Obviously, you know, I feel like there’s a question of is it like an LLM with access to the physical world or you’ve obviously had companies that are have been started specifically around these domains, right?
顯然有一個問題是：這是一個可以存取實體世界的 LLM，還是你會看到為這些領域而生的公司？ - Like an isomorphic in biology or periodic in in material sciences or physical intelligence and robotics.
像生物領域的 Isomorphic、材料科學的 Periodic、或是 Physical Intelligence 跟機器人。 - What’s your kind of gut instinct on the extent to which it makes sense to pursue some of these things like independent with different model architectures versus like all within the context of one place?
你的直覺是，這些東西獨立用不同的模型架構去做比較合理，還是全部統整在同一個地方比較合理？
用於科學領域的模型架構
- Yeah, I think it’s kind of similar to you know my answer about like the um UI for you know for codex which like I I would build around the capabilities of a technology and not around it limitations so much.
這跟我回答 Codex 的 UI 問題有點像：我會圍繞一項技術的能力去建構，而不是太繞著它的限制去設計。 - Um so you know you definitely like if you have something that like can suddenly design like a huge amount of like interesting like chemical or biological experiments like yeah I mean it makes sense to uh you know build labs that enable that.
所以如果你手上的東西突然之間能設計出大量有趣的化學或生物實驗，當然你就會想去蓋能支援這件事的實驗室。 - You know, I think if we if we did get to a place where like the model is like very capable of designing high quality experience.
我覺得如果我們真的走到模型很能設計高品質實驗的那個境界。 - It also makes sense to like have it work with humans in a loop, right?
那讓它跟人類以迴圈方式合作也是合理的。 - Like we shouldn’t think of it as like oh it’s either you kind of automated fully and you have this like fun thing using some tools on the side.
我們不應該把它當成二分法：要嘛全自動化，要嘛就是個在旁邊用工具的有趣玩具。 - Like we will get to a world where like it’s just very natural to be collaborating with um you know AI scientists that are that are working hard on a problem.
我們會走到一個世界，跟在努力攻克問題的 AI 科學家合作會是非常自然的事。 - » Yeah, it’s so interesting.
» 對，這很有意思。 - It’s almost like a different vision.
這幾乎是一種不同的願景。 - It’s like one world where this works is like hey you just train a model you know to basically run these endto-end tasks and like be the automated like you know uh biologist or you know chemist or whatever it is and there’s another one which is like well you’re building really tools to you know both propose run kind of work in tandem with a bunch of human researchers » I mean you know I wouldn’t necessarily categorize it as I mean you know of course there are tools in some sense but I think
一個版本是你訓練一個模型去跑端到端的任務，變成自動化的生物學家、化學家等等；另一個版本是你其實是在打造工具，既能提出想法、又能執行，跟一群人類研究員協作。» 我不會把它歸到工具這邊；某種意義上它們當然是工具，但我覺得 - like you know we will get to a point where they’re driving a lot of the like design and and ideation for the whole process.
我們會走到一個地步，它們會主導整個流程裡的設計與發想。 - Yeah, with with like an LLM architecture, but just like you know being able to figure out the right way, the right kinds of experiments to run and and then actually design it.
以 LLM 架構為基礎，能想清楚要跑什麼樣的實驗、並真的設計出來。 - And yeah, when it comes to like different architectures and you know,
至於不同架構的部分， - I mean, you know, for sure like you know like natural language reasoning like the kind of the kind of things u that that we’re prioritizing that gives you a lot of generality like there there are things that are that you know you kind of want to train it you want to train a different model to to model right you know I think even like yeah if if you want to create a very good you know G model I I don’t think like large language models are like the most efficient way to go about this although
我們優先處理的自然語言推理這類東西給你很多通用性，但還是有些東西你會想訓練另一個模型來建模。比方說你要做一個很好的 generative 模型，我不認為大型語言模型是最有效率的做法，即使 - they might result in the best model eventually but uh you know I think it’s similar for like uh you know protein folding or or other task of this kind.
它最終可能是最好的模型。蛋白質折疊或類似任務也是同樣的情況。 - » Yeah.
» 對。 - So you think it makes sense to have like some independent efforts around that but obviously the like you know that will end up being paired with like a core really good researcher large language model that is you know helping drive a bunch of this stuff.
所以你覺得在這些領域做一些獨立的努力是合理的，但最後這些會跟一個核心、非常強的研究員級大型語言模型搭配起來，由它來推動很多事情。 - » Yeah.
» 對。
思維鏈監控及其意義
- I want to also make sure just to talk about AI safety because I think that’s an area that you’ve done a lot of really pioneering work on.
我也想確保我們聊到 AI 安全，因為這是你做了很多開創性工作的領域。 - Um and you know I’m not sure all our listeners will be familiar with uh you actually did some really interesting work across the labs right uh and and were focused on you know chain of thought monitoring and so maybe to start just talk tell us a little bit about that work and and you know uh you know what you found.
我不確定所有聽眾都知道，你其實跟幾個實驗室一起做過很有趣的跨實驗室工作，聚焦在 chain of thought monitoring（思維鏈監控）上，能不能先稍微介紹這項工作跟你們的發現？ - » Yeah so this is um a realization that actually we had um around the time we actually saw like the first um reasoning models of kind of the current crop.
» 這是我們在看到目前這一代第一批推理模型時就有的體會。 - We realized that like okay like well this works right and we were pretty uh you know we were thinking a lot about what this means we kind of were like okay like probably the word really changes over the next I don’t know year or two or three you know we were thinking what this means for for safety and for for our ability to kind of understand what these models are doing and we realize that because of the way we train these models that because we don’t supervise the reasoning process directly
我們意識到：好，這個方法真的行得通。我們當時花很多時間思考這代表什麼意義，也想說世界大概會在接下來一兩三年真的改變。我們在思考這對安全以及我們理解這些模型在做什麼的能力代表什麼意思，然後意識到：因為我們訓練模型時並不直接監督它的推理過程， - right it’s not like you know chpt is trained to kind of um you know be be polite and nice and like Um, and » it always tells me I have great ideas.
不像 ChatGPT 是被訓練成有禮貌、友善。» 它總是跟我說我的點子很棒。 - » Yeah.
» 對。 - Well, you know, that’s a separate issue, right?
這是另外一個問題了。 - Like, but but you know, but but like even assuming it’s like aligned exactly in the way we would want it to, which is definitely not, you know, uh, sick ofic like it’s still kind of not going to be uh, you know, there are just still still some things it’s not going to reveal about its motivations and time because, you know, maybe it would be unsafe or maybe it would be unkind.
就算假設它完全按照我們想要的方式對齊好了（絕對不是討好型），它還是不會把自己所有的動機都揭露出來，因為有些可能不安全、有些可能不客氣。 - um um or you know or maybe because it’s not maybe it’s actually not aligned the way we think but it wants to hide that right and uh and the way we train the reasoning models like the the the train of thought doesn’t have any of that it’s not optimized to uh to be in any particular way because it’s just not not directly great it’s only great in how it relates to like producing a high quality output um and realize this is actually a very powerful paradigm time for being able to interpret what
或是它其實沒有我們以為的那麼對齊，卻想把這件事藏起來。而我們訓練推理模型的方式，使得思維鏈不會被優化成任何特定樣子，因為它不會被直接評分，只有在它與產出品質的關聯上才會被評分。這是一種非常強大的範式，讓我們能解讀 - the model is doing, right?
模型正在做什麼。 - It’s actually not a very different idea from uh um mechanistic interpretability, right?
這其實跟 mechanistic interpretability（機制可解釋性）不是差很多的想法。 - Because in mechanistic like the idea is again like you kind of have this model, you have these activations of the model um that you know are not directly supervised to predict any label.
因為在機制可解釋性裡，你有這個模型、有模型的各種 activation（激活值），它們並沒有被直接用來預測任何標籤而受到監督。 - they’re they’re kind of like indirectly supervised but you know the model kind of has never been trained with like any sort of like uh you know inspection of the of these activations and so these activations might reveal something about this in inner workings but the big advantage of the chains of thought is that you know by default they are in English right and so it’s so much easier to understand what is going on especially you know as the concepts get more advanced u and the other
它們是間接被監督的，模型沒有被用「檢查這些 activation」的方式訓練過，所以這些 activation 可能透露出模型內部的運作。但思維鏈的一大優勢是預設使用英文，所以要理解它在做什麼簡單太多了，尤其是當概念越來越進階時。另一個 - interesting thing is um you know we were just talking about how probably you know how how we believe in in the future where we go uh well these models work for a very long time they work autonomously right and so there there is much more of this reasoning uh and so you know if this is a big axis of how the capability of these models increases um that the sort of our ability to supervise them will will scale uh uh comately.
有趣的地方是，我們剛剛聊到我們相信未來這些模型會運作很長時間、自主運作，所以這類推理會變得更多。如果這是能力增長的一大軸線，我們監督它們的能力也會等比例擴展。 - Yeah, this really comes down to this principle though that like you know you’re not supposed to supervise the train of thought and so this is actually something uh when we originally you know we’re releasing the preview model like we made this decision to like hide the chains of thought and » yeah I remember » and um you know for me that was the primary motivation that was the reason like I didn’t really even want to consider releasing it in different ways you know there definitely was a
這其實歸結到一個原則：你不應該監督思維鏈。所以我們當初推出 preview model 時就決定把思維鏈藏起來，» 對我記得，» 對我而言這是最主要的動機，也是我完全不想考慮用其他方式釋出的原因。當時內部 - bit of internal discussion about this but like the reason I felt very strongly like we should we should just hide it is because of this.
確實有一些討論，但我非常強烈覺得應該把它藏起來，理由就是上面這件事。 - Uh then there was this other concern that like I didn’t initially think about but I think was also like very valid of like well you know like this model is going to be distilled to some extent blah blah uh and you know and that’s definitely also been like a big factor here.
另外還有一個我一開始沒想到但我覺得也很成立的考量：這個模型某種程度上會被 distill（知識蒸餾），這也是一個重要因素。 - Uh but but yeah but I actually think that like this uh you know allowing the models some sort of private space uh oh and by the way like why do I think it’s important that we don’t like you know show this change of thought in product you know um if if if I’m saying like the important thing is not to supervise them during training well I think if we did show in if we like established a paradigm where like oh you just show this chains of thought in product uh eventually you kind of have to
我其實覺得讓模型有某種私人空間很重要。為什麼我覺得在產品裡不顯示思維鏈很重要？如果我說訓練過程中不監督它很重要，那如果我們在產品裡把思維鏈秀出來、建立了這種範式，最終我們還是會被迫去 - train them right like you’ll have to train them for the same reasons you have to train like whatever models you ship.
訓練它，就跟你要訓練任何要出貨的模型一樣的理由。 - Um and I just think that » we might not all want to know what the chain of thought our model has that gets to a response for » right I mean you know I think I think it’ll be useful to some extent and we are trying to capture most of that value you know either with like chain of summaries uh which I think are kind of like a little bit of a stop gap.
我只是覺得，» 我們可能不是全部人都想知道模型抵達某個回應時的思維鏈。» 對，我覺得它一定程度上有用，所以我們也嘗試用 chain of summaries（思維鏈摘要）來抓住那份價值，我覺得那還是一個權宜之計。 - I think the longer term solution here is having the model actually talk to you in real time which you know the later the latest version of Codex kind of do latest version of of the reasoning GP models kind of do but I think I think that will get much better um yeah but but yeah I think there’s something very exciting here about just like not u not having the training signal fight against us right and not not Yes because yeah I think if you If you want to be able to understand what the model
我覺得長期的解法是讓模型能跟你即時對話，最新版的 Codex、最新版的推理 GPT 模型已經有點這個意思，我覺得還會變好得多。不讓訓練訊號跟我們作對這件事非常令人興奮。因為如果你想長期理解模型 - does in the long term, but you know you’re scaling a method that is like kind of going directly against that, it’s you’re probably not going to have a good time, right?
在做什麼，卻又擴展一個跟這目標直接作對的方法，那你大概不會有好結果。 - That’s the other side of the better lesson.
這是 bitter lesson 的另一面。 - Uh and so this decoupling I think is a very it’s an idea that gives me a lot of hope for our ability to at least understand um you know how these models motivations and generalization evolve as they get better as they as they work for longer.
所以這種解耦對我來說是一個很有希望的想法，讓我們至少能理解，隨著模型變強、運作時間變長，它們的動機與泛化是如何演變的。 - Um yeah, I don’t think it’s a complete solution to AI as alignment by a long shot.
我不認為這是 AI 對齊的完整解答，差得還遠。 - I think it’s just another tool in our in our toolbox.
它只是我們工具箱裡多出來的一個工具。 - Uh but I am hopeful that building our toolbox with technical tools like this, we can actually continue chipping away at the fundamental problems here.
但我希望透過像這樣一件件技術工具把工具箱建起來，我們能持續削減這些根本性的問題。
理解長期對齊
- » Yeah, it seems like almost like over the, you know, medium term, it’s like something that’s going to be incredibly helpful.
» 對，中期來說這會是非常有用的東西。 - Probably not the catchall solution for for long-term alignment.
但可能不是長期對齊的萬靈丹。 - Yeah, I mean I think it’s a tool that can help us understand like I think it’s actually very useful to like build understanding of long-term alignment, right?
我覺得它是個能幫我們理解長期對齊的工具，對於建立對長期對齊的理解其實非常有用。 - For example, there has been this very exciting quark um from um um um from a planning collaboration with other labs uh on uh model scheming where they investigate uh you know depending on kind of what environment you pro you put the model in, how you train it like is it is it prone to like start kind of like having hidden objectives that it pursues and you know what enables that that whole line of work is chain of fat monitoring right is this notion of like oh you can actually inspect what
例如，最近跟其他實驗室在模型 scheming（陰謀行為）上的合作產出了很令人興奮的點：他們研究依據你把模型放進什麼環境、怎麼訓練，它會不會開始發展出它會去追求的隱藏目標，以及什麼條件讓這件事發生。整條研究路線的基礎就是思維鏈監控，也就是你可以真的檢視 - the most motivations are uh so you know and I think from that like that might take us in a completely different in terms of mitigations right like maybe the right way is like changing the pre-training data of the model or maybe it’s something like uh you know the inoculation prompting from a topic like I think I think those are very interesting ideas but I think like having this ability to like understand is very helpful to to evaluate these » yeah it’s almost like foundational for any
模型的動機。從那裡出發，緩解方法可能會走向完全不同的方向：也許正確的做法是改變模型的預訓練資料，或是像 Anthropic 提出的 inoculation prompting（預防式提示）。這些想法都非常有趣，而具備「能理解」的這種能力，對評估這些方向很有幫助。» 對，它幾乎像是任何 - further uh area of research what are like the other research areas within alignment that you’re paying attention to or that you think are promising you know areas to focus on Um yeah, I think I think a lot of the a lot of the like longer term challenge with alignment is about generalization, right?
後續對齊研究的基石。對齊領域裡還有哪些研究方向你特別關注、覺得有前景？對齊的長期挑戰很大部分關乎泛化。 - Like we can train our models to do well and and and and or you know at least mostly to some extent like we we can mostly kind of control their behavior in the in the things that that you know are in distribution that that we train for.
對於我們訓練過的分布內行為，我們大致能控制模型的行為。 - Um, but you know the things that are worrisome is like well what happens when animal is asked to do something very very different or it finds itself in a very different situation or it’s like much smarter than it ever was before and and and you know it has all these capabilities.
但令人擔心的是：當模型被要求做非常不同的事、或發現自己處在非常不同的情境、或變得比以往聰明很多、擁有全部這些能力時，會發生什麼事？ - It’s like we haven’t really kind of thought about how to train for and so yeah so so I think I think you know the study of like this kind of longer term value alignment is really a study of generalization like what are the values that the model falls back on.
這些情況我們還沒真的想清楚怎麼訓練。所以長期價值對齊的研究其實就是泛化的研究：當模型退回預設時，它依靠的是什麼價值觀。 - Um like one line of research I’m very excited about here and something that we’re uh investing in quite a bit is uh understanding like how that um how the generalization falls back onto the pre-training data.
我很興奮、也投入不少資源的一條研究路線，是理解這種泛化是怎麼退回到預訓練資料上的。 - Um um yeah and yeah I I I think there’s quite a lot there.
我覺得那裡面有很多東西可以挖。 - I guess over like you know the last six months have your concerns around alignment increased decreased like how do you you know where are we kind of trending overall uh you know with this work » I I I will speak to like the the the longer term challenges of like fignment right or like what happens when you have very smart models the the way my thinking about the problem has evolved over the past few years is definitely kind of gone from you know oh is this like very nebulous problem that
過去六個月，你對對齊的擔憂是變多還是變少？整體趨勢如何？» 我就談長期的對齊挑戰，也就是「當你擁有非常聰明的模型時會發生什麼」的那個問題。我過去幾年對這個問題的想法，確實從「這是個非常模糊、 - like is just like very hard to even grapple with or define uh to like oh you know I think we can actually make prog progress at it by very concrete technical solutions and technical insights.
難以掌握或定義的問題」，轉變成「我覺得我們真的能用非常具體的技術解法跟技術洞見在這上面取得進展」。 - And this is why we’ve really been uh viewing alignment as like just a core part of of research and really uh you know making sure that like we are you know designing our reasoning models uh thinking about this and we are you know and we are kind of like conducting our alignment research with like these reasoning models in mind and so forth.
這就是為什麼我們真的把對齊當成研究的核心部分，確保我們在設計推理模型時就把它考慮進去，並以這些推理模型為前提進行對齊研究。 - Um so I think my general kind of uh belief that there’s like a research path here that actually gets us to an extremely happy world uh has increased quite a lot.
所以我對「存在一條能把我們帶到極佳結局的研究路徑」這件事的相信度，已經提升不少。 - Um, at the same time, right, I think uh my timelines to very capable models have definitely decreased a lot, right?
同時我對非常強的模型何時到來的時程預估，也顯著縮短。 - I think we’re we’re not that far, right?
我覺得我們離那個時點不遠了。 - Again, I don’t think these are models that are smarter than all the ways, but I think these are models that are just very transformative.
我不是說這些模型在所有面向都比人類聰明，而是說它們會帶來非常顛覆性的改變。 - And so, I’m quite optimistic like we can keep a good grip on like how we’re doing on the alignment problem, how to roughly evaluate the risks of of of of our models or or the problems with them.
我相當樂觀地認為，我們可以掌握對齊問題的進展，粗略評估模型的風險或問題。 - you know, but I do think we have to be, you know, as an industry as really prepared to like take trade-offs and, you know, and possibly, you know, slow down development uh um depending on what we see.
但作為產業，我們真的要準備好做取捨，也可能要視情況放慢開發步調。 - It » it’s already interesting to see a lot of this work happening across the major labs.
已經能看到很多這類工作跨主要實驗室同時在發生，很有意思。
業界在對齊議題上的合作
- You know, the fact that you did this in collaboration with I think Anthropic and Deep Mind and you know, it seems like uh has that just come up organically or imagine like is there a lot of like alignment talk between you know, the the major players, you know, uh given I guess the three of you are really at the forefront of all this?
這次你們跟 Anthropic 和 DeepMind 一起做這個，是很自然地形成的嗎？還是在主要玩家之間，對齊的對話本來就很多？畢竟你們三方幾乎站在這件事的最前線。 - There’s definitely some I mean there’s definitely like shared interest in this topics.
絕對有一些，大家對這些題目有共同的興趣。 - Yeah.
對。 - » I want to shift a little bit to going inside OpenAI.
» 我想把話題稍微轉到 OpenAI 內部。
OpenAI 的研究組織
- I feel like no no company probably or the world has been more interested in over the last uh 2 three years and you know I think particularly what it’s like to run a research organization.
我覺得過去兩三年大概沒有哪家公司比 OpenAI 更受世人關注，尤其是「經營研究組織」這件事。 - You know we talked a little bit about this uh previously but you talked before about how it’s you know important part of your job is giving researchers you know uh to to kind of have comfort and space to you know almost be cave dwellers right and think about what the models will look like in a few years.
我們之前聊過一點。你以前提到，你工作的重要一環是給研究員舒適感跟空間，讓他們幾乎像穴居人一樣去想幾年後的模型會長什麼樣子。 - Um, you know, we were kind of alluding to it earlier.
我們前面稍微有提到。 - We’re also in a time where it feels like there’s just massive competitive race and you know, uh, it’s it’s it’s certainly, you know, everyone’s going really gung-ho on these coding models.
我們也正處於競爭激烈的時期，大家對寫程式模型都非常拼。 - I’m wondering like how do you actually operationalize this balance today and and you know, anything you’ve kind of changed in your thinking, you know, overseeing this organization around the right way to do this?
你如何實際操作這種平衡？在帶領組織的思考上，有什麼你改變了的想法？ - you know I focus on on just high quality experiments recognizing you know are we actually making progress being honest with ourselves and you know and promoting honesty about about the results um I don’t think that has changed right and and uh you know even though our work will evolve a lot I believe we still have quite a lot of work left to do and so I don’t think it’s like oh you know we need to wrap up all our projects uh um you know very very quickly so yeah I don’t think those
我聚焦在高品質的實驗、誠實面對「我們到底有沒有進展」、並鼓勵大家對結果誠實。我覺得這點沒有改變。即使我們的工作會大幅演進，我相信還有很多事要做，所以我不認為我們需要非常快地把所有專案都收尾。 - fundamentals change I think what what does change is uh you know a level of urgency to really kind of bring some of these things that we think are most promising uh to fruition » and then obviously you know I feel like there’s been um you know some very public internal moments of open AI over over the years you’ve been here for a long time as you kind of reflect back like what were some of the difficult decisions that you guys made that maybe were like 5149 that really you know defined the
這些基本面不會變。改變的是，要把我們覺得最有潛力的那些事情真的做出來的急迫感提高了。» 多年來 OpenAI 有過一些非常公開的內部時刻，你在這裡很久了，回頭看，有哪些 51/49 的艱難決定，真正定義了 - company or any any any as you think back of the movie of the last you know seven eight years of your life um you know the key moments that kind of stick out to you.
這家公司？回顧你過去七八年的這部電影，有哪些關鍵時刻特別鮮明？ - Well, yeah.
嗯，對。 - I mean, there’s certainly a number of, you know, dramatic moments, uh, like this.
確實有不少戲劇性的時刻。 - Um, you know, I think the ways the company underwent the most change is not really this like snap changes, snap decisions, but more like just like shifts and and how it operates, right?
但我覺得公司經歷最大的改變，並不是那種突然的變動或瞬間的決定，而比較像是營運方式的轉變。 - I would say like opening has gone for a couple phases.
OpenAI 經歷了幾個階段。 - you know when I joined at the start of 2017 2017 very much kind of uh felt like very academic lab pursuing like a lot of different ideas not so you know scaling pill in practice uh and I think that was like the first like big change with the data product with GPT we’ve kind of moved to okay like we actually are going to have to buy big computers we’re actually going to have to um scale things we going to have to develop the science of scaling we’ll have to develop the infrastructure for it
2017 年我加入時，這裡很像學術實驗室，追求很多不同想法，scaling pill（擴展論）在實務上還沒落地。第一次大改變是隨著 GPT 這個資料產品而來：好，我們真的得買大電腦，真的得去擴展、發展擴展的科學、發展相應的基礎設施。 - um and so that kind of started the second phase of of okay now we’re scaling right like we’re we’re we’re still going to pursue like a lot of these basic research ideas but we are going to evaluate them like for the act are this are they scalable um um then yeah then there was this interesting period I talked about earlier right where you kind of have » chat GPT is this big thing yeah I mean I thought it would look a little bit differently right like I think I I was actually surprised that
這開啟第二階段：我們在擴展，仍然追求許多基礎研究想法，但會以「是否可擴展」去評估它們。然後就是我前面提到的那個有趣時期，» ChatGPT 變成一個巨大的東西。我以為這一切會長得不一樣，其實是 - like text models I was pleasantly surprised like text models are actually kind of the first thing.
文字模型成為第一個爆紅的東西，令我驚喜。 - I thought we would be in a world where like it’s more the kind of like you know video style uh uses of generative AI are kind of like the first » uh the first big thing to take off and like and we’ll have to like trade off like pursuing the kind of longer longer term text based research.
我以為我們會在一個影片風格的生成式 AI 應用先爆紅 » 的世界裡，然後必須在追求長期文字研究上做取捨。 - Uh so yeah so so so but yeah but I think definitely like we anticipated that like this sort of tension would arise right where like you have a thing that is kind of like popular now but it’s like you know you believe it’s going to evolve quite a lot before you get to where you’re going and so I think that’s kind of the phase we’ve been in for a while um and yeah I think now we’re we’re like uh um well yeah I mean we believe we are kind of like starting to be in this phase where yeah we’re
我們的確預期這種張力會出現：你手上有一個現在很紅的東西，但你相信它還會大幅演進才會到達你的目標。我們在這個階段已經一段時間了，現在我們相信自己正在進入一個新階段，我們真的 - actually deploying AGI or you know deploying models that are actually very economic transformative.
在部署 AGI，或者說部署具有經濟顛覆性的模型。 - » No, it’s uh it certainly seems that way.
» 對，看起來確實是這樣。 - Well, I guess we always like to end interviews with a standard set of quickfire questions which are basically me just stuffing all my overly broad questions I couldn’t fit anywhere else.
我們通常會用一組固定的快問快答收尾，基本上就是我把塞不進其他地方、但想問得又太廣的問題都集中在這裡。
未來 AI 的影響與社會意涵
- Uh so if you you’ll shamelessly indulge me uh you know I guess to kick it off would love what’s one thing you’ve changed your mind on in the AI world in the last year?
那就厚著臉皮問你。過去一年在 AI 這個領域，你改變了想法的一件事是什麼？ - Yeah, I mean I I think I think it’s really, you know, starting to reconcile this tension between, you know, the AI that you build ultimately is something that affects the world, but, you know, until you until you kind of get pretty close, it’s like a pretty theoretical thing that you’re just kind of, you know, u training and developing algorithms for.
我真的是開始在調和這個張力：你打造的 AI 最終會影響世界，但在你還沒走到夠接近那一點之前，它感覺很理論，你只是在訓練跟開發演算法。 - And so, you know, recognizing that okay, now we actually need um we really need to um you know make a lot of pro progress and focus on like how actually we’re deploying this technology and um in a while.
意識到：好，現在我們真的需要在這件事上取得很大進展、聚焦在我們實際上如何部署這項技術。 - This is definitely something I’ve been I’ve been thinking about a lot lately.
這絕對是我最近一直在想的事。 - » Yeah, it’s so interesting.
» 對，這很有意思。 - basically like you know uh outside of chat it was almost like more in the in the abstract or research hill climbing you know with some usage in the real world and then in this last year we’ve obviously seen you primarily via coding agents just you know it it trickle in you know in in a pretty massive way.
基本上在 ChatGPT 以外，你們之前比較像是在抽象面或研究端爬坡，在真實世界有一些使用，而去年我們主要透過 coding agent，看到它以相當大規模地滲進來。 - » Yeah I I I I think I I believe is kind of going in the same direction as like the coding models where like it’s actually going to be something um you know very useful it’s going to be something that’s like a meaningful part of of of people’s lives.
» 對，我覺得它會往寫程式模型那個方向走，變成真的非常有用、並成為人們生活中有意義的一部分的東西。 - when you say going in the same way you mean just like executing longer term tasks or more like you know the » I feel that’s part of it right but also just um you know coming to become like a dependable trustworthy assistant or compion » yeah it’s amazing to watch the way younger people use jet I’d argue it’s it’s already pretty much there for uh the way a lot of folks in in high school and college and you know uh seem increasingly you know comfortable using it um you know I wouldn’t be a
你說同樣的方向，是指執行更長期的任務，還是 » 我覺得那是一部分，但也包括變成可靠、值得信賴的助理或伴侶。» 看年輕人用 ChatGPT 的方式很驚人，我會說對很多高中生、大學生來說其實已經差不多到了那個地步，大家用它越來越自在。 - shameless podcaster if I didn’t ask a top researcher you know timelines for a few things I think particularly interesting is the stuff outside of the core LM world and so think there’s a lot of buzz around robotics these days.
我身為一個不要臉的 podcaster，怎麼能不跟頂尖研究員問幾個時程。我覺得特別有意思的是核心 LM 世界以外的東西，最近機器人就有很多聲量。 - Do you have any like in I mean obviously it’s hard to pinpoint like a moment robotics quote works but I think you know whether it’s finding scaling laws or finding some sort of like chatbtesque moment for robotics.
你有直覺嗎？機器人「開始真的有用」的那一刻很難指出，但不管是找到 scaling law，或是機器人界的 ChatGPT 時刻。 - » Yeah.
» 對。 - I mean I definitely think there are like very promising algorithmic ideas there that I I believe are going to work that are you know not too dissimilar from the space of ideas.
我絕對覺得那邊有很有潛力的演算法想法，我相信會成功，而且跟這一塊的想法空間差異不大。 - So I’m I’m quite optimistic about about timelines there.
所以我對那邊的時程相當樂觀。 - Uh although I do think they’re longer than like the kind of the virtual um AI.
雖然我覺得會比純虛擬 AI 的時程來得長。 - » Obviously I’m sure you think a lot about you know cuz you’re always thinking about the next frontier for what these models can do.
» 你肯定常想這件事，因為你一直在想這些模型下一個能做什麼的前沿。 - Um you know just the impact on on society as a whole as you think about this kind of pace of continued model improvement.
當你想到模型持續進步的步調時，對整體社會的影響也會一起想。 - You know what’s maybe one thing that you think we’re underthinking right now as a society in terms of the impact of these models?
就這些模型的影響而言，有哪一件事你覺得社會上目前想得太少？ - Yeah, I I I think getting to a point where so much intellectual work um can be automated I think comes with pretty big problems that I don’t think have obvious solutions.
走到「大量心智工作都能被自動化」的那個點，會帶來一些大問題，而且我覺得沒有顯而易見的解法。 - One natural is a question of jobs and you know concentration of wealth and I suspect this requires like real policy maker involvement.
一個自然的問題是工作機會跟財富集中，我懷疑這需要政策制定者真的介入。 - Yeah, I’ve heard some kind of optimistic takes on how is this resolved, but I think I think at a at fundamental level it does seem like you know some things that like used to be very valuable used to kind of cost a lot and used to provide something like now can be done pretty cheaply and you know in the long term it should be a good thing but I think it does lead like I think it can happen quite quickly.
我聽過一些樂觀的看法，但基本層面上，以前很有價值、很貴、能提供某種產出的東西，現在可以很便宜地被完成。長期來看是好事，但我覺得這件事可能會發生得相當快。 - and there is a related question of you know you really can like if you actually have you know an automated research laboratory an automated company that can do so many things like it can be controlled by a very small number of people right it can be it can do a lot right and this gets this gets you know even more crazy when you have robots but but you don’t need to have robots and you know I think figuring out like what does governance of such things looks like look like right like what are
另一個相關問題：如果你真的擁有一個能做很多事的自動化研究實驗室、自動化公司，它可能由非常少數的人控制，卻能做非常多事。有了機器人之後這件事會更誇張，但其實不需要機器人就已經夠瘋狂。要搞清楚這類東西的治理長什麼樣子， - these like organizations that like so powerful and yet maybe made of like only a couple of people like what how to think about these things I think is uh it’s a new question we have to grapple with our society when speaking of other new questions one thing that’s very top of mind for me I I recently had a kid and I’ve been thinking a lot about like you know what is his life going to look like in in 10 years um you’re really close to this stuff how has your work on on on AI changed the way you
這種組織可能威力巨大、卻由少數幾個人構成，該怎麼看待？這是我們社會必須面對的新問題。說到新問題，我最近生了孩子，我一直在想十年後他的人生會是什麼樣。你離這些東西這麼近，你對 AI 的工作怎麼改變了你 - think about like the way in in which you know this next generation should be raised » a task for all of us right is to build the AI right build a world in a way where uh you know at the end of the day humans have the agency right humans set the the direction right and you know maybe a lot of the the technical challenges that we cherish right now will become more of a you know past time that’s something that we really kind of like needs to do in order to make progress and and the challenges
對下一代養育方式的思考？» 我們所有人的任務，就是把 AI 建好、把世界建成：最終人類擁有能動性，由人類來設定方向。也許我們現在珍視的很多技術挑戰，會變成為了進展而必須去做的「過去式」，而真正的挑戰會 - will be more and like figuring out like what are the things that are important what are the things we should go do you know I think that that will still be you know I think I think you know in that world like people can end up with you know more things to do and definitely more more exciting things to to do and you know I think I think you still want like to have an understanding of you know of like uh you know some understanding of like you know technology like all all the kind of like uh
更多在於「想清楚哪些事情重要、我們應該去做什麼」。在那個世界，大家可能會有更多事情可以做、而且是更令人興奮的事情可以做。你還是會希望對技術有一定程度的理解， - basic you know education however you want to acquire it for the sake of being able to think about these problems.
以任何你想要的方式取得基礎教育，目的是為了能思考這些問題。 - » Well this has been fascinating man I really appreciate you sitting down and and talking about so many different things.
» 這真的很迷人，感謝你坐下來聊這麼多不同的事。 - Um, I want to make sure to leave the last word to you.
我想把最後一個麥克風留給你。 - Like anything you uh want to point our listeners to, whether it’s research you’re doing or products you’re excited about or really anything you’d like to uh to plug uh the floor is yours.
你有什麼想推薦給聽眾的嗎？你的研究、你興奮的產品、任何你想宣傳的，舞台交給你。 - Um, you know, anything I’m sure there’s tons of threads people want to uh pull out of this conversation.
我相信這場對談裡有很多線索大家會想再往下拉。 - » I think the set of problems we just discuss, right, and also the questions around alignment, monitorability, I I I think I think those are growing to be very urgent challenges.
» 我們剛剛討論的這些問題，以及對齊、可監控性相關的問題，我覺得會逐漸成為非常急迫的挑戰。 - And I don’t think there are challenges only for AI researchers, right?
我不認為這些挑戰只屬於 AI 研究員。 - I think there are challenges challenges for policy makers, but also also just things we have to think through as a society and uh yeah, I I’m you know, I’m happy to see some discourse starting to arise and I I think we need more of it.
它也是政策制定者的挑戰，也是我們必須以整個社會的角度去想清楚的事。我很高興看到相關討論開始浮現，我覺得我們需要更多這類對話。 - » Yeah.
» 對。
結語
- Well, I thought I could talk to you for hours more, but I’d be doing the world a great disservice by keeping you from your actual work of continuing to improve these models.
好吧，我覺得還可以再聊好幾個小時，但如果因此耽誤你繼續改善這些模型的正事，我就是對世界的損失了。 - Thank you so much for doing this.
非常感謝你答應這次訪談。 - This was a ton of fun.
真的超好玩。 - » Thank you.
» 謝謝你。
Podcast 收尾
- I’m Jacob Efron and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what’s happening with models and what it means for businesses in the world.
我是 Jacob Efron，這裡是 Unsupervised Learning，在這個 podcast 我有機會跟 AI 領域最聰明的人對談，問他們一大堆關於模型進展以及這對世界上各行各業意味著什麼的問題。 - As I hope is clear, I have a ton of fun doing this.
希望大家看得出來，我做這件事非常快樂。 - It’s a nights and weekends project in addition to my day job as an investor at Redpoint.
這是我在 Redpoint 當投資人這份正職之外的夜間與週末專案。 - But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends.
但我們能邀請到這些了不起的來賓，真的是因為各位聽眾訂閱節目、分享給朋友。 - It’s really what ultimately makes this whole thing work.
這才是讓整件事能持續運作下去的關鍵。 - And so, please consider doing that.
所以請考慮這麼做。 - And thank you so much for your support and listening.
非常感謝你的支持與收聽。 - We’ll see you next episode.
下集再見。