新宿天鹅

OpenAI 将崩溃调试视为流行病学研究，修复了存在 18 年之久的 GNU libunwind 漏洞_我的网站

一 |     作者 | Steef-Jan Wiggers     译者 | 平川          OpenAI 的工程师们花了数周时间，试图解释 Rockset 中那些神秘的崩溃问题。Rockset 是一款 C++ 数据基础设施服务，为 ChatGPT 的搜索和数据插件提供支持。函数似乎会返回错误的内存地址，栈指针在执行过程中似乎会偏移 8 个字节。    The Israel-Hamas conflict is taking a "horrendous toll" on families, humanitarian organizations like UNICEF decried this week, amid reports of the slaughter and kidnapping of children and attacks on civilian infrastructure that have killed, injured or displaced the most vulnerable.In the days since Hamas' surprise assault on Israel, images from both regions have shown crying children running through the street and cowering in bomb shelters after airstrikes. In Gaza, the bodies of dead children killed in shelling were covered in blankets and carried by their fathers in funeral processions. In the kibbutz of Be'eri, one of the largest in Israel, more than 100 bodies of Israeli citizens were discovered on Monday, with women, children and the elderly "brutally butchered," the Israel Defense Forces said. Israeli children have also been among those reported kidnapped by Hamas terrorists.

A special edition of "20/20" at 10 p.m., Wed., Oct. 11, on ABC looks at life under threat in Israel and Gaza and how this attack is different from what we’ve seen in this region in the past.

"Nothing justifies the killing, maiming or abducting of children -- grave rights violations which UNICEF wholeheartedly condemns. Yet less than 72 hours after the outbreak of horrific violence in Israel, reports indicate that grave rights violations against children are rampant," UNICEF Executive Director Catherine Russell said in a statement on Monday. "Many children have been killed or injured, while countless others have been exposed to the violence."According to Palestinian authorities, 900 people have been killed in Gaza so far -- including 260 children and 230 women. The number of children killed in Israel is unclear; at least 900 people have died and 2,600 others have been injured, officials said, though did not specify how many were children. Prime Minister Benjamin Netanyahu said in a speech Monday that among the "atrocities" committed by Hamas, children have been "executed with the rest of their families."In remarks at the White House Tuesday, President Joe Biden described "Hamas' bloodthirstiness" as reminiscent of ISIS rampages -- including "stomach-churning reports of babies being killed."At least 100 civilians and soldiers have also been taken hostage by Hamas militants, Israeli officials said. Hamas leaders on Monday threatened to begin killing hostages one by one and filming the executions if their demands are not met.Among those abducted were 12- and 16-year-old brothers, their mother told ABC News. The woman, who asked not to be identified for security reasons, said the boys were abducted Saturday by Hamas militants who burst into a safe room at their father's home on a kibbutz near the Gaza border."I want the world to demand the release of those innocent civilians. I want these children and women and babies back home, and I want my children back home," the mother said. "I can't take a shower without thinking of them being held hostage in some dirty pit somewhere. I can't eat, I can't sleep. I don't think human beings treat people like this. I'm sorry. I want the world to know, to demand those hostages to be returned to their homes."In response to the assault, Israel on Monday carried out a "complete siege," cutting off power and blocking food and water from being delivered to the Gaza Strip -- where, according to the CIA, nearly 40% of the population of 2 million is under the age of 15.UNICEF is "extremely alarmed" about those measures, spokesperson James Elder said at a press briefing Tuesday in Geneva."This will add another layer of suffering to the existing catastrophe faced by families in Gaza," Elder said. "Depriving children of access to food and essential services puts their lives at risk, as do attacks on civilian areas and infrastructure."According to UNICEF, 80% of those who live in the Gaza Strip rely on some form of humanitarian assistance.The conflict has led to "grave humanitarian consequences," Lynn Hastings, a humanitarian coordinator for the Gaza Strip for the United Nations Office for the Coordination of Humanitarian Affairs, while calling for all parties to abide by international humanitarian law."Civilians, especially children, medical facilities, humanitarian personnel health workers, and journalists must be protected," Hastings said in a statement Tuesday. "Captured civilians must be released immediately and unconditionally."UNICEF has also called on all parties to protect children from harm, in accordance with international humanitarian law."I remind all parties that in this war, as in all wars, it is children who suffer first and suffer most," Russell said.。团队提出的每一种假设都面临着有力的反证。这个 Bug 似乎根本不可能存在。

二 |          他们原本以为是一个 Bug ，结果却发现是两个互不相关的 Bug ，它们只是很巧合地在同一时间被发现了。这一突破性发现并非来自对单个崩溃事件的深入排查，而是源于转向了他们所说的“流行病学调试”：构建一条管道，自动分析过去一年中生产环境的每一个核心转储文件，然后寻找整体规律，而不是对单个案例进行推断。

三 |          该团队让 ChatGPT 编写了一个脚本，用于下载每个核心文件的开头部分，提取寄存器数据，过滤已知的误报，并将每次崩溃标记为“返回空指针”、“栈对齐错误”或其他类型。他们将该脚本并行应用于过去一年中的所有 Rockset 核心转储文件。

四 | 他们很快就发现了相关性。

五 | 原本从症状上看属于同类的问题，实际上对应两组特征完全不同的崩溃事件。

六 |          这些因栈对齐错误导致的崩溃均源自同一个 Azure 区域，有明确的起始日期，而且从未出现在长期运行的节点上。团队追踪发现，这些崩溃源自一台物理主机，其 CPU 正在悄无声息地产生错误的结果。它既没有过热，也没有抛出机器检查异常，而只是数学运算默默出了错。

七 | 将该主机从服务中移除后，因栈对齐错误导致的崩溃便完全消失了。         在剔除硬件崩溃问题后，剩余的由“返回空指针”导致的崩溃问题便变得可控了。

八 | 此前，团队曾排除了 C++ 异常展开的原因，因为他们认为自己找到了反例：在未使用异常的代码路径中发生了崩溃。但这些反例全都来自有硬件损坏的故障集群。一旦剔除了这些干扰因素，剩余的所有崩溃便都是发生在异常展开过程中了。         问题发生的根本原因是 GNU libunwind 的 _Ux86_64_setcontext 函数中有一个已经存在 18 年的竞争条件。在 C++ 异常展开过程中，libunwind 会在栈上合成一个 ucontext_t 结构体，填充所需的寄存器状态，然后调用 _Ux86_64_setcontext 将控制权转移给清理处理程序。

九 | 问题在于：_Ux86_64_setcontext 在从旧结构体中读取指令指针的操作尚未完成之前，就将栈指针（%rsp）更新为指向新的栈帧。一旦 %rsp 发生变化，该结构体便不再属于活动栈的一部分，也不再受内核红区的保护。如果信号恰好在 %rsp 更新与 %rip 读取之间的这一时间窗口内到达，内核就会在该结构体之上构建其信号帧，指令指针遭到破坏，函数便会跳转到 NULL 或垃圾地址。         竞争窗口的宽度正好为一条指令。以现代处理器的时钟频率计算，这大约相当于 100 皮秒。在大多数程序中，这种情况根本不会被触发。

十 | OpenAI 的 Rockset 使用了 timer_create 函数，每隔几毫秒的 CPU 时间就发送一次 SIGUSR2 信号，为的是实现轻量级的按查询记账，这样产生的信号发送事件远多于传统的应用程序。正是这种高频的信号发送，将只在理论上可能发生的竞争状况转化成了实际生产环境中的崩溃。

十一 |          该团队将一个修复方案和一个自包含的重现示例提交到了 GNU libunwind，并通过验证证实，其他展开器（如 libgcc）不存在这个问题。该修复方案通过重新排序指令，确保在更新 %rsp 之前先读取 %rip，从而彻底消除了这个时间窗口。

十二 |          该团队对这一教训的总结值得全文引用：          最重要的步骤并非巧妙地解读汇编代码，也不是对细节的深入了解，而是构建一个高质量的数据集。如果没有这个数据集，我们就会把两种截然不同的现象混为一谈，并试图通过推理来理清这种混乱。

十三 | 一旦获得了准确且完整的全量数据，问题的结构便显而易见了。         如果你的团队正在排查难以解释的生产环境崩溃问题，请检查你们是否将多个 Bug 混为一谈。那些看似与所有假设都不相符的症状，实际上可能并不矛盾；它们可能与两个不同的假设相符，而你却无意中将它们混淆了。洞察问题结构的最快途径，并非对单个案例进行更深入的分析，而是获取涵盖所有故障案例的完整、带标签的数据。         这篇完整的工程技术博文包含了详细的栈内存示意图、存在漏洞的汇编指令，以及揭示出两种不同类型故障的崩溃率可视化图表。         https://www.infoq.com/news/2026/07/openai-libunwind-core-dumps/          声明：本文由 InfoQ 翻译，未经许可禁止转载。

Current article:http://9dx7yzy.shangfochengcongshuaizhuang.buzz/qr8fgg/20260826/98544.html

Published on:20:11:33