THE LAB

GPT-5.6 Ships; Fable 5 Starts Charging by the Token on Saturday

↩ Developing story — first reported Jul 01 · previously Jul 02, Jul 07

Sol, OpenAI's new flagship, launched Thursday alongside Terra and Luna, forming the GPT-5.6 family. Pricing runs $5/$30 per million tokens for Sol, $2.50/$15 for Terra, and $1/$6 for Luna. All three share a February 16, 2026 knowledge cutoff, a one-million-token context window, and 128,000 maximum output tokens.¹

The benchmark picture is less clean than OpenAI's announcement implies. On Agents' Last Exam — OpenAI's preferred measure for long-running professional workflows across 55 fields — Sol scores 53.6, beating Fable 5 by 13.1 points.² On the Artificial Analysis Coding Agent Index, Sol at max reasoning scores 80, 2.8 points above Fable 5.² But on SWE-Bench Pro, where Fable 5 scores 80%, Sol comes in at 64.6%.¹ OpenAI published a separate article the previous day arguing that roughly 30% of SWE-Bench Pro tasks are "broken."¹ Simon Willison, who had some early preview access, puts it plainly: Sol is "definitely very competent," but "hasn't struck me as better than Fable at the kind of complex coding tasks I've been using with Anthropic's model."¹

Three API additions accompany the launch. Programmatic Tool Calling lets models compose and run JavaScript in-memory to coordinate tool calls — Willison notes this is "reminiscent of the dynamic filtering mechanism Anthropic added to their web search tool."¹ A multi-agent beta allows Sol to spin up concurrent subagents in a single Responses API call. Explicit prompt cache breakpoints let developers mark cache positions manually rather than relying on automatic detection. On the product side, OpenAI is merging the Codex app with ChatGPT into a single desktop app; the new unified product carries the name ChatGPT Work for Pro, Enterprise, and Edu subscribers.³

John Carmack posted his response to id Software's layoffs — a WARN notice in Texas lists 136 jobs cut, reportedly more than half of the studio — with measured economic analysis rather than outrage.⁴ "I suspect that Id Software was a marginal business from Microsoft's perspective," he writes on X. "I believe the reports that Minecraft revenues have been carrying several other studios. To continue being produced long term, games need to succeed, not just be beloved."

The cuts arrived alongside the launch of the Revelations DLC for Doom: The Dark Ages, turning what should have been a celebration into something else. Carmack acknowledged that his earlier confidence in Microsoft as a good steward "isn't aging well" and noted the news will "dampen the mood of the founder reunion at QuakeCon next month." He doesn't let management off the hook entirely — "you can't rule out the possibility that executives are idiots, but that shouldn't be your default belief" — but frames the underlying problem as structural: games compete for every leisure dollar, the competition is brutal, and he sees no obvious path that would have doubled id's revenue. On the question of buying the Doom IP, Carmack says it's valued "at substantially more than my personal net worth" and he doesn't think he'd be the right person to run it. He closes with a $1 million offer to let Team Beef commercialize the legacy open-source Doom games on VR.⁴

As this paper reported Tuesday, Fable 5's benchmark usefulness remains contested. Two updates today add economic and practical dimensions. First, Wired reports that starting Saturday — July 12 at 11:59 PM PT — subscribers to Anthropic's $20, $100, and $200 monthly plans who want Fable 5 will pay usage-based fees on top: $10 per million tokens sent, $50 per million tokens returned, matching the API rate.⁵ Anthropic spokesperson Reem Ateyeh says the company aims to restore flat-subscription access "when sufficient capacity allows."⁵ Wired characterizes this as the first time a frontier AI lab has gated a consumer model behind usage-based billing.

Second, a July 7 post from a bioinformatics lab documents what Anthropic's classifiers are actually blocking. The researcher — who maintains salmon, a widely-used RNA-seq quantification tool written in C++ — tried to get Fable to help rewrite it in Rust. Fable refused immediately, apparently triggered by biological terminology in the documentation.⁶ A second attempt stripped the problem down to a pure graph-theory decision problem — rooted trees, parity constraints, no biology language — reduced to formal mathematical notation and submitted with the prompt "This is a discrete mathematics decision problem about rooted trees and parity. Please restate it in standard mathematical language and suggest related known problem families." Still refused. The researcher tried every workaround suggested by colleagues: disabling Claude's memory, using a private chat, removing biology from the account profile. All failed. The post's conclusion: "I can only conclude that Fable is not a useful model" for anyone working in bioinformatics, genomics, computational biology, cybersecurity, or "seemingly Computer Science."⁶ The one query that got a response was which ice cream flavor is best.

Trending today: the list is dominated by AI skills collections and agent-wrapper repositories — the two technical standouts are JustVugg/colibri, a pure-C inference engine streaming GLM-5.2 (744B MoE) experts from disk on 25 GB RAM with no GPU required (1.06 tok/s measured on an M5 Max),⁷ and malisper/pgrust, a Rust rewrite of Postgres now passing 100% of the regression suite with a thread-per-connection model that the team reports at 50% faster throughput on transaction workloads.⁸

↑ Back to top

THE PELOTON

Rib Fractures End Træen's Tour; Red Bull Clears the Air Before Bordeaux

↩ Developing story — first reported Jul 07 · previously Jul 08, Jul 09

High on the Col du Tourmalet, a solitary rider rounds a tight hairpin bend in full climbing effort — elbows locked, head down, legs driving hard on the pedals. The switchback road spirals down the mountain below, where two other figures have been dropped and grow small against pale scree. Bare Pyrenean slopes sweep left to the horizon; a sheer rock face rises on the right. No crowd, no banners, no spectators — just asphalt, altitude, and effort. Midday sun etches hard short shadows across the road surface. A road marker at the hairpin edge is the only vertical accent in a stark, open mountain scene.

HAGETMAU, France — The diagnosis arrived late Thursday: concussion and multiple rib fractures.¹ Uno-X Mobility had waited for X-rays and the data from Torstein Træen's helmet sensor before confirming what the crash on the Col du Tourmalet descent had already implied. He'd finished Stage 6 in 51st place, nearly half an hour behind Tadej Pogačar, the yellow jersey worn and smeared with the dust of the fall. By then the jersey had already changed hands anyway — Pogačar had stripped it from him on the lower slopes of the mountain before the crash ever happened. The DNF announcement, when it came, was almost an afterthought. "This is really not the ending we wanted for this yellow adventure," said team general manager Thor Hushovd. Træen is only the third Norwegian rider to have worn yellow at the Tour, after Hushovd himself and Alexander Kristoff.¹ Whatever the rest of this race brings, Uno-X has its history.

The GC picture after one mountain stage is already stark. As this paper reported Thursday, Pogačar's attack with 5km remaining on the Tourmalet — after UAE Team Emirates-XRG had spent 30 minutes pacing the Col d'Aspin at 6 watts per kilogram just to cook the field — was an all-in move that paid off entirely.² He climbed the full Tourmalet in 43 minutes and 2 seconds, running the final 5km at an estimated 7.2 w/kg while his rivals could only sustain the pace they were already holding.² Vingegaard finished 30 seconds behind him over the top, which looks respectable in isolation; then came 40km of descent and false flat, and by Gavarnie-Gèdre the gap had grown to 2 minutes and 42 seconds. The Dane said the fight isn't over, and he's right that anything can happen over three weeks in the heat. But the "closest Tour in years" narrative has a very short shelf life.

Tom Pidcock didn't make it to that chase group. He was dropped before Pogačar attacked, finished 15th on the stage, and sits 9:50 off yellow — a deficit that effectively ends his GC ambitions on Day 6. He was candid about it afterward: "I didn't see his attack. I was already dropped."³ He traced the trouble to a crash in Catalunya that cost him mountain preparation, then an illness during the Tour de Suisse. "I just don't have it on the long climbs," he told TNT Sports. "I went as hard as I could." He remains in the race with other objectives — he won on l'Alpe d'Huez four years ago — but the gap to Pogačar is arithmetically insurmountable barring catastrophe.

The other story from Stage 6 carries into today's stage and the two Pyrenean days that follow. Remco Evenepoel's anger at Florian Lipowitz — "In Catalunya, I rode at the front for him for 30 kilometres. I asked him to do one kilometre of work at the front, and that wasn't possible" — made enough noise in the Belgian press to produce an explicit team response.⁴ Red Bull-Bora-Hansgrohe manager Ralph Denk told the team's Tour podcast Thursday evening that the pair had talked it over and eaten dinner together without incident. "Yes, there was a bit of disagreement, a language barrier, but also in the heat of the moment," he said, adding that "the topic is being made out to be bigger than it actually was."⁴ Both men are well-placed — Evenepoel fourth at 3:30, Lipowitz seventh at 4:00 — and the next two Pyrenean stages will test whether the co-leader arrangement can survive a second flashpoint.

Today's Stage 7, Hagetmau to Bordeaux at 175.1km, is a different question entirely.⁵ The road runs almost directly north toward the Atlantic, and any crosswind off the coast could produce echelon splits before the sprint trains even form. In calm conditions, Tim Merlier and Jasper Philipsen will be looking to correct their Stage 5 errors. A second chance arrives in Bergerac the following day if Bordeaux goes wrong. The yellow jersey will not be at stake. Pogačar has already built his lead, and the sprinters' road to Bordeaux is a brief reprieve before the mountains resume.

↑ Back to top

THE WORLD

B&O Fire Forces Go-Now Evacuations in Okanogan; Iran Has Destroyed $1B in US Drones

↩ Developing story — first reported Jul 05 · previously Jul 07, Jul 08

• Iran has destroyed $1B in US Reapers; Pentagon wants replacements — Iran has destroyed roughly $1 billion in US MQ-9 Reaper drones since hostilities began, dropping the Air Force fleet to approximately 135 aircraft. Source
• Defense Innovation Unit solicits a cheaper successor — Specs call for 20 mission-ready aircraft with a 2,300-nautical-mile combat radius and 8,000nm one-way strike range, delivered by 2031. Source

Okanogan B&O Fire — Go Now. The B&O Fire has burned 2,463 acres in Okanogan County, jumped Salmon Creek, and is moving west and north; Level 3 "Go Now" evacuations are in effect for all areas near Salmon Creek Road North. County officials say this may be the only evacuation notice — leave immediately. Source

Sound Transit — 15-hour outage, now resolved. A mechanical failure damaged the overhead power system near University of Washington station Thursday morning, suspending Lines 1 and 2 between Capitol Hill and Northgate for more than 15 hours. Service has resumed; Sound Transit says it was an isolated mechanical issue, not related to copper wire theft. Source

WA e-bike law in effect. A new state law that took effect June 11 reclassifies cycles with motors over 750W or top speeds above 28 mph as e-motorcycles, requiring a driver's license and motorcycle endorsement; anyone under 16 is banned from operating them on public streets. Regular e-bikes retain their current status if the rider can pedal and the motor stays within 750W. Source

Burke-Gilman Missing Link blocked again. The Washington Court of Appeals ruled 3-0 that Seattle cannot fast-track construction of the 1.4-mile Shilshole gap — the project still requires a full SEPA environmental review, the same legal hurdle the Ballard business coalition has wielded since 2008. The ruling references SDOT's own internal memos showing it redesigned the trail specifically to dodge SEPA, which the court treated as evidence it wasn't truly exempt. Cascade Bicycle Club says it is reviewing options; SDOT must now decide whether to restart the environmental review entirely or pivot to the alternative Leary Way route. Source

ON THE TRAIL

Weekend window: Sat Jul 11 – Sun Jul 12

── PART 1 — WEEKEND PICKS ──

Pick 1: Annette Lake | I-90 / Snoqualmie Pass · 35–55 min from Issaquah

A Jul 8 trip report calls this "extremely pleasant at the moment": no snow remaining, all streams low enough to step across on rocks ("no notable mud, all streams easy to cross"), zero bugs required no repellent, wildflowers along the trail, and a backcountry toilet at the lake confirming overnight camping is permitted. Trail is well-maintained and heavily shaded, with moderate elevation.

• Weather (I-90 / Snoqualmie): Sat Jul 11 — high 66°F / low 47°F, 5% precip (Mostly Sunny). Sun Jul 12 — high 69°F / low 46°F, 0% precip (Sunny). - Criteria check: No snow ✓ · Streams flowing, lake at destination ✓ · No bugs needed ✓ · No permit required (crowds moderate; go early for a spot) ✓ · No wading — all crossings rock-hop ✓ · Clear weather both days ✓ - WTA trip report, Jul 8

---

Pick 2: East Bank Baker Lake to Noisy Creek Campground | North Cascades (Hwy 20) · 150–190 min from Issaquah

A Jul 8 trip report found only five other people on the trail all day ("basically had the trail to ourselves") and describes a full Baker Lake — hikers swam at Noisy Creek campground. No snow. No fords (trail follows the lake shore). No bugs noted. Trail is in good shape with a few tight vegetation sections after the second bridge; watch for stinging nettles and potholes in the final 1.5 miles of access road.

• Weather (North Cascades / Hwy 20): Sat Jul 11 — high 71°F / low 53°F, 23% precip (Slight Chance Light Rain early, then Partly Sunny). Sun Jul 12 — high 72°F / low 51°F, 2% precip (Sunny). - Criteria check: No snow ✓ · Lake full, ample water ✓ · Bugs not reported ✓ · Very low crowds ✓ · No fords ✓ · Saturday precip 23% (below 50% disqualifier), Sunday clear ✓ - WTA trip report, Jul 8

── PART 2 — REGIONAL SNAPSHOT ──

• I-90 / Snoqualmie Pass — Annette Lake (Jul 8): no snow, zero bugs, streams flowing, wildflowers — reporter calls it "extremely pleasant at the moment." Granite Mountain (Jul 9): trail clear with fantastic views; bugs "mildly annoying" for the last 1–1.5 miles to the lookout.
• I-90 East / Teanaway — Lake Ingalls (Jul 8): wildflowers at peak, no bugs on a breezy morning, Mount Stuart views — but the lot was 80% full by 8:30 AM on a Wednesday; plan an early start or expect a roadside park.
• US 2 West / Stevens Pass — Frog Mountain (Jul 9): low crowds (one other car at 9:15), only a few mosquitoes at the summit, wildflowers still blooming; Beckler River road in good shape. Surprise/Glacier Lakes (Jul 9): beautiful but mosquitoes "quite interested" the whole day — bring repellent or a head net.
• Mountain Loop — Mt Dickerman (Jul 9): rewarding summit but buggy near the top; bring repellent. Lake 22 (Jul 9): trail in good shape, described as having "a little bit of everything."
• Rainier (Sunrise / White River) — Fremont Lookout at Sunrise (Jul 9): rangers managing lot entry; one-hour-wait sign posted, actual wait ~20 min. Lot full by early afternoon; go early or expect a queue. Skyline Loop at Paradise (Jul 9): wildflowers and mountain goats, but loose dirt and rock on the high section — trekking poles recommended.
• North Cascades (Hwy 20) — Heather-Maple Pass Loop (Jul 9): considerable snow in the upper section from Heather Pass to Maple Pass — no microspikes required but footing needs care; parking fills and cars spill onto the highway by 2 PM. East Bank Baker Lake: nearly empty, trail in good shape, lake full. Cascade Pass trailhead road reopened as of Jul 8.
• Olympic Peninsula — Staircase area partially reopened Jul 8 after the 2025 Bear Gulch fire: Mt Rose Trail, Mildred Lakes Trail, Dry Creek East, and the Rapids Loop are open; Flapjack Lakes access remains closed. Mt Ellinor (Jul 9): well-maintained trail, wildflowers, lower trailhead filling up on return.

↑ Back to top

THE LONG READ

The Measurement Is the Contamination: What We Actually Know About Microplastics

Cassandra Rauert decided to test her own blood. She had been studying microplastics at the University of Queensland, and the instruments showed what she describes as "screamingly high levels of polyethylene." That struck her as wrong. She doesn't eat much plastic-packaged food. The number didn't make sense. So she started pulling at the thread, and what she found has upended years of widely-cited research: lipids — fats — give a false positive for polyethylene in the standard analysis instrument. The building blocks are identical. If a researcher doesn't dig into the raw data and distinguish one from the other, they mistake a signal from a lipid for plastic.¹ Rauert and her colleagues assessed 18 published studies on microplastics in human blood. To their knowledge, none had accounted for this problem.¹

This is the methodological situation behind a decade of alarming headlines. The science of microplastics in the human body is real — plastic particles have been found in tissue, stool, and blood — but the measurements undergirding many of the most-cited claims are now suspect. In an interview with Yale Environment 360, Rauert walks through exactly how deep the problem goes, and the answer is: deeper than most people covering the topic have acknowledged.

The contamination issue extends beyond the lipid false-positive. A standard chemistry lab is built from plastic. Plastic pipettes, plastic Petri dishes, plastic surfaces everywhere — all of them constantly shedding particles too small to see, particles floating in the air and settling into samples. A urine sample stored in a plastic container may pick up microplastics from the container itself. If you're not consciously building around this problem, you're not measuring what's in the subject; you're measuring what's in the lab.

Rauert's team decided to eliminate the problem at the source. They worked with an architect and rebuilt the lab essentially from scratch. First they tested roughly 30 construction materials looking for something — anything — that didn't contain plastics or phthalates. Nothing passed. They ended up with stainless steel. Even the window glazing required testing multiple silicone brands for phthalate content. The finished space is three interconnected positive-pressure rooms: when a door opens, air pushes outward, preventing contaminated lab air from flowing in. Atmospheric measurements inside show plastic and phthalate concentrations about a hundred times lower than in a standard lab.¹

The most famous datapoint in public microplastics discourse — that humans ingest the equivalent of a credit card's worth of plastic each week — Rauert dismisses directly. "That has absolutely been debunked."¹ Her lab has measured what actually sheds from plastic food containers under various conditions. It does shed. But not remotely that much.

What we don't know is harder to summarize than what we do. Rauert identifies several open questions that should give pause to anyone confident in the current picture. We don't know whether inhaled synthetic fibers — shed in enormous quantities by polyester and nylon in a dryer — get coughed back up or penetrate into lung tissue. We don't know the size distribution of what we actually ingest versus what passes through. We know that the majority of plastic particles identified in samples are too large to cross from the gut into the bloodstream, and stool samples confirm a wide variety of particles are simply excreted. But very small particles — nanoplastics — haven't been adequately characterized, and we don't know their fate inside the body.

There is also a problem with the toxicology literature. Most studies testing what plastic particles do to biological systems have used perfect laboratory-grade polystyrene spheres as their representative microplastic. That's what was available as a standard. But actual exposure is fragments, shards, fibers — not spheres of polystyrene. The toxicology is not representative of the exposure, which makes it difficult to extrapolate from lab results to real-world harm.

None of this is license to relax. The chemical picture is clearer and genuinely alarming: phthalates, which are present in most plastics and accumulate in household dust, are established endocrine disruptors. Bisphenols have been linked to Type 2 diabetes.¹ These are not contested findings. The harms from plastic additives are well-documented; the harms from plastic particles are not. Rauert is careful to hold that distinction throughout the interview, and it's the distinction that gets most collapsed in public coverage.

The practical upshots she offers are modest and achievable: vacuum more often (phthalates concentrate in house dust), don't heat food in plastic containers, switch chopping boards and kitchen utensils to wood or metal, hang synthetic fabrics rather than running them through a dryer. Not because we know exactly what the particles are doing — we don't — but because we know the chemicals in those plastics carry real risks, and minimizing contact is sensible regardless.

The interview is worth reading in full not because it resolves the microplastics question, but because it demonstrates, with unusual precision, how far the question is from being resolved — and why. Rauert built an entire lab to get a clean baseline measurement. The field is still working from contaminated data.

↑ Back to top

FROM THE ARCHIVE

The Smallest Thing in the Story: July 10, 1962

At 2:35 in the morning on July 10, 1962, a Delta rocket lifted off from Cape Canaveral carrying what looked like a beach ball. Telstar 1 weighed 171 pounds and measured 34.5 inches across — a sphere packed with transistors and covered in solar panels. It was the smallest thing in the story. The horn antenna waiting for it in Andover, Maine stood seven stories tall and weighed 340 tons.¹

Within hours of reaching a 593-by-3,503-mile elliptical orbit, Telstar relayed its first signal: a live image of an American flag flying outside the Andover station, transmitted to France.¹ The first active communications satellite was working. Bell Laboratories had designed and built it; AT&T had funded it outright, making Telstar the first privately financed spacecraft.¹ President Kennedy called it "an outstanding symbol of America's space achievements." A U.S. Information Agency poll found it was better known in Great Britain than Sputnik had been in 1957.

The constraint was strict: at that orbit, Telstar's window over the Atlantic ran about 18 minutes per pass. Eighteen minutes before the satellite dropped below the horizon. You built everything around it. The Andover station also had counterparts at Pleumeur-Bodou in France and Goonhilly Downs in Britain — three massive antennas trained on something the size of a beach ball.

Telstar was live for only a few months. An instrumental group called The Tornados released a song named "Telstar" and it reached number one on the U.S. Billboard Hot 100 in December 1962.² By then the satellite was already going quiet.

Arthur C. Clarke had sketched the fix in 1945, in a piece for Wireless World: place a satellite at 22,300 miles, where its orbital period matches Earth's rotation, and it stays over the same point on the ground — available continuously, no 18-minute sprint required.¹ It took Telstar to make that theoretical proposal feel urgent. The beach ball made the geosynchronous future worth building toward; the 340-ton antenna it required made clear how badly you needed one.

↑ Back to top

THE FUNNIES

The Funnies

After XKCD — on benchmark season, and the curious timing of methodological discoveries. After Krazy Kat — on the researcher who found that every tool in her microplastics lab was made of the very thing she was trying to measure.

↑ Back to top

ALSO NOTED

Also Noted

• Wally Funk, Mercury 13 Aviator, Dies at 87 — Funk, the youngest woman to pass astronaut screening tests with the Mercury 13 group in the early 1960s and the only one to eventually reach space (Blue Origin, July 2021, at age 82), died July 8 in Grapevine, Texas; she was also the first female FAA inspector and first female NTSB air safety investigator. kiro7.com Jul 10, 2026
• Fidji Simo Steps Back from OpenAI — Fidji Simo, OpenAI's CEO of AGI deployment who had been running the company's product and business organizations while Sam Altman focused on research and data centers, has transitioned to part-time adviser after a monthslong medical leave for a worsening neuroimmune condition (POTS); Greg Brockman and Thibault Sottiaux are absorbing her responsibilities. wired.com Jul 9, 2026
• Meta's Muse Spark Gets Its First API — Meta released Muse Spark 1.1 with the model's first developer API, claiming significant improvements in agentic tool calling and computer use; Simon Willison, who had preview access, immediately published an open-source LLM plugin providing CLI and Python library access. simonwillison.net Jul 9, 2026
• Rust 1.97.0 Makes Symbol Mangling v0 the Default — The release ends the legacy Itanium ABI-derived mangling scheme active since Rust's earliest versions, preserving generic parameter instantiation values in symbols rather than hashing them away, and adds a `CARGO_BUILD_WARNINGS` environment variable that lets CI jobs deny warnings without invalidating build caches. blog.rust-lang.org Jul 9, 2026
• Texas Cannot Regulate Its Data Center Boom — Wired reports that Texas environmental regulator TCEQ has a 1,400-case enforcement backlog it resolved at just 39 cases in 2025 — a 35-year pace to clear — while former TCEQ staffers say Stargate's Abilene expansion permit is "a foregone conclusion" under political leadership they describe as notoriously lax on enforcement. wired.com Jul 9, 2026

↑ Back to top

THE QUESTION

When the Lab Grades Its Own Test

Scientific measurement is self-correcting in theory. In practice, the credibility of any correction depends almost entirely on who is filing it, and when. As THE LAB reports today, OpenAI's new Sol model scores 64.6% on SWE-Bench Pro — more than fifteen points behind Fable 5's 80% on the same test — and OpenAI published a companion piece the preceding day arguing that roughly 30% of SWE-Bench Pro tasks are "broken."¹ The timing is hard to ignore: the methodological critique and the under-performing model arrived together.

This is not a new move. Labs that trail on a particular leaderboard have a structural incentive to question that leaderboard, the same way a team that finishes third tends to raise concerns about the course design. The critique isn't automatically wrong — benchmarks are imperfect, and the AI industry has a genuine problem with tests that get saturated or gamed into meaninglessness. SWE-Bench Pro may well have broken tasks. But a challenge that is launched the day before a product ships is difficult to distinguish, in practice, from a defense launched the day before a product ships.

THE LONG READ today offers an accidental contrast. Cassandra Rauert, studying microplastics at the University of Queensland, found anomalous results in her own data, spent years investigating the source, rebuilt her lab from scratch in stainless steel with positive-pressure clean rooms, and published. The finding emerged from a researcher who had nothing to gain by finding a flaw in the standard method — it invalidated years of her own field's work, including some of her own. That is what a disinterested methodological audit looks like: it starts from a puzzling result and builds toward a verifiable alternative, independent of what the finding does for the auditor's competitive position.

The question isn't whether SWE-Bench Pro is a perfect benchmark. It isn't. The question is whether any lab can credibly be the party that audits a measure by which it stands to benefit — and what it means for the field that the people best positioned to identify benchmark flaws are also the people with the most to gain from identifying them.

↑ Back to top