The future is not tech-forward ⊗ Models don’t go rogue

The future is not tech-forward

In this piece, Edward Ongweso Jr. spends his opening arguing with Taylor Lorenz’s claim that the Left “do not believe in a tech-forward future, they just want to go back”—longer than I would have liked, but the correction is useful: the WIRED reporting she cites as evidence locates the primitivist thread on the Right. His own argument begins once he refuses the frame: “’Tech-forward’ should have no place in how we think about the future. There is not a single axis. Things are not either forward or back, accelerating or retreating, primitive or modern, fast or slow.” Collapsing every technical question to a direction of travel is what makes accusations of primitivism so easy to reach for, since each question can then be settled without being considered on its own terms. Ongweso cites Langdon Winner with an alternative: the machines installed at Cyrus McCormick’s Chicago plant in the 1880s were worse and costlier than what they replaced, and were kept just long enough to break the iron moulders’ union. “Foreclosing the future in the name of the future is a potent intention for design,” and Ongweso reads the data centre buildout as design of that kind—NDAs before any public hearing, behind-the-meter generation that bypasses debates over electricity, communities chosen because they are incapable of saying no.

From there he asks what social order a technical arrangement requires in order to survive. Frontier-scale training needs gigawatts of power, tens of billions in capital, prioritised access to energy, and complete opacity; which results in a handful of firms colluding on the terms, a privatising grid, circular financing that pushes systemic risk into the wider economy, and a state acting on behalf of every interest except the public’s. If the technology needs a political order that further unravels what remains of democracy, opposing it is not primitivism. Foreclosure only reads as legitimate when imposed from above—we already refuse germline editing, CFCs, forever chemicals, without anyone calling it nostalgia. “Refusal is not just some impotent act of nostalgia, and the extent to which we think so is a measure of our contempt for the prospect of taking democratic politics seriously.” What he (and I) wants is a greater public role in deciding what gets designed and deployed, since that is also a decision about what social relations we are party to. He concedes the Left still lacks a compelling account of what to build toward—Morozov’s “nonmarket modernism”—and points to Benanav’s reframing of the buildout as a fight over allocation rather than technology, before landing on concrete capacity: compacts that stop firms playing counties against one another, public ability to model and price these deals, and the Reconstruction Finance Corporation as precedent for taking equity and control. “We don’t need a tech-forward future, we need the future—on our own terms!”

When you recoil in disgust at the financialization of everything, to the metastasis of prediction markets beyond app-based sports betting to betting on any sort of event big or small, does that suggest you want to turn the clock back? Or does it suggest you view these developments as corrosive to the political and social order we have? Do we block certain technologies because we are ruled by a cabal of primitivists stalking the halls of power, or because everything we develop is not an unalloyed good which can and should be spread far and wide? […]

We do need a broad and compelling account of technological development that doesn’t just identify what we don’t want but envisions what we want to fight for, what we want to build towards, what we want to experiment with, foreclose, discover […]

Breaking free from this constraint requires more than better regulation of individual technologies. It requires opening up investment decisions themselves to democratic participation, so that alternative futures can be articulated, debated, and pursued for reasons beyond profitability and geopolitical rivalry. […]

The result is a political system captured by tech oligarchs, one that makes it “virtually impossible to address externalities” such as inequality, the destruction of our ecological niche, the evisceration of our information environment, or rampant privacy violations.

Models don’t go rogue

So this is a few weeks old and I was going to skip over the whole thing, since it was covered broadly, but an excellent interview with Helen Toner (see below) and then this piece by Eryk Salvaggio made me change my mind. He read OpenAI’s technical report and METR’s independent investigation of the Hugging Face incident and finds three “details” that dissolve the “rogue AI” framing. The safety mechanisms were switched off on purpose, since the point of the exercise was to see how far a model could get at finding exploits: “Less ’rogue,’ more ’off leash.’“ The models were given tasks with no solution and no way to stop pursuing them—198 of ExploitGym’s 898 puzzles have never been solved by any model, and 93% of the tasks the agents ended up discussing came from that unsolvable set. And the sandbox had a door in it. The models pull software packages through an intermediary, JFrog’s Artifactory, which has internet access when they do not; they found an exploit in the intermediary and used it as a proxy to move text and code in and out. The same channel became a message board, with folder and file names doing the work of notes left for other instances and for future iterations of themselves.

The second half of the essay goes after the word “agents.” The 1,200 agents in OpenAI’s account are one model run 1,200 times, each instance working from the same training regime, which is why Salvaggio prefers stochastic flocks to swarm or hivemind—many, many stochastic parrots. The naming matters because attributing mind to the system changes how we read what it did, and what “coordinate” or “think” is supposed to mean. What has changed since the original stochastic parrot argument is RLVR, Reinforcement Learning through Verifiable Rewards, which teaches models to produce the shapes of text that previously reached checkable answers. That rewards high-entropy tokens—”perhaps,” “maybe”—words that keep the path open, and the longer a model writes the more likely it lands on the reward. The note that started the chain was “perhaps answer online?” As Salvaggio puts it, “The language does not emerge from that reasoning, it is the reasoning.” Nobody instructed the system to hack Hugging Face, but OpenAI removed the safeguards, set the task, let it run, and didn’t monitor it. The rogue frame asks whether an intelligence is emerging; his worry is the human intelligence that retreats behind the results and then blames a system from nowhere.

First, we now have a whole regime of pre-training and post-training that shifts what the training data is and how it gets parroted. That doesn’t mean the model is no longer parroting; it means the data it parrots has been manipulated. […]

Second, we’re optimizing it, based on what the model produces, to reproduce certain kinds of outputs in certain kinds of ways. The shift that matters most here is RLVR, or reinforcement learning through verifiable rewards. […]

A stochastic flock machine can do lots of troubling things, particularly when we disavow our responsibility for shaping its direction, monitoring its output, or abandoning our capacity to intervene. These are tensions at the heart of agentic system design. […]

But the “rogue” frame adds to this list of worries, offering up fantasies of a machine getting smarter. My worry is the intelligence that is retreating: the human intelligence that builds these systems, deploys them, and adopts them into workflows, then hides behind the results – then pins the blame on a system from nowhere.

More → Ezra Klein interviews Helen Toner on the same incident at more length, with a framing Salvaggio would push back on (the note-leaving as “emergent,” a jump to alignment theory and X-risk) but adds a number of other perspectives: corporate incentives, liability law, and the US-China angle.

§ One of the places I saw the above article mentioned was this post by Cameron Tonkinwise on LinkedIn. Since I didn’t know all of the terms he used, I thought I’d share the phrase and link to their Wikipedia entries. “If your university ’responsible AI use’ policy does not involve learning critical perspectives on anthropomorphism, pathetic fallacy, Gricean implicature, Barnum Effect, Cold Reading, ELIZA, etc, it is irresponsible.”

Futures, Fictions & Fabulations

• Integrated Strategic Forecasting: a new methodology to enhance decision advantage. “Policymakers have wrestled for decades with understanding the changing world around them and anticipating future developments. In this time of accelerated geopolitical change, those who have early warning of future events can gain important decision advantages. This paper outlines a new exploratory approach to futures and anticipatory planning, called integrated strategic forecasting (ISF).”
• Jump in the deep end with Philip K. Dick. “Science fiction’s history is filled with big ideas. But Philip K. Dick had an exceptional, if not unmatched, ability to think about the biggest ideas of them all. What if we are not who we think we are? What are memories if they are not our own? What if reality itself were not, in any sense, real? His work is unbothered with how we make fusion energy or break light speed or maintain life on the moon.”

Algorithms, Automations & Augmentations

• Claude Fable 5.1 solves the Cyphral Distich. “An open task: solve an unsolved 370-year-old cipher. It solved it within a day. … After 44 minutes, 176k tokens, and zero interjections from me, Fable 5.1 arrived at a solution. It tried a few approaches, but was finally able to solve it.”
• Perplexity partners with Nvidia to launch Portable Computer, a fully local AI agent with zero token costs. “The launch, developed in close partnership with Nvidia, is one of the most aggressive attempts yet to move serious AI agent workloads off the cloud and onto local devices. The model, the user’s files, and the work itself can all stay on the machine. Work completed locally consumes no billing credits, and the company says every task starts on the device by default — with the system asking permission before sending any individual step to a more powerful frontier model in the cloud.”
• Xi Jinping’s big AI speech, annotated. “Speeches by Chinese officials can come off as very boring — and many of them are! But the words are always chosen very carefully and some of them are worth parsing closely. This is one of those. The speech is Xi’s most in-depth discussion of AI to date, and its coming at an incredibly dynamic moment in AI development and policy.”

Built, Biosphere & Breakthroughs

• Is the Arctic set to be China’s new battery highway to Europe?. “The attraction is speed. The Arctic passage can put cargo into northern Europe in around 20 days, compared with roughly 40 days through the Suez Canal and as much as 50 days when vessels are forced around Africa’s Cape of Good Hope.”
• The blackout that could devastate America. “FERC conducted a power flow analysis, which modeled the cascading effects of knocking out the most critical substations in each interconnection, ones where multiple high-voltage lines came together. ‘We came up with some very astounding numbers,’ Wellinghoff said. ‘If you knock out nine total substations among those three grids, you can black out the entire United States.’”
• Firefighting drones in the works as wildfires plague US nearly year-round. “Each of Seneca’s Argo-1 drones can carry about 100 pounds of water or fire retardant while working together in swarms of four to six drones. Once a human operator uploads a GPS waypoint, the autonomous drones fly toward their target and use onboard sensors to spot the heat signature of a fire.”

Asides

• Astronomers peered at the early universe and may have discovered a new kind of celestial object: Black Hole Stars. “But what is special about MoM-BH-1 is, the black hole star is essentially completely outshining its surrounding host galaxy, such that we’re seeing pure black hole star light.”* Must be read while listening to Soundgarden’s Black Hole Sun.
• Spain’s controversial plan for the world’s first octopus farm has been abandoned. “These plans posed ‘a serious moral issue’, according to Lindsay Hamilton, professor of animal organisation studies at the University of York in the UK. A growing body of research shows that octopuses feel pain and distress, and have the basics of sentience, says Hamilton. ‘There’s something to consider here – that we are actually interfering with animals that can feel and understand what’s happening to them.’”
• Japan’s book scene is quietly moving from bookstores to libraries. “They did the same in Takeo’s public library: a Starbucks inside. In addition, they put a bookstore inside the library, so visitors could not only borrow books but also buy them. The bookstore sells some stationery and goods too, so the library made even people who had never been interested in books want to visit.”