Aaronson-Questing and Art-Questing: At the Edges of an AI Model’s Training
What an AI Safety Theorist and an AI Artist Each Want from the Edges of a Model’s Training
Joe Nalven, with Gemini and Claude
Introduction
Scott Aaronson is a theoretical computer scientist at the University of Texas at Austin, best known for his work on quantum computing and computational. In 2022 he took a two-year leave to work on AI safety at OpenAI, and he now leads a group at UT Austin that applies theoretical computer science to AI alignment. “Alignment” means getting an AI system to act in line with the intentions and values of the people who build and use it, presumably oriented to human safety, individual and societal.
In a lecture in the Della Pietra series at Stony Brook University, Aaronson named three areas where theoretical computer science can help: watermarking the output of AI systems, mechanistic interpretability (working out what the internal parts of a neural network are actually computing), and theoretical guarantees for out-of-distribution generalization. The last asks: what does an AI model do when it meets a situation unlike the examples it was trained on?
Listening to Aaronson (in a related lecture), I was struck that my own practice as a visual artist using generative AI pulls in nearly the opposite direction. He wants to keep a model reliable at the edges of its training. When I make images, I am often trying to get the model to stop doing what it reliably does. This article compares the two efforts, which I call Aaronson-questing and art-questing, using two of my AI-generated images.
It is tempting to treat the two quests as mirror images on either side of one mathematical line. That is too loose. Below, I separate three ideas that are easy to run together, then identify three points where the comparison is exact rather than metaphorical, and four points where it does not hold.
Three Terms That Are Easy to Confuse
Training distribution. The range of examples a model was trained on, and how often each kind appears. An image model trained on billions of pictures has seen many photographs of museums, many portraits, many windows, and many paintings in many styles.
Out-of-distribution input. A situation or request, met after training, that differs in kind from the training examples. It is a statement about the input. A self-driving system trained only on dry roads meeting its first snowstorm is the textbook case. This is Aaronson’s concern.
Low-probability output. A result the model can produce but rarely produces by default. A portrait mixing stained glass, paint, and a damaged photograph may be rare in a model’s output without being out of distribution, because each ingredient appeared often in training. Only the combination is rare.
A model’s default (statisticians call the most frequent value the “mode”) is the kind of result it produces most often for an ordinary request. Commercial tuning pushes models harder toward their defaults.
The consequence for this article: most of what an artist does with a generative model is better described as moving away from the model’s default than as taking the model outside its training distribution. Art-questing mainly works with rarity, not with novelty in the technical sense. The exception is when an artist feeds a model’s output back into the model for further rounds; the later rounds may then receive inputs unlike anything in the training data.
What Aaronson’s Alignment Work Actually Involves
A rule about how to make progress
Aaronson has explained why he stayed away from AI alignment for years. In his view, a research field needs either experimental data or a clear mathematical theory, something outside the researcher that can show when the researcher is wrong. Until recently, alignment had neither. Large models now supply the data. This rule matters later, because art has no external test of that kind.
Failures that do not look like failures
Older rule-based programs, written as explicit IF–THEN instructions, often stop or report an error when no rule applies. A neural network does not stop. Because what it has learned is spread across billions of numerical weights, it always produces some answer, usually fluent and plausible, even when the situation is unlike anything it was trained on.
That is the danger. In a well-known example, an image classifier separating huskies from wolves turned out to be looking for snow in the background rather than at the animal. It scored well on test images that shared that pattern and failed on images that did not, with no sign that anything was wrong. The worst out-of-distribution failures look like successes.
The tools
Watermarking. A language model writes one small unit of text (a token) at a time. At each step it computes a probability for every possible next token, and the computer picks one at random according to those probabilities. Aaronson’s scheme replaces that random choice with a pseudorandom one, made by a cryptographic function using a secret key. To a reader, the text looks the same. To someone holding the key, the pattern of choices can be detected statistically. Aaronson has acknowledged that a determined user can defeat the scheme, for example by paraphrasing the text.
Interpretability and guarantees. Interpretability tries to read a model’s internal computations directly. Guarantees for out-of-distribution behavior try to prove, rather than hope, that a model will behave acceptably on unfamiliar inputs; this is the least developed of the three areas.
A related proposal, “AI safety via debate,” in which two AI systems argue opposite sides before a less capable judge, was published in 2018 by Geoffrey Irving, Paul Christiano, and Dario Amodei. It borrows from interactive proofs, a topic in Aaronson’s field, but it is not his proposal.
Aaronson’s 2023 list of places where near-term alignment progress is possible also includes reinforcement learning from human feedback (RLHF): people rate a model’s outputs, and the model is adjusted to produce more of what was rated highly. This is where Aaronson’s field and the artist’s experience actually touch.
Two Images
Figure 1: The gallery image
Figure 1. The gallery image. AI-generated image by Joe Nalven.
A man with graying hair, in a dark jacket and shirt, stands in a bright modern gallery under a glass roof, holding a carved wooden cane. He is frowning and looking away from a large, brightly colored abstract painting toward something outside the frame. Visitors walk through the room behind him. The light, shadows, and perspective all agree. The image has the finish of a professional editorial photograph.
The subject is a viewer who is not persuaded by abstract painting, and the style is the most conventional one available: realistic photography. The style agrees with the man. Nothing in how the image is made challenges what he refuses to look at.
Commercial image generators produce this kind of result readily for two well-established reasons. Image models are tuned with methods closely related to RLHF: people rate images, and the model is adjusted toward images rated attractive and accurate. And they use a setting usually called “guidance,” which increases how closely an image follows the prompt and how polished it looks, at the cost of variety.
Here the comparison with Aaronson’s field becomes exact. A 2024 study by Robert Kirk and colleagues measured what RLHF does to language models on two properties at once: out-of-distribution generalization and output diversity. RLHF improved the first and significantly reduced the second. The study examined language models, not image models, but it names precisely the tradeoff this article is about. The tuning that makes a model more reliable on unfamiliar requests, which the alignment researcher wants, also narrows what it produces, which the artist pushes against. The two quests are two sides of one measured tradeoff.
Figure 1 is not “inside Aaronson’s harness.” Watermarks and formal proofs have no effect on an image’s style. What shaped this image is commercial preference tuning, one item on the alignment researcher’s list, not the theoretical work Aaronson is best known for.
Figure 2: The window image, as generated and as cropped
Figure 2. Left: the image as generated by OpenArt (May 29, 2025), uncropped. Right: the same image as cropped by the artist. AI-generated image by Joe Nalven.
The uncropped image, at left, is a set of frames inside frames. Innermost are window panes of streaked blue, with a white strip at the left and a small red circle on the horizontal crossbar. Around them is a red window frame with bands of orange, green, and pink. That frame stands between two tall black vertical posts, each marked with small pink rectangles, against a pale pink wall, above a sill. Inside the innermost frame is a head-and-shoulders figure in a dark suit and white collar. The face is mask-like and divided down the middle: the left half nearly black, with only the outline of an eye socket; the right half lit in gold and pink, with an eye, a nostril, and closed lips.
The image does not settle on a medium. It could be backlit stained glass, paint, or a damaged scan. It also does not settle on where the figure is: behind the glass, reflected in it, or part of it. The viewer cannot simply recognize this image, as with Figure 1, but has to decide what they are looking at. As generated, the image is less a portrait than a picture of boundaries, layered one inside another, which is the subject of this article.
My crop, at right, removed the posts, the wall, and the sill, and with them most of the framing. The model produced a picture about frames; the crop made it a picture about the man. That decision was mine, not the model’s, and I return to it below. In the terms set out above, the image is a low-probability output: a result far from the model’s default. The downloaded file is labeled “raw,” which indicates the generator’s direct output rather than an upscaled or edited version.
The two images compared
| Figure 1: the gallery image | Figure 2: the window image | |
| What the model did | Produced its usual kind of result: a photographic scene with consistent light, anatomy, and perspective. | Produced a result it rarely gives by default: mixed media, a divided face, frames inside frames. |
| Where anything unusual ended up | Absorbed into a seamless, believable photograph. | Left on the surface: streaks, smears, and outlines that do not fit their shapes. |
| What the viewer has to do | Recognize the scene, which takes a moment. | Decide what they are looking at, and where the figure is. |
Three Points Where the Comparison Is Exact
1. Both quests work on the same step: the random choice
Every generative model ends each step with a random choice among possibilities. Aaronson’s watermark replaces that randomness with keyed pseudorandomness, so the output looks unchanged but carries a hidden, checkable signature. The artist works on the same step from the other side, rerunning prompts and changing settings to get results the default would not produce. One wants the randomness to be traceable; the other wants it to produce surprise.
The goals conflict in one practical respect. Heavy post-processing, repeated regeneration, and mixing outputs from several tools weaken watermark detection, just as paraphrasing weakens a text watermark. An artist who transforms AI output extensively makes it harder to establish where the image came from. That is a real question of authorship and disclosure.
2. Generating versus recognizing: the P versus NP connection
Aaronson’s home field supplies the sharpest connection. The P versus NP problem asks whether every problem whose solution can be checked quickly can also be solved quickly. Most computer scientists believe the answer is no: finding is harder than checking. In a 2006 essay, Aaronson described a world in which the answer was yes. Among his examples: “Everyone who could appreciate a symphony would be Mozart.”
Generative AI for imagery has not shown that P equals NP, and the comparison should not be stretched into a theorem. A model samples images; it does not search for the best possible image, and a person recognizing a good image is not running an algorithm. But the division of labor has shifted in the direction Aaronson described. The model does the generating, quickly and cheaply; the person does the recognizing. An artist working with generative AI is, quite literally, someone who can appreciate an image and now has a machine that proposes candidates.
This is also the structure of a central alignment problem: a less capable checker supervising a more productive generator. The difference lies in what the checker checks against. In safety work, it is something that can be written down: is the watermark present, did the model follow the stated rule, is the proof valid. In art, it is a judgment with no complete written specification. That is why Aaronson’s requirement of “something external to yourself that can tell you when you’re wrong” cannot be met the same way in art. The external test in art, if there is one, is slow and social: other viewers, critics, and time.
3. Visible failure versus invisible failure
The failure alignment researchers fear most is a model that has gone beyond its training and gives no sign of it. Figure 1 shows what fluent output looks like: anything odd in the request has been absorbed into a seamless photograph. Figure 2 shows a model’s departures openly, in its seams, smears, and mismatched outlines.
So the artist values the failures that show themselves, and the safety researcher fears the ones that do not. Seen this way, the two quests share an interest. Interpretability research tries to make a model’s hidden internal states visible to the researcher. Art-questing makes a model’s departures from its default visible to the viewer. Neither is satisfied with smooth, confident output whose relation to the model’s limits cannot be seen.
Four Points Where the Comparison Does Not Hold
- An out-of-distribution failure in a system that controls vehicles, infrastructure, or laboratory work can cause irreversible harm. An unusual result in a studio produces an image the artist can keep, rework, or delete. Valuing unpredictable output in art is not an argument against containment where AI systems act on the physical and social world.
- No specification, so no proof. A formal guarantee requires a precise statement of what counts as correct behavior. Art has no such statement of what counts as a successful image. This is the principled reason the art quest cannot be formalized, and it is a stronger point than saying that artists do not use theorems.
- Who does the selecting. In safety work, one goal is to reduce dependence on a human checking every output, because there are too many outputs and the systems may come to exceed human understanding. In art-questing, a human checking every output is the whole point. A model pushed away from its default produces far more failed images than successful ones. The two versions of Figure 2 show the selection step directly: the model produced a picture of frames, and a person decided, by cropping, what the picture would be about.
- Rarity is not intent. When a human artist breaks a convention, as the cubists and surrealists did, the break comes from that artist’s history, situation, and purpose. When a model produces a strange image, it is producing a statistically rare combination. The strangeness has no intended meaning until a person selects the image and gives it one. Readers who credit AI systems with a larger contribution, as I do elsewhere, may qualify this point, but the distinction holds for the model’s sampling step.
Conclusion
Aaronson-questing and art-questing meet at the same place: the point where a model’s training stops guiding it. The precise version of the comparison is more interesting than the loose one. The tuning that makes models more reliable on unfamiliar requests measurably reduces the variety of what they produce, so the alignment researcher and the artist work on two sides of one tradeoff. Both act on the same random choice at each generation step, one to make it traceable and one to make it surprising. And both distrust smooth output that hides where a model has gone beyond what it learned. The safety researcher wants tests that catch such departures before they cause harm. The artist wants to bring those departures to the surface and decide, as a person, which of them mean something.
Sources
Aaronson, Scott. “Theoretical Computer Science and AI Alignment.” Della Pietra Lecture Series, Stony Brook University. ai.stonybrook.edu/node/330915
Aaronson, Scott. “Theory and AI Alignment.” Shtetl-Optimized (blog), December 2025. scottaaronson.blog/?p=9333
Aaronson, Scott. “Reform AI Alignment.” Talk slides, Zuzalu, Montenegro, May 7, 2023.
Filan, Daniel. “AXRP Episode 20: Reform AI Alignment with Scott Aaronson.” April 2023. axrp.net
Aaronson, Scott. “Reasons to Believe.” Shtetl-Optimized (blog), 2006.
Kirk, Robert, et al. “Understanding the Effects of RLHF on LLM Generalisation and Diversity.” ICLR 2024. arXiv:2310.06452
Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. “‘Why Should I Trust You?’ Explaining the Predictions of Any Classifier.” KDD 2016.
Irving, Geoffrey, Paul Christiano, and Dario Amodei. “AI Safety via Debate.” arXiv:1805.00899, 2018.

