AI Vendors Are Hiding Their Reasoning. Here Is Why That Matters

AI Vendors Are Hiding Their Reasoning. Here Is Why That Matters

A paper dropped last week with an unusually good domain name: stolen-thoughts.com. The research behind it is not a thought experiment. It is a working attack against the APIs that a lot of businesses are building on right now.

Here is the short version: Anthropic, OpenAI, and Google return encrypted chain-of-thought blocks when you call their reasoning models. Those blocks are meant to stay opaque. The researchers figured out how to replay a reasoning trace from a frontier model into a weaker sibling model, jailbreak the weaker one, and recover the original chain of thought. Effectively, they extracted the thinking that vendors are actively trying to keep hidden.

Simon Willison flagged the paper on August 11, 2026, and his summary is worth reading in full at simonwillison.net. But the implications for anyone running business workflows on top of these APIs go further than the paper itself.

What "Encrypted Reasoning Traces" Actually Are

When you use a reasoning model like o3, Claude's extended thinking, or Gemini's equivalent, the model does not just return an answer. It works through a scratchpad first, and some of that scratchpad gets passed back to you in an encrypted or opaque block. The idea is that you can hand this block back to the model in a later turn to maintain continuity, but you are not supposed to be able to read it yourself.

Vendors have incentives to keep this opaque:

• The reasoning process reflects proprietary training decisions
• Visible chain-of-thought can make models easier to manipulate
• It gives vendors something to differentiate on beyond raw benchmark performance

The problem the researchers found is that "opaque" is not the same as "secure." If those encrypted blocks can be replayed across sessions, across users, and across different models (all of which the paper confirms), then the boundary between your reasoning session and someone else's is a lot thinner than advertised.

The Attack in Plain Terms

The replay attack works roughly like this:

1. A reasoning trace is captured from a call to a powerful frontier model
2. That trace is fed into a weaker model from the same vendor's family
3. The weaker model is then jailbroken using standard techniques
4. Because the weaker model is now operating partly on the frontier model's reasoning scaffold, it surfaces content from the original chain of thought

This is not a theoretical side channel. It is a practical method for extracting what a model was actually thinking when it produced an output, bypassing the encryption wrapper the vendor put around it.

The cross-session and cross-user replay findings are the part that should get attention from anyone building multi-tenant AI features. If reasoning traces can cross user boundaries, that is a potential data leakage path in any product where different customers share the same underlying API calls.

Why This Is Not Just a Research Curiosity

Most businesses using LLM APIs are not doing novel AI research. They are automating support, drafting content, parsing documents, routing customer queries. The reasoning traces those calls generate may include fragments of the prompts, the data being processed, or the logic the model used to reach a conclusion.

You probably have not thought about those traces as sensitive data. After this paper, you should.

Three specific risks are worth naming:

Prompt leakage. Your system prompt is often part of what influences the reasoning trace. If that trace is recoverable, your proprietary prompt logic is recoverable too. For businesses that have invested in careful prompt engineering to differentiate their product, this is an IP concern, not just a security one.

Data leakage across users. If you are running a SaaS product where different customers interact with the same LLM integration, the cross-user replay finding means customer A's context could theoretically surface in customer B's session under the right attack conditions. This is the kind of incident that ends with a breach notification email.

Jailbreak amplification. Using a frontier model's reasoning trace to help jailbreak a weaker model means that safety work done at the top of the capability ladder can be undermined by attacking a cheaper, less hardened model. If your application uses a mix of models for cost optimization, the weakest link in your chain matters more than you might expect.

What the Vendors Have Said (and Not Said)

At the time of writing, there are no public statements from Anthropic, OpenAI, or Google specifically addressing the stolen-thoughts.com findings. The paper is recent, published in August 2026 on alphaXiv, and vendor response timelines vary.

It is worth noting that OpenAI is simultaneously rolling out ads in ChatGPT, integrating with AWS through a new Daybreak cybersecurity capability set, and expanding into clinical AI adjacent territory (Google's AMIE medical consultation research is a separate effort but signals the direction the whole sector is moving). These are not companies that are standing still. But rapid product expansion and tight security discipline do not always move at the same speed.

The Daybreak cybersecurity integration with AWS Bedrock is actually interesting in this context. OpenAI is positioning itself as a security tooling vendor at the same time that researchers are finding security gaps in the underlying API infrastructure. The cognitive dissonance is not lost on anyone paying attention.

What Operators Should Actually Do

You do not need to stop using these APIs. But you do need to be more deliberate about what you put into them.

Audit what goes into your prompts. System prompts that include customer PII, internal pricing logic, or proprietary business rules are now higher-risk than they were a month ago. Strip out anything that does not need to be there.

Treat reasoning traces as potentially sensitive outputs. If you are logging API responses, which you should be doing for debugging, make sure reasoning trace blocks are handled with the same access controls as any other sensitive log data. Do not write them to a public S3 bucket. Do not include them in analytics pipelines that aggregate across customers.

Think about your multi-tenant exposure. If you are building a product where multiple end users share your API integration, get clarity from your vendor on session isolation. Ask specifically: can a reasoning trace from one session be replayed in another user's session through your integration? Most vendors will not have a crisp answer yet, but asking the question creates a paper trail.

Stay closer to your AI infrastructure. Abstraction layers feel efficient until they obscure something important. Knowing which model version, which API endpoint, and which response format your application is actually using is basic hygiene, and it becomes more important as attack surfaces like this one are documented.

Watch the research. The stolen-thoughts.com paper is peer-reviewed work surfaced by credible commentators. This is how security research is supposed to work: publish the finding, give vendors time to respond, let practitioners adapt. Following researchers like Simon Willison and reading the alphaXiv preprints relevant to your stack is legitimate operational risk management, not paranoia.

The Bigger Pattern Here

The AI supply chain is maturing fast and the security research is catching up to the deployment reality. A year ago, most LLM security concerns were about prompt injection and model hallucination. Now the concerns include API-level replay attacks, cross-session data leakage, and the interaction effects between models of different capability tiers.

This is exactly what happens in any technology platform as it scales. The early adopters build on it, the researchers probe the attack surface, and the vendors patch and harden. We are somewhere in the middle of that cycle right now.

The businesses that come out of this period well are the ones treating their AI integrations with the same diligence they would apply to any third-party data processor. That means vendor reviews, access controls, logging, and staying current on what the research community is finding.

If you are building AI-powered workflows and want a second pair of eyes on how you have structured your integrations, talk to us. And if you are looking at how these kinds of security considerations apply to CRM and customer communication workflows specifically, NuvenarHub is built with data handling as a first-class concern, not an afterthought.

The reasoning traces your vendors are hiding from you are, apparently, not as hidden as they thought. Worth knowing.