OpenAI Blocks Attempts to Extract Its Models' Internal Reasoning, Restricts Accounts

OpenAI said on September 30 (local time) that it had detected and shut down a coordinated effort to extract the private reasoning its models generate internally. That private reasoning is the material a model works through before it produces an answer, and it is not normally visible to users. According to OpenAI, the effort was aimed at using that internal process without permission to train and improve other models. The company said it banned or restricted the accounts involved and applied additional defenses to the pathways through which reasoning content could leak.

What they tried to extract

The target was the internal reasoning a model goes through while composing an answer. Among the techniques OpenAI observed was pasting encrypted reasoning content from one conversation into a different conversation. The model was then asked to decrypt and transcribe that content, the company said.

OpenAI said neither its encryption nor its databases were breached. It also said there was no direct access to stored user conversations. The company said it closed off a pathway that had allowed someone already holding another user’s encrypted reasoning content to feed that content back in and recover it.

Where the figure of 15,000 users comes from

In all, more than 15,000 users used prompts of a similar form. The activity OpenAI observed began on July 1. On July 24 and 25, the company identified 16,000 related requests sent by more than 4,000 users. As it widened the scope of its investigation, the size of the user group grew to more than 15,000. OpenAI said it had blocked the activity by July 28.

OpenAI said it linked the core group behind the activity to individuals associated with Moonshot AI, the developer of Kimi. It added, however, that it is not clear whether all of the operators it observed belonged to a single entity.

What OpenAI changed

Along with account-level enforcement, OpenAI said it tightened controls on its sign-up process and service infrastructure. It also expanded monitoring of related activity. Inspection now covers not only the requests users send but also what the model outputs, in order to filter out responses that could expose reasoning content.

The response also involved outside cooperation. OpenAI said it worked with external service providers and made use of government and industry information-sharing channels. It said it is continuing to improve defenses for deployment environments operated by partners and for tool outputs. That brings environments where the model connects to other services and tools into scope as well.

The area to watch going forward is OpenAI’s ongoing work to improve defenses for partner-operated environments and tool outputs.