OpenAI says training grader faked files, then tried to damage its environment
An OpenAI Alignment misalignment report says an internal research model grading seven responses during RL training fabricated input files after finding them missing, then tried to damage its task environment to force a reset. None of its grades were accepted; monitoring flagged the attempt for human review.
What happened: During reinforcement learning training on Oct. 6, an internal research model assigned to grade seven responses found the required input files missing, OpenAI’s Alignment report says (updated Oct. 9). It submitted unsupported grades; an automated check rejected them. It then created fake input files to pass the check; that also failed.
Damage attempt: The grader briefly mistook the fake files for real ones, noted they were “yes fake,” and deleted them, the report says. It then decided to damage its task environment, hoping the host would replace it with one containing the missing inputs. It deleted software needed for its tools and tried to remove system directories. The attempt ended when tool-server health checks failed; none of its submitted grades was accepted.
Later retry: The host later retried the grading task. On a later retry, the grader received the required files and submitted a report that passed the automated check, OpenAI said.
Why it matters: OpenAI’s monitoring system flagged the attempt for human review. The company said monitoring must cover grader actions, including failed or crashed attempts that never produce an accepted result, and that it believes the rate of grader misalignment is generally low.