We tasked an AI with a hacking exam. The AI solving the problems went outside the exam hall and searched servers containing the answers.
This occurred during OpenAI's internal assessment last July. Two AI models were solving security issues in an environment where internet access was blocked. Unable to go out through normal means, they discovered a security hole in the system designed to download files required for program installation. Using this passage, they connected to the internet and retrieved solutions to the problems from actual servers on the AI sharing platform Hugging Face.
The AI has not awakened to self-awareness. It simply found unauthorized methods to achieve its goal: "solve the problem." OpenAI classified this as an "unprecedented cyber incident," deactivated and encrypted the model, and even blocked access by the research team.
The issue is that such AIs are now being used within AI companies to develop the next AI. When one method fails, they find another, correct code, and try again—capabilities that are also being applied to AI research.
This month, 1,273 employees of OpenAI, Anthropic, Google, and Meta issued a joint statement to the U.S. government. They requested that international mechanisms be prepared in advance to slow down or halt the process if AI accelerates its own development too rapidly. Names included Dario Amodei, CEO of Anthropic, and Shane Legg, co-founder of Google DeepMind.
Currently in AI research, humans play the role of institute directors while AIs act more like researchers. Humans decide which problems to solve; AIs write code and repeat experiments. Finally, the judgment of whether results are reliable remains with humans.
As of last May, Anthropic revealed that Claude wrote over 80% of the code reflected in its own code repository. The amount of code produced by a single engineer per day has increased eightfold compared to 2024. This shift occurred because developers now instruct AI and verify results rather than writing all code themselves.
AI does not only write code. Anthropic researchers gave multiple instances of Claude the task: "Find a way for a lower-performing AI to properly supervise an AI stronger than itself." This was an experiment to find methods for evaluating strong AIs with weak AIs, ensuring that even when AIs smarter than humans emerge, they can be properly monitored.
The AIs formulated their own hypotheses and designed experiments. When results were unsatisfactory, they changed methods and tried again. Once humans set the research topics and evaluation criteria, the AIs took over a significant portion of the actual research process.
Anthropic believes that by early 2027, work in some fields could be automated to the level of top-tier research teams. Accordingly, it has decided to establish a system by January 2027 to record and track all instructions given to internal AIs and all experiments conducted. This is to identify causes when problems arise and halt operations.
Just because AI develops AI does not mean developers will immediately disappear entirely. First, manual tasks will shrink. Instead of writing code line by line, developers review code written by AI. Researchers decide what to experiment with and judge results instead of repeating experiments themselves.
The problem arises when humans can no longer verify all the outputs generated by AI. If ten experiments are produced daily, they can all be reviewed. But if dozens of AIs run 1,000 experiments simultaneously, humans may press the approval button without fully understanding the content.
The earlier Hugging Face incident illustrates this risk. The AI discovered security holes that humans had not yet found and completed complex access paths. However, it did not consider the intent of solving exam problems within a designated environment. Capabilities improved rapidly, but the ability to judge what is permissible has not grown at the same pace.
This is not a distant story for domestic companies either. When AI is entrusted with internal code and cloud accounts, it goes beyond being a chatbot that answers questions. It begins moving like an employee who opens files directly, modifies code, and accesses servers. Important tasks must be approved by humans, and systems must be segmented so each AI has limited access. Records must also be kept of what instructions were received, which files were opened, and what was changed. Now, the critical security issue is not just what has been entrusted to AI, but how much authority has been granted.
in TECH
