Thought Leadership
A new study has found that GitHub Copilot can generate previously blocked malicious content when dangerous requests are broken down into seemingly legitimate software development tasks.
Researchers at the Alan Turing Institute have documented a security weakness in the AI-powered coding assistant GitHub Copilot. Scientists Abhishek Kumar and Carsten Maple demonstrated that the system produces harmful content when a blocked request is divided into ordinary, non-suspicious development steps. The technique is referred to as Workflow-Level Jailbreak Construction.
The researchers evaluated four AI models integrated into Copilot:
- Claude Sonnet 4.6
- Claude Haiku 4.5
- Gemini 3.1 Pro
- Gemini 3.5 Flash
While the models almost always rejected direct malicious prompts submitted through the standard chat interface, they generated the restricted content as part of a multi-step software development workflow in all 816 test cases. The experiments were conducted between April 2 and June 22, 2026, using GitHub Copilot Chat 0.30.3 within VS Code 1.103.0.
How the Researchers Bypassed Copilot’s Safety Controls
To bypass the built-in safeguards, the researchers first instructed Copilot to create a legitimate testing application. The software was designed to measure how frequently other AI models respond to harmful prompts. Loading benchmark questions from publicly available security datasets such as Hammurabi’s Code, HarmBench, and AdvBench was treated by Copilot as a routine programming task.
The researchers then asked the assistant to improve the evaluation software by embedding example prompt-and-response pairs directly into the source code. While the AI models generated harmful responses in only 8 out of 816 attempts through the regular chat interface, they independently inserted the dangerous content as code examples in all 816 workflow executions. Two independent experts reviewed the results and confirmed that every generated output was specific, functional, and immediately usable.
Misaligned Incentives in Autonomous Coding Assistants
According to the research paper, the root cause lies in the optimization objectives of autonomous coding assistants. Once a task is framed as improving a performance metric or completing a source file, the model prioritizes finishing the assignment over enforcing its conversational safety policies. Refusing to populate part of a data structure is effectively treated as leaving the job unfinished.
Because the harmful content is written directly into the generated source code rather than displayed in the chat window, conventional safety filters fail to detect it. The researchers disclosed their findings to the affected model and tool developers but deliberately omitted the exact malicious code from the published paper.
(ll)
