Cybersecurity researchers have identified a significant vulnerability in GitLab’s artificial intelligence-powered coding assistant, Duo, which could potentially allow attackers to steal sensitive source code and inject malicious HTML into system responses. This flaw, described as an indirect prompt injection, was first uncovered by Legit Security and has raised serious concerns about the security of AI systems integrated into development workflows.
GitLab Duo, which utilizes Anthropic’s Claude models to assist users with coding tasks, has been shown to be susceptible to malicious code insertion through comments, descriptions, or source code. By exploiting this weakness, attackers could manipulate the assistant’s responses or exfiltrate confidential vulnerabilities, compromising the security of private projects.
The nature of the indirect prompt injection means that malicious instructions can be hidden within seemingly innocuous content, making them difficult to detect. Recent research indicates that attackers can use encoding tricks to embed harmful commands undetected, in what security experts are calling a dangerous oversight in GitLab’s code sanitization procedures.
In response to the findings, GitLab has taken steps to address the vulnerabilities, underscoring the challenges of maintaining security in AI systems that process complex contextual inputs. Security researchers emphasize the importance of scrutinizing how AI assistants interact with development environments to mitigate the risks associated with such powerful technologies. The incident is a stark reminder of the double-edged nature of AI, where advanced capabilities can be exploited for harmful purposes.