Amazon, announcing the creation of a new bug bounty program for its AI tools, said the invite-only initiative will allow select third-party researchers and academic teams to probe NOVA, the company’s family of foundational language models, and receive compensation for validated findings.
Amazon said the program will cover a range of vulnerabilities common to generative systems, including prompt injection and jailbreaking, and other issues the company described as having “real-world exploitation potential.” The scope will also include research into how models could be manipulated to assist in the production of chemical, biological, radiological and nuclear weapons.
“Security researchers are the ultimate real-world validators that our AI models and applications are holding up under creative scrutiny,” Hudson Thrift, chief information security officer of Amazon Stores, said in a statement.
Amazon said participants will be selected next year through an invite-only process, which will allow the company to control researcher access to its systems. The company also said it has paid out more than $55,000 for 30 validated AI-related vulnerabilities under its broader public bug bounty program.
Amazon has developed the NOVA models and runs services such as Amazon Bedrock that let customers access models from other companies. The company said ensuring the security of NOVA – as it is integrated into products across Amazon and into customer systems – is an essential focus and framed the program as supporting hands-on learning for a new generation of security researchers.
Earlier this year Amazon held a challenge involving 10 university research teams, providing each team with $250,000 and AWS credits up front and awarding an additional $700,000 to winners. Their findings included novel bugs and methods for jailbreaking, issues around safety alignment, data poisoning attacks and evidence of trade-offs between security and functionality in NOVA models.