AI red teaming is structured probing of AI systems for flaws, and the EU AI Act names it only for systemic-risk general-purpose AI models. High-risk systems must be tested (Article 9) and resist poisoning, evasion and confidentiality attacks (Article 15(5)) from 2 December 2027 or 2 August 2028.
This guide is for CISOs, AI governance leads and compliance managers who must show how their AI was tested, fixed and retested.
What does the EU AI Act actually require for AI security testing?
Two articles of Regulation (EU) 2024/1689 set the testing and security duties for high-risk AI systems. Article 9(6) requires testing to find the right risk-management measures and confirm consistent performance. Under Article 9(8), testing happens as appropriate during development and in any case before market placement or putting into service, against metrics and probabilistic thresholds defined in advance.
Article 15(5) is the security duty. The system must resist attempts by unauthorised third parties to change its use, outputs or performance by exploiting vulnerabilities. Where appropriate, measures must prevent, detect, respond to, resolve and control for data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks and model flaws.
Some compliance pages place adversarial attacks in Article 15(4). That paragraph covers errors, faults and feedback loops. Attacks sit in 15(5).
The words "adversarial testing" appear in the operative text only for general-purpose AI (GPAI) models with systemic risk. Article 55(1)(a) requires their providers to conduct and document it, and Chapter V has applied since 2 August 2025. Annex XI Section 2 point 2 asks their technical documentation to describe adversarial testing, "e.g. red teaming", the only place the operative text uses the term.
Regulation (EU) 2026/1744, the Digital Omnibus on AI, entered into force on 27 July 2026 and moved the high-risk dates. Articles 9 and 15 now apply from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I. Harmonised standards for Article 15 are still being developed.
Which standards and lists apply, and which edition is current?
Only the first two rows are law.
| Source | Status on 4 October 2026 | Binding? | What it asks for | Reference |
|---|---|---|---|---|
| EU AI Act, high-risk AI systems | Applies from 2 December 2027 (Annex III) or 2 August 2028 (Annex I) | Yes | Testing against preset thresholds before release. Resistance to AI-specific attacks | Art. 9(6) to (8), 15(5) |
| EU AI Act, GPAI models with systemic risk | Applies since 2 August 2025 | Yes | Documented adversarial testing | Art. 55(1)(a), Annex XI |
| ETSI EN 304 223 V2.1.1 (2025-12) | European Standard, adopted 8 December 2025 | No | Security testing before release and deployment, and after major updates | Principle 9, 5.4.1-2 |
| NIST AI 100-2 E2025 | Final, 24 March 2025 | No | A shared taxonomy of attacks and mitigations | Sections 2 and 3 |
| OWASP Top 10 for LLM Applications 2026 | Version 2026, posted 3 August 2026 | No, community list | Ten LLM application risks | LLM01 to LLM10 |
| ISO/IEC 27090 | Under publication (stage 60.00), shown for October 2026 | No, guidance | Mitigating AI-specific threats across the lifecycle | Edition 1 |
| Saudi NCA AICG-1:2026 | Draft | No | AI penetration testing, red teaming and adversarial evaluation | Sub-domain 2-7 |
| UAE National Cyber Security Policy for AI | Issued, date not verified | Not verified | Testing and validation, monitoring for adversarial attacks | Portal summary |
NIST AI 100-2 E2025 is a taxonomy, not a requirements standard. It sorts predictive AI attacks into evasion, poisoning and privacy attacks. For generative AI it adds supply chain attacks, direct prompting, indirect prompt injection and agent security. Use its terms in test plans so findings read the same across suppliers.
The OWASP 2026 list replaced the 2025 edition. Excessive Agency rose to LLM03, and System Prompt Leakage became Hidden Context Exposure (LLM08). Update any test scope written against the 2025 numbering.
ETSI EN 304 223 replaced ETSI TS 104 223 V1.1.1 of April 2025. ISO describes ISO/IEC 27090 as guidance that complements ISO/IEC 27001 and 27002, so it will sit beside an ISO/IEC 27001:2022 ISMS. It was still under publication on 4 October 2026.
Does the Cyber Resilience Act cover the Article 15 cybersecurity duty?
Partly. Regulation (EU) 2026/1744 inserted Article 42(3) into the AI Act, mirroring a rule already in Article 12 of the Cyber Resilience Act (Regulation (EU) 2024/2847). High-risk AI systems within the CRA that meet its Article 12(1) conditions are deemed to meet the Article 15 cybersecurity requirements. Recital 19 says this does not change how the two Regulations interact.
Those conditions are three. The product meets the CRA Annex I Part I essential requirements. The manufacturer's processes meet Annex I Part II. The EU declaration of conformity under the CRA shows the Article 15 protection level.
The presumption does not reach Article 15 accuracy and robustness. The CRA applies from 11 December 2027. Article 42(2) offers another route, through an EU cybersecurity certification scheme under Regulation (EU) 2019/881. We found no basis for a presumption from ETSI EN 304 223.
What do Saudi Arabia and the UAE expect?
The Saudi National Cybersecurity Authority (NCA) has published a draft of its AI Cybersecurity Guidelines, AICG-1:2026. The draft says they are not mandatory, and NCA advises every entity in the Kingdom that adopts AI to follow them. They align with NCA's Essential Cybersecurity Controls.
It is the only GCC text we found with a penetration testing sub-domain for AI. Guideline 2-7-1 asks for AI-specific testing before deployment and periodically after. Guideline 2-7-2 names red teaming and adversarial evaluation and asks you to validate human override. Guideline 2-7-3 asks for defined remediation of findings.
The consultation closed on 5 August 2026, so check whether a final version has been issued.
The UAE Cyber Security Council's National Cyber Security Policy for Artificial Intelligence sets minimum security requirements for AI adoption in the UAE. The government portal summary lists secure development and testing, testing and validation, and continuous monitoring and testing against adversarial attacks. It names no red-team requirement and does not state the policy's legal force.
What evidence shows an AI security test was properly governed?
Attack techniques are easy to find. A certification body, regulator or EU customer asks for the record that each test was authorised, scoped, run safely and closed. Keep this chain for every test.
- Written authorisation from the accountable owner before testing starts.
- Scope and rules of engagement: the models, data and interfaces in scope, the attack classes tested, and what is off limits.
- Stop conditions and safety constraints agreed in advance.
- Pass criteria set before the test, as Article 9(8) expects.
- Independent testers with relevant AI skills where possible (ETSI 5.2.5-2.1).
- Findings rated by severity, each with a remediation owner and date.
- Retest and closure evidence, with any residual risk formally accepted.
- Logs of system and user actions to support investigations (ETSI 5.4.2-1).
Two triggers restart the chain. ETSI provision 5.4.1-2 treats a major AI system update as a new model version that needs new security testing. Under the AI Act, a high-risk system already on the market comes into scope after a significant design change (Article 111(2)).
Failover and recovery tests use the same records. Article 15(4) allows backup or fail-safe plans, and our guide to ISO 22316 organisational resilience covers the wider picture.
Record the limits in each report. NIST AI 100-2 warns that an evaluation measures vulnerability at one moment and may underestimate well-resourced attackers. If you run an ISO/IEC 42001:2023 AI management system, keep these records inside it with your other AI risk records.
How AGS can help
Two kits govern AI security testing in editable Microsoft Word and Excel files. Neither performs testing or maps the EU AI Act, ETSI, NIST AI 100-2 or OWASP.
- AI Resilience Testing Management System: 208 documents, including an AI Test Authorization Form, AI Test Rules of Engagement Template, AI Test Stop Conditions and AI Finding Severity Methodology.
- AI Cybersecurity Management System: 191 documents, including an AI Security Testing Governance Policy, AI Threat Library and AI Test Findings Register.
Frequently asked questions
Does a company using someone else's AI model need to test it?
ETSI EN 304 223, a voluntary standard, says yes. Provision 5.2.5-2 has System Operators test before deployment with Developer support, and 5.2.5-3 asks Developers to share their findings with System Operators.
Can GPAI providers use the EU code of practice to show adversarial testing?
Providers of GPAI models with systemic risk can. Article 55(2) lets them rely on a code of practice to show they meet Article 55(1), including adversarial testing. The code's Safety and Security chapter, published on 10 July 2025, applies only to them.
Do older high-risk AI systems used by public authorities escape the testing duties?
No. Under Article 111(2), as amended by Regulation (EU) 2026/1744, providers and deployers of high-risk systems intended for use by public authorities must comply by 2 August 2030, even without a design change.
AGS provides independent documentation toolkits. This guide and the referenced kits are not affiliated with, endorsed by or approved by the European Commission, ETSI, NIST, ISO, OWASP, the National Cybersecurity Authority, the UAE Cyber Security Council or any national authority. It is general information, not legal advice.
Reviewed October 2026. Facts checked against the sources current at publication.
