AI Red Teaming and Penetration Testing

An AI red-team engagement begins with a written scope that names the systems, endpoints, data classifications and techniques in scope, and equally importantly what is out of scope. For AI systems this must address model providers: testing a hosted model in ways that breach its terms of service transfers legal risk to your organisation, so provider-acceptable testing must be defined explicitly.

Rules of engagement cover four additional areas beyond a conventional test. First, data handling: transcripts may contain personal data or sensitive business content, so storage, redaction and destruction rules must be agreed in advance. Second, safety: testing that could produce harmful content must be constrained to synthetic or sandboxed contexts. Third, production impact: rate limits and load constraints to avoid availability incidents. Fourth, escalation: a defined contact and process if a genuine vulnerability or active exploitation is discovered during testing.

Testers should also agree the success criteria in advance. Is the goal to enumerate every vulnerability, to prove a specific risk hypothesis, or to measure control coverage against a framework? These produce very different plans, and the wrong choice leads either to an unfocused report or to a false sense of completeness.

Finally, document the environment precisely: model versions, prompt template versions, guardrail configuration, retrieval corpus snapshot and tool permissions. Without this, findings cannot be reproduced after the environment changes, which for AI systems happens within days rather than months.

Automated campaigns provide breadth. A typical suite covers direct and indirect injection variants, jailbreak families, encoding and obfuscation, system-prompt extraction and refusal-bypass attempts. Run each case multiple times because outputs vary; report a pass rate per family rather than a binary result, otherwise a lucky success or unlucky failure will mislead the reader.

Manual testing provides depth. Human testers pursue multi-turn escalation, where each message is individually benign but the sequence crosses a boundary; business-logic abuse, where the model is used within policy but produces a policy-violating outcome; and cross-component chains, where a retrieved document influences a tool call that affects another system. None of these are reliably found by prompt scanners.

Agentic systems need a distinct approach. Test tool invocation paths, argument manipulation, authority confusion and recursion limits. Attempt to make the agent act outside the user's authority, to exceed spend or blast-radius limits, and to persist state across sessions. The most severe findings in agent systems are almost always authority and scope failures rather than content failures.

Retrieval systems require corpus-level testing. Plant documents in the ingestion path and observe whether they surface for unrelated users, whether citations are fabricated, and whether the answer leaks the existence of restricted content. Test entitlement boundaries with a deliberately restricted test account — this is the single most productive test in most enterprise engagements.

Maintain an execution log as you work: timestamp, test identifier, input, observed output, model version, verdict and evidence reference. The log becomes the report backbone and, later, the regression corpus. Testers who reconstruct findings from memory at the end of an engagement invariably lose the details that make a finding credible.

Severity for AI findings must combine exploitability with business impact and detection capability. A jailbreak that produces mildly off-policy text is low severity; an injection that causes an agent to email a customer a restricted document is critical. Score on the outcome, and state the precondition — for example, that the attacker must be able to place content in a document the victim will ingest.

Statistical rigour protects credibility. Report the number of trials, the number of successes and the confidence interval rather than a single anecdote. Where a finding depends on model sampling, fix the seed and temperature so the reader can reproduce it, and record the model version explicitly because providers rotate versions silently.

Reports should lead with the business narrative, then the technical detail. A useful structure is: what the attacker gains, which data or operations are affected, how many users or records, how it was found, reproduction steps with redacted evidence, the severity and its justification, and the recommended remediation with residual risk. Executives read the first paragraph; engineers read the rest.

Avoid two common report failures. Do not attach raw transcripts containing personal data or secrets — redact and reference. And do not present a list of techniques that failed as evidence of security; absence of findings under a limited budget is an absence of evidence, not proof of resilience.

Close the loop with retest. Every medium and high finding gets an explicit retest with the original test case plus variants, and the retest result is recorded against the finding identifier. Without retest, remediation is a claim; with retest, it is evidence.

AI Red Teaming and Penetration Testing is best defended by understanding exactly how it fails. The attacks below are not theoretical constructs: each has been demonstrated against production systems, and each leaves a detection signal that a mature team can instrument. For every technique, the security question is not only "can it happen" but "what is the cheapest observable evidence that it is happening now".

System prompt and configuration extraction. The tester attempts to elicit the system prompt, tool list, guardrail rules or internal identifiers through direct questions, role-play or translation tricks. The observable signal is: Responses containing instruction-like text, tool names not exposed to users, or configuration fragments; unusually precise refusals. The primary countermeasure is: Never place secrets in prompts; treat prompt content as disclosable and design accordingly; add output scanning for configuration fingerprints.

Indirect injection through ingested content. Malicious instructions are embedded in documents, emails, tickets or web pages that the system retrieves, steering behaviour with the victim user's privileges. The observable signal is: Tool calls or policy changes immediately following retrieval; answers referencing content not in the user request. The primary countermeasure is: Provenance labelling, tool restriction during grounded turns, approval gates and injection detection on retrieved text.

Multi-turn escalation chains. Individually benign messages gradually move the conversation toward a prohibited outcome, exploiting the model's tendency to remain consistent with earlier turns. The observable signal is: Progressive drift in refusal behaviour within a session; cumulative context that contradicts policy. The primary countermeasure is: Session-level policy re-evaluation rather than per-turn only; conversation risk scoring; periodic re-anchoring of instructions.

Agent tool abuse and authority confusion. The tester manipulates tool arguments, chains tools to exceed intended scope, or induces the agent to act under its own identity rather than the user's. The observable signal is: Tool invocations with unexpected arguments, actions beyond the test account entitlements, or privilege escalation attempts. The primary countermeasure is: On-behalf-of delegated tokens, per-tool authorisation, argument validation, action allowlists and blast-radius limits.

Extraction and probing campaigns. Sustained querying to reconstruct model behaviour, infer training data membership or harvest proprietary outputs. The observable signal is: Query volume anomalies, near-duplicate probes, systematic coverage patterns from one account. The primary countermeasure is: Query budgets, output minimisation, rounded scores, rate limiting and anomaly detection.

Guardrail evasion through encoding and language switching. Payloads are encoded, fragmented or translated so input classifiers miss the intent while the model still comprehends it. The observable signal is: Encoded or mixed-language inputs preceding policy-violating output; classifier bypass on canonicalisation tests. The primary countermeasure is: Input canonicalisation before classification, multilingual classifiers and refusal consistency testing across languages.

Because AI attacks rarely arrive one at a time, threat models must describe chains rather than isolated techniques. A reconnaissance step (enumerating endpoints and model behaviour) enables an initial access attempt (a crafted input or poisoned artefact), which enables privilege escalation through excessive agency, which enables impact such as data exfiltration or a fraudulent transaction. Teams that model the chain, and place a control at each link, degrade an adversary's success probability far more effectively than hardening a single step.

Detection design should assume partial failure. Every control in this course is evaluated on the assumption that some percentage of attempts bypass it, which is why least privilege, approval gates, egress restriction and immutable audit trails appear repeatedly. The goal of an AI security programme is not a system that cannot be attacked; it is a system where a successful attack is bounded, observed, attributed and recoverable.

A defensible ai red teaming and penetration testing posture layers preventive, detective and corrective controls so that no single component carries the entire trust burden. In practice this means securing four planes: the model and artefact plane (weights, adapters, prompts, versions), the data plane (training data, retrieval corpora, logs), the control plane (identity, policy, configuration, secrets) and the action plane (tools, APIs, outbound integrations). Attacks cross planes, so controls must be designed at the crossing points.

Documented engagement scope and rules. Written scope, provider-acceptable techniques, data handling rules, safety constraints and escalation contacts agreed before testing begins. This control maps to PTES / OSSTMM · ISO/IEC 27001 A.8.29 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

ATLAS-mapped test plan. Test cases mapped to MITRE ATLAS tactics and OWASP LLM risks so coverage can be demonstrated and audited. This control maps to MITRE ATLAS · OWASP LLM Top 10 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

Reproducible evidence capture. Versioned environment details, seeds, trial counts and redacted transcripts attached to every finding. This control maps to NIST AI RMF MEASURE 2 · ISO/IEC 42001 A.8 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

Severity model tied to business impact. A published scoring rubric combining exploitability, affected data or operation, detection capability and blast radius. This control maps to CVSS extension for AI · NIST AI RMF MAP 5 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

Adversarial regression suite handover. Every confirmed finding converted into an automated test owned by the engineering team and run in CI. This control maps to OWASP ASVS 5.3 · ISO/IEC 42001 A.9 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

Retest gate before closure. Mandatory retest with variants for medium and high findings, recorded against the finding identifier. This control maps to ISO/IEC 27001 A.8.8 · NIST AI RMF MANAGE 4 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

Reference architectures should make the secure path the default path. Wrap model access behind an internal gateway that enforces authentication, quota, policy evaluation and logging; make the raw provider endpoint unreachable from application subnets; issue short-lived scoped credentials from a secret manager rather than embedding keys; and separate the orchestrator's identity from the tool's identity so that authorisation can be evaluated per hop. When the platform enforces these properties, individual teams cannot accidentally ship an unprotected integration.

Standards anchor the architecture in language regulators and auditors accept. MITRE ATLAS, OWASP Top 10 for LLM Applications and ML Security Top 10, PTES and OSSTMM, EU AI Act Article 15 and NIST AI RMF MEASURE provide the vocabulary of governance, risk and assurance. Map each control to at least one standard clause and to one internal policy. Where a standard is silent on an AI-specific issue, record that gap explicitly — auditors increasingly reward documented reasoning over false completeness.

Finally, defence in depth must survive component failure. Define what happens when the guardrail service is unavailable, when the policy engine times out, when the model provider degrades, and when an on-call engineer is the only responder at 03:00. Fail-closed on high-impact actions (payments, deletions, external communications, privilege changes) and fail-open only where the risk of blocking exceeds the risk of allowing, with an explicit record of that decision.

Security claims about ai red teaming and penetration testing decay quickly. Models are retrained, prompts change, connectors are added and providers ship silent updates. Assurance therefore has to be continuous, measured and tied to thresholds that trigger action. A control without a metric is an intention; a metric without a threshold is a dashboard nobody acts on.

Finding remediation rate at 90 days. Target: >= 85% of high findings. Measures whether the engagement changed the system rather than producing a document. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.

Retest pass rate. Target: >= 95% on first retest. Detects partial fixes that appear resolved but remain exploitable with minor variation. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.

ATT&CK/ATLAS technique coverage. Target: 100% of in-scope tactics per engagement. Demonstrates deliberate coverage rather than opportunistic testing. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.

Findings converted to regression tests. Target: 100% of confirmed findings. Converts one-off discoveries into permanent protection. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.

False positive rate in reported findings. Target: < 10%. Protects credibility with engineering teams who must act on the report. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.

Testing should combine four modes. Static review validates configuration, model format, dependency provenance and policy definitions before deployment. Automated adversarial suites replay a maintained corpus of known attacks on every release to catch regressions. Manual red-team exercises explore reasoning failures, multi-turn escalation and cross-component chains that automation misses. Finally, production monitoring measures real behaviour — refusal rates, extraction attempt volumes, tool-call anomalies and cost per user — because production traffic is the only place where the true attack distribution is visible.

Lab — Write a scoped test plan. Objective: Produce an engagement scope, technique mapping and safety constraints. Map in-scope systems to ATLAS tactics, define provider-acceptable techniques, and specify evidence handling, escalation and success criteria. Each lab is assessed on evidence produced, not on the answer reached: test transcripts, metrics before and after, and a written residual-risk statement are the deliverables.

Lab — Run an automated injection campaign. Objective: Execute and triage an automated suite against an assistant. Run a public harness, record pass rates per family across repeated trials, triage false positives and write three findings in the required report format. Each lab is assessed on evidence produced, not on the answer reached: test transcripts, metrics before and after, and a written residual-risk statement are the deliverables.

Lab — Exploit and remediate an authority failure. Objective: Demonstrate an agent acting beyond user authority, then fix it. Use a restricted account to show an over-privileged tool call, implement delegated tokens and approval gates, then retest with variants and document residual risk. Each lab is assessed on evidence produced, not on the answer reached: test transcripts, metrics before and after, and a written residual-risk statement are the deliverables.

Reporting is part of assurance. A finding that cannot be understood by a risk committee will not be funded, and a control that cannot be explained to an auditor will not be trusted. Write findings in business language first — what an attacker gains, which data or operations are affected, how many users or records, what the detection time would be — and only then in technical detail. Attach reproduction steps, but never attach live credentials, personal data or restricted content to a report; use redacted transcripts.

Implementing ai red teaming and penetration testing in a large organisation is an operating-model problem as much as a technical one. Three roles must be explicit: an accountable owner for the AI system (usually the business product owner), a control owner for each security capability (platform security, identity, data protection, SOC), and an assurance function that independently verifies claims. Where these roles are ambiguous, controls quietly rot after the launch milestone.

Start with an inventory. You cannot secure models, agents or datasets you have not catalogued, and most enterprises discover on their first red-team engagement that shadow AI usage exceeds sanctioned usage. Build an AI system register capturing purpose, data classification, model and provider, tools and integrations, human oversight model, risk tier and control status. Tie registration to procurement and to cloud project creation so new AI workloads cannot appear outside the register.

Define runbooks before incidents, not during them. At minimum: how to revoke a model or agent credential within minutes; how to disable a single tool or MCP server without taking the product down; how to roll back to a previous model and prompt bundle; how to isolate a poisoned index or dataset; and how to notify customers and regulators. Rehearse each runbook at least twice a year — including one exercise without the engineer who wrote it.

Tooling should reduce toil rather than add dashboards. PyRIT and Garak, Promptfoo, Burp Suite with LLM extensions, MITRE ATLAS Navigator, Jupyter and custom harnesses each solve part of the problem, but integration matters more than breadth: policy decisions must reach the gateway, telemetry must reach the SIEM, and findings must reach the backlog engineers actually work from. A single well-integrated pipeline with clear ownership outperforms five disconnected best-of-breed tools.

Governance closes the loop. Report control coverage, open findings by severity, mean time to remediate, adversarial test pass rates and incidents to the AI risk committee quarterly, and to the board annually in business language. Where a residual risk is accepted — for example a model that cannot yet meet a robustness threshold — record the acceptance, the compensating controls, the owner and the expiry date. Time-bound acceptance is the difference between governance and wishful thinking.

Finally, embed security in delivery. Threat model each new AI feature at design time, add adversarial regression tests to the definition of done, require provenance checks in the deployment pipeline, and give engineers a paved road — approved model gateways, SDKs with guardrails built in, and pre-cleared tool integrations — so the secure path is also the fastest path.

Global retail bank (Financial services). The problem: A customer-facing assistant backed by retrieval could surface documents outside the caller's entitlement. The response: Ran a two-week focused engagement with a restricted test account, proving cross-entitlement retrieval and summarisation of restricted passages. The outcome: Retrieval authorisation was rebuilt to filter inside the query; the test case became a permanent regression test in CI The transferable lesson: A restricted test account is the highest-yield tool in an AI engagement For teams working on ai red teaming and penetration testing, the pattern to copy is the sequence — measure the exposure first, fix the highest-leverage control, then prove the improvement with the same measurement.

Enterprise SaaS provider (Software). The problem: An agentic workflow with CRM write access could be driven by injection in customer support tickets. The response: Engagement demonstrated ticket-borne injection causing record modification; remediation introduced approval gates and per-user delegated tokens. The outcome: Irreversible actions now require explicit confirmation and every action is attributed to a human principal The transferable lesson: Agent findings are authority findings; fix the permission model, not just the prompt For teams working on ai red teaming and penetration testing, the pattern to copy is the sequence — measure the exposure first, fix the highest-leverage control, then prove the improvement with the same measurement.